A note on the architecture
This article describes lessons from a production agricultural technology platform. Some implementation details, operational parameters, identifiers, and business-specific logic have been generalized or omitted to avoid exposing proprietary or sensitive information. The goal is to explain the engineering problems, decisions, and principles rather than the private details of the system.
The booking that happened twice
We had built idempotency to prevent duplicate tractor bookings.
Then duplicate tractor bookings started appearing.
The first thing I checked was the idempotency logic.
It looked correct.
That was the problem.
The mechanism wasn't necessarily broken.
The assumption underneath it was.
The platform was designed for field agents working in environments where network connectivity could be intermittent or unavailable. An agent might be in a village registering a farmer, selecting a tractor, choosing hiring dates, and completing a booking while the phone had no reliable connection to the backend.
The application therefore couldn't simply say:
"No internet? Come back later."
The agent needed to continue working.
The operation had to be stored locally and delivered to the server when connectivity returned.
That sounds like an offline mobile-app problem.
It quickly became a distributed-systems problem.
Once a device can create an operation without an active connection to the server, a much harder set of questions appears:
- What happens when synchronization times out?
- What if the server processes the request but the response never reaches the phone?
- What if the mobile client retries?
- How does the server know that two requests represent the same business operation?
- What happens if two identical requests arrive at almost exactly the same time?
- What happens if the database transaction succeeds but Kafka is temporarily unavailable?
- What happens when the booking succeeds but nobody is notified?
These were the problems that shaped the architecture.
And, more importantly, they changed how I think about reliability.
1. Designing for a Network That Cannot Be Trusted
The fundamental requirement was simple:
A field agent should be able to continue important work even when the network disappears.
The workflow looked roughly like this:
- Select or register a farmer.
- Select a tractor or agricultural tool.
- Select the relevant dates.
- Complete the booking.
- Store the operation locally if connectivity is unavailable.
- Synchronize it when connectivity returns.
The backend had dedicated offline workflows for farmer onboarding, tractor hiring, and agro-tool hiring, with staging records that move through synchronization states such as pending, synced, and failed.
On the mobile side, the synchronization engine maintains offline forms, processes pending operations, retries failed synchronization, and recovers operations that may have been left in a syncing state after an interruption.
The important architectural distinction is this:
Creating an operation and delivering an operation are two different things.
The agent creates the booking locally.
The network eventually delivers it.
Those events may happen seconds, minutes, or much longer apart.
Caption:
The offline workflow separates creating an operation from delivering it to the backend.
This distinction becomes important once the same operation can be delivered more than once.
2. The First Line of Defense: Idempotency
The standard solution to repeated delivery is idempotency.
Each offline operation needs a stable identity.
In the mobile application, the offline form carries an idempotencyKey. During synchronization, that existing key is included with the request rather than creating a new identity for each network attempt. The synchronization request sends the value both as part of the request data and through the X-Idempotency-Key header.
The conceptual model is:
One logical booking
│
▼
Operation ID: A
│
┌────┼────┐
▼ ▼ ▼
Attempt 1 Attempt 2 Attempt 3
│ │ │
└─────────┼─────────┘
▼
Same operation
If the server sees the same operation ID again, it should not perform the business operation from scratch.
Instead:
Request #1
│
├── Key = A
▼
Create booking
│
▼
Synced
Request #2
│
├── Key = A
▼
Existing operation
│
▼
Return existing result
That's what idempotency is supposed to provide.
But this is where an important assumption appears:
The server can only deduplicate requests that the client identifies as the same operation.
Code Snippet 1 — Persistent Idempotency Identity
final formData = Map<String, dynamic>.from(form.formData);
formData['idempotency_key'] = form.idempotencyKey;
final headers = {
'Content-Type': 'application/json',
'X-Idempotency-Key': form.idempotencyKey,
'Accept': 'application/json',
};
Caption:
The idempotency key belongs to the logical offline form, not to an individual HTTP attempt.
3. The Idempotency Mechanism Wasn't Broken
Then we encountered duplicate bookings.
The natural question was:
"Why didn't idempotency stop this?"
The investigation revealed a more subtle failure mode.
An earlier client retry path could generate a fresh idempotency key for a retry of the same logical draft.
From the backend's perspective, that changes everything.
Imagine one booking being attempted three times:
ONE LOGICAL BOOKING
│
┌──────────┼──────────┐
▼ ▼ ▼
Retry 1 Retry 2 Retry 3
│ │ │
Key A Key B Key C
│ │ │
▼ ▼ ▼
Server Server Server
│ │ │
▼ ▼ ▼
Booking Booking Booking
1 2 3
The server isn't seeing:
A
A
A
It is seeing:
A
B
C
There is therefore nothing for a key-based deduplication mechanism to connect.
The server isn't necessarily malfunctioning.
It has been given three different identities.
Diagram 2 — The Idempotency/Retry Failure
Caption:
The server can only deduplicate what the client identifies as the same operation.
This is one of the most important lessons I took from the incident:
Idempotency isn't simply a backend feature. It's a contract between the producer and consumer of an operation.
The identity of the business operation has to survive every delivery attempt.
The clean solution was to ensure that the operation receives its identity when it is created and that every synchronization attempt reuses that identity.
That is the model reflected in the current synchronization implementation: the form carries its persistent idempotency key, and synchronization reuses it.
But production systems sometimes require another consideration:
What can we safely do while an upstream problem is being corrected?
The backend still needed to protect the integrity of the booking workflow.
So I introduced a second layer of duplicate protection.
Not as a replacement for idempotency.
As a defensive backstop.
5. The Imperfect Safety Net
The backend gained an additional short-window, multi-attribute duplicate-detection mechanism.
The idea was straightforward:
If a new submission looks strongly like a recently synchronized booking, don't blindly create another business record.
This was intentionally not treated as the primary correctness mechanism.
It was a containment mechanism.
And that distinction matters.
A heuristic can have false positives.
It can have false negatives.
A legitimate booking might resemble another booking.
A retry might arrive outside the heuristic's window.
So the architecture remained:
Stable operation identity
│
▼
Idempotency
│
▼
Database uniqueness
│
▼
Concurrency handling
│
▼
Defensive duplicate backstop
The backend implementation history explicitly describes this as a mitigation for the client retry problem rather than a substitute for correct idempotency semantics.
That led to an important engineering principle for me:
A defensive heuristic should contain a failure mode, not become the system's definition of correctness.
6. Then Concurrency Exposed Another Problem
Solving duplicate delivery introduced another question.
What happens when two requests with the same idempotency key arrive at almost exactly the same time?
A common implementation starts with:
Does this key already exist?
If not:
Create it.
Sequentially, this appears safe.
Under concurrency, it isn't.
Consider two requests:
Request A Request B
│ │
▼ ▼
Check key Check key
│ │
"Not found" "Not found"
│ │
└──────────┬───────────────┘
▼
INSERT
/ \
▼ ▼
Winner Conflict
Both requests can perform the existence check before either one has committed its insert.
Then both attempt to insert the same unique value.
PostgreSQL correctly rejects one of them with a uniqueness violation.
At the database level, everything is behaving correctly.
At the application level, however, treating that as an ordinary failure would be misleading.
The losing request may simply be:
the second attempt to perform an operation that another request has just successfully created.
Diagram 3 — Concurrent Idempotency Race
Caption:
A check-then-insert sequence can race under concurrency; the database constraint becomes part of the application-level resolution.
7. Turning a Database Error Into a Business Outcome
The first implementation instinct might be:
UniqueViolation
↓
500 Internal Server Error
But that isn't necessarily what happened.
The database may simply be telling us:
"Another request got there first."
I introduced _is_idempotency_conflict() to distinguish the relevant uniqueness conflict from unrelated integrity errors.
Code Snippet 2 — Idempotency Conflict Detection
@staticmethod
def _is_idempotency_conflict(error: IntegrityError) -> bool:
return "idempotency_key" in str(
getattr(error, "orig", error)
)
Caption:
The application distinguishes an idempotency-key conflict from unrelated database integrity errors.
But detecting the conflict wasn't the final step.
The service then needs to resolve the request against the operation that won the race.
Conceptually:
INSERT
│
┌─────┴─────┐
│ │
Success Conflict
│ │
▼ ▼
Continue Read winner
│
┌─────┴─────┐
▼ ▼
Synced Pending
│ │
▼ ▼
Return result Retry later
If the winning operation is already synchronized, the losing request can receive that result.
If the winning operation is still being processed, the system can communicate a pending state rather than falsely declaring the business operation failed.
This changed how I think about database exceptions.
A database exception is sometimes evidence of a state transition elsewhere, not evidence that the user's business operation failed.
8. The Lost-Response Problem
There is another reason idempotency is essential.
Suppose the server successfully creates a booking.
Then the response is lost.
Mobile Server
│ │
│──── Create booking ──────►│
│ │
│ Save ✓
│ │
│◄──── Response ────────────X
│ lost
│
│──── Retry ────────────────►│
The mobile application doesn't know whether the booking succeeded.
From its perspective:
"The request failed."
Retrying is therefore completely reasonable.
The danger is allowing that retry to create another booking.
Stable operation identity solves the uncertainty:
Retry
│
▼
Same operation ID
│
▼
Existing operation
│
▼
Return existing result
The current synchronization engine also explicitly handles duplicate/conflict responses by treating the local form as synchronized rather than repeatedly attempting to create the operation.
This illustrates a broader distributed-systems principle:
The client doesn't always need to know what happened immediately. The system needs a safe way to resolve uncertainty when the client asks again.
9. A Successful Database Write Isn't the End
Once the booking is successfully stored, another problem appears.
Other parts of the system may need to know about the operation.
Maybe an event needs to be published.
Maybe another service needs to consume it.
Maybe a notification needs to be sent.
This creates another failure boundary.
Consider:
Save booking
│
▼
PostgreSQL ✓
│
▼
Publish event
│
X
Kafka unavailable
Now the business operation exists.
But the event doesn't.
If we reverse the order, we create the opposite problem:
Publish event
│
▼
Kafka ✓
│
X
Database transaction fails
Now another system has received an event for an operation that never became durable.
We needed the database to become the durable source of truth for both the business state change and the intention to publish its event.
10. The Transactional Outbox
The solution was the transactional outbox pattern.
The business write and the corresponding outbox event are persisted within the same database transaction.
Conceptually:
┌────────────────────────────────┐
│ DB TRANSACTION │
│ │
│ Create booking │
│ + │
│ Create outbox event │
│ │
└───────────────┬────────────────┘
│
COMMIT
│
▼
Relay Worker
│
▼
Kafka
The important guarantee is:
If the business transaction commits, the intent to publish the event is also durable.
The outbox implementation explicitly inserts the event as part of the same database transaction as the business operation and leaves publication to a later relay process.
Diagram 4 — Transactional Outbox → Kafka → Notifications
Caption:
The business change and the intent to publish its event become durable together; delivery happens asynchronously.
Code Snippet 4 — Transactional Outbox Write
For publication, use the actual relevant portion of your emit_event() implementation rather than a generic pseudocode transaction.
The important source-level pattern is:
db.execute(
text("INSERT INTO outbox_events (...) VALUES (...)"),
{...}
)
with the function called within the same database transaction as the business write.
Caption:
The outbox records the intent to publish while the business transaction is still the source of truth.
Important editorial rule: don't expose the full internal schema, private topic names, internal identifiers, or production payload structure.
11. Let the Relay Worry About Delivery
Once the event is safely stored, the original booking request doesn't need to remain responsible for delivering it to Kafka.
A background relay can take over.
The architecture becomes:
Business transaction
│
▼
PostgreSQL + Outbox
│
▼
Relay worker
│
▼
Kafka
The relay can retry delivery independently.
This also introduces another concurrency question:
What happens if multiple relay workers are running?
We don't want two workers to claim the same pending event unnecessarily.
The relay uses database locking with FOR UPDATE SKIP LOCKED when claiming available outbox work.
Code Snippet 3 — Safely Claiming Outbox Events
Show the relevant query portion:
.with_for_update(skip_locked=True)
Caption:
FOR UPDATE SKIP LOCKED allows multiple workers to claim available events without waiting on rows already locked by another worker.
Again, the interesting part isn't the syntax itself.
It's the reason it exists.
The database is being used not just as storage, but as part of the coordination mechanism between concurrent workers.
12. Then We Found Something That Wasn't Broken
At this point, the system was getting much better at processing operations reliably.
But there was another question:
Who knows that the operation succeeded?
An offline hire could synchronize successfully.
The database could contain the booking.
The event could be persisted.
The downstream processing could work.
And yet the human workflow could still be silent.
There was no guarantee that everyone who needed to know would immediately receive useful confirmation.
This wasn't an idempotency bug.
It wasn't a database bug.
It was a workflow visibility gap.
Technical correctness and operational correctness aren't always the same thing.
13. Making Successful Synchronization Visible
The synchronization flow was extended with a notification path for successfully synchronized offline hires.
A successful synchronization can now lead to downstream communication such as:
- SMS to the farmer;
- in-app notification;
- group push notification;
- admin-facing notification;
- relevant dashboard cache invalidation.
The notification implementation also treats those downstream actions as best-effort consequences rather than allowing a notification failure to invalidate the underlying business operation.
That distinction matters.
We don't want:
SMS provider unavailable
↓
Booking failed
The booking already succeeded.
The notification is another operation.
So the architecture becomes:
Booking
│
▼
Commit
│
▼
Event
│
┌─────────┼─────────┐
▼ ▼ ▼
Farmer Agent Admin
SMS Push Dashboard
A system isn't fully successful merely because the database is correct.
The people who depend on that state need a reliable way to know what happened.
14. Testing the System Like the Network Hates You
This experience changed how I think about testing offline systems.
The conventional test looks something like:
Submit booking
↓
Receive response
↓
Booking created
That's necessary.
It isn't enough.
For an offline synchronization system, the interesting tests deliberately violate the assumptions.
What happens when connectivity disappears during synchronization?
What happens when the server processes the request but the response disappears?
What happens when the client retries?
What happens when a retry uses the wrong operation identity?
What happens when two requests arrive simultaneously?
What happens when the database commits but Kafka is unavailable?
What happens when the relay worker crashes?
What happens when notification delivery fails?
These scenarios aren't edge cases in the philosophical sense.
They are normal possibilities in distributed systems.
The architecture needs a deliberate answer for each one.
15. Build a Failure Matrix
One useful way to reason about these systems is to map failures to expected outcomes.
| Failure | Desired outcome |
|---|
| No connectivity | Operation remains available for synchronization |
| Synchronization timeout | Operation can be retried |
| Response lost after server success | Retry resolves to the existing operation |
| Same operation ID submitted again | One business effect |
| Incorrect retry identity | Defensive duplicate protection may limit damage |
| Concurrent same-key requests | One wins; the other resolves against the winner |
| Database succeeds, Kafka fails | Outbox preserves the event |
| Relay worker fails | Event remains available for later processing |
| Multiple relay workers | Workers safely claim available work |
| Notification fails | Business operation remains successful |
| Dashboard data becomes stale | Relevant cache can be invalidated |
The question changes from:
"Did an error occur?"
to:
"What state should the system eventually reach after this failure?"
That is a much more useful question for distributed systems.
16. Reliability Is a Chain, Not a Feature
Looking back at the architecture, there wasn't one feature called "offline reliability."
There was a chain of mechanisms.
Mobile App
│
▼
Offline operation
│
▼
Stable operation ID
│
▼
Idempotency
│
▼
Duplicate protection
│
▼
Database constraints
│
▼
Concurrency resolution
│
▼
Transactional outbox
│
▼
Relay → Kafka
│
▼
Notifications / Admin
│
▼
Observable outcome
Each layer exists because a different assumption can fail.
Local persistence doesn't prevent duplicate delivery.
Idempotency doesn't work if operation identity changes.
Database uniqueness doesn't automatically tell the application what a conflict means.
An outbox doesn't guarantee that every downstream consumer will process an event immediately.
Notifications don't automatically guarantee that a dashboard is current.
Reliability emerges from the interaction between these mechanisms.
17. What I Would Design Differently Today
If I were designing the system from scratch today, I would make operation identity a first-class concept from the beginning.
A booking is a business operation.
An HTTP request is merely one attempt to deliver that operation.
Those should not be treated as the same thing.
ONE BOOKING
│
Operation ID
ABC123
│
┌──────────┼──────────┐
▼ ▼ ▼
Attempt 1 Attempt 2 Attempt 3
Every attempt should preserve the same identity.
I would also explicitly model synchronization as a state machine:
CREATED
│
▼
PENDING_SYNC
│
├──────────────┐
▼ ▼
SYNCED FAILED
│
▼
NOTIFIED
The exact states can differ between systems.
The principle is more important:
Don't collapse uncertainty into failure.
A timeout doesn't necessarily mean failure.
A duplicate doesn't necessarily mean failure.
A uniqueness conflict doesn't necessarily mean failure.
A temporary broker outage doesn't mean the booking itself failed.
Those distinctions should exist in the architecture.
18. I Would Treat Heuristics as Defense in Depth
The duplicate-detection backstop was useful.
But I wouldn't want it to become the primary correctness mechanism.
The hierarchy should remain:
Stable operation identity
↓
Database uniqueness
↓
Concurrency handling
↓
Business-rule safeguards
The strongest guarantees should establish correctness.
Heuristics should help contain damage when something upstream behaves unexpectedly.
That distinction is important because a system can become dangerously complicated when a workaround gradually becomes the mechanism everyone assumes is responsible for correctness.
19. I Would Make Observability Business-Aware
Infrastructure monitoring is necessary.
CPU.
Memory.
Latency.
Database health.
Queue depth.
Worker health.
But an offline synchronization system needs business-level observability too.
I'd want to know:
How many operations are waiting?
How many have failed?
How many have retried?
How many duplicate attempts were detected?
How many idempotency conflicts occurred?
How long do operations remain pending?
How many outbox events are waiting?
How many downstream notifications failed?
Those metrics answer a different question.
Not:
"Are my servers running?"
But:
"Is the business workflow actually progressing?"
That distinction becomes increasingly important as systems become more distributed.
20. Idempotency Is a Contract, Not a Checkbox
It's easy to say:
"We have idempotency."
But that statement isn't enough.
You also need to answer:
What makes two requests the same business operation?
If the answer is an operation ID, that identity needs to survive retries.
If the identity changes with every attempt, the server cannot reliably connect those requests.
This is why mobile and backend components need a shared understanding of operation identity.
The mobile application owns the logical operation.
The network merely transports attempts to synchronize it.
21. Retries Change the Semantics of an Operation
A retry isn't simply another function call.
It may represent:
- a request that never reached the server;
- a request that reached the server but timed out;
- a request that succeeded but whose response disappeared;
- an operation whose outcome is genuinely unknown.
Those scenarios can all look similar from the client.
The backend therefore needs a way to answer:
"What happened to this operation?"
rather than simply:
"What should I do with this HTTP request?"
That's why stable identity, durable state, and idempotent processing matter.
22. Exactly Once Isn't Always the Goal
Distributed systems often make people think about "exactly once."
In practice, guaranteeing that a distributed operation is attempted exactly once can be extremely difficult.
A more practical objective is:
Allow repeated delivery while keeping the business outcome correct.
The network can deliver an operation more than once.
The client can retry.
A worker can restart.
An event can be processed again.
The architecture should still converge toward the correct business state.
This is particularly important in offline-first applications because intermittent connectivity naturally creates uncertainty around delivery.
23. Failure Needs Business Meaning
One of the strongest lessons from this work was that technical errors and business failures aren't always the same thing.
Consider:
Timeout
That doesn't necessarily mean:
Booking failed.
Consider:
Unique constraint violation
That doesn't necessarily mean:
Booking failed.
Consider:
Kafka unavailable
That doesn't necessarily mean:
Booking failed.
Consider:
Notification provider unavailable
That definitely doesn't necessarily mean:
Booking failed.
A robust system needs to distinguish:
transport failure
processing failure
business failure
and
uncertainty about the outcome.
When all four become:
500 Internal Server Error
the system loses important information.
24. Offline-First Is a Distributed-Systems Problem
This is probably the biggest lesson I took away from the project.
Offline-first is often described as a mobile development technique:
Store data locally and synchronize it later.
That description is technically true.
It doesn't capture the difficulty of doing it reliably.
Once synchronization becomes important, you quickly encounter:
Unreliable connectivity
↓
Uncertain delivery
↓
Retries
↓
Idempotency
↓
Concurrency
↓
Transactions
↓
Asynchronous events
↓
Recovery
↓
Eventual consistency
The device and server can temporarily disagree.
Two requests can arrive simultaneously.
A response can disappear.
A worker can crash.
A message can be delayed.
A downstream service can be unavailable.
These are distributed-systems problems.
The fact that one participant happens to be a mobile phone doesn't make them any less real.
25. Architecture Has to Account for the Human Environment
Perhaps the most important lesson wasn't technical.
It was architectural empathy.
A field agent doesn't experience:
"HTTP request timeout."
They experience:
"I don't know whether this booking went through."
A farmer doesn't experience:
"Eventual consistency."
They experience:
"Will the tractor actually come?"
An administrator doesn't experience:
"Cache invalidation."
They experience:
"Is the information I'm looking at accurate?"
Good architecture therefore cannot stop at the API boundary.
It has to account for the human workflow that the software exists to support.
26. What Reliability Means to Me Now
The experience changed the questions I ask when designing backend systems.
I don't only ask:
Does the API work?
I ask:
What happens if the network disappears?
What happens if the request succeeds but the response disappears?
What happens if the client retries?
What happens if the retry doesn't preserve operation identity?
What happens if two requests arrive simultaneously?
What happens if the database commits but the message broker is unavailable?
What happens if a worker crashes?
What happens if a notification fails?
What happens if different components temporarily disagree?
And perhaps most importantly:
What happens when the system doesn't know whether something succeeded?
Those questions naturally lead toward:
- stable operation identities;
- idempotency;
- durable synchronization state;
- database constraints;
- concurrency-aware conflict handling;
- transactional outboxes;
- asynchronous recovery;
- explicit failure states;
- observability.
Conclusion: Design for the World as It Is
When we design software, it's natural to imagine the happy path:
User
↓
Request
↓
Server
↓
Database
↓
Response
Real systems rarely behave that cleanly.
The network disappears.
Requests time out.
Responses get lost.
Clients retry.
Workers crash.
Messages are delayed.
Two requests arrive at almost exactly the same time.
Dependencies become temporarily unavailable.
And users press the button again because they don't know what happened.
Reliable architecture doesn't eliminate those realities.
It gives them somewhere safe to go.
A timeout becomes a retry.
A duplicate request becomes an existing result.
A concurrency conflict becomes a resolvable state.
A broker outage becomes a pending outbox event.
A notification failure doesn't invalidate a successful booking.
An offline operation eventually converges with the backend's view of reality.
That is what building software for low-connectivity environments taught me:
The network may be unreliable. The system's understanding of the business outcome cannot be.
About the Author
Betini Akarandut is a Backend & Cloud Engineer focused on backend architecture, distributed systems, offline-first applications, cloud infrastructure, and DevOps.
His work involves building and operating production systems across technology-driven environments, with a particular interest in designing reliable software for situations where connectivity, infrastructure, and other assumptions cannot always be taken for granted.
He writes about lessons from building real-world software systems—especially the engineering problems that only become visible when architecture meets production reality.
