A customer clicks Place order. Your API returns success. The order row is in the database. But the service that processes orders never sees an event.
This is not a hypothetical edge case invented by Kafka. It is a failure window in a common Spring Boot pattern: save to a database, then publish to a broker as a second operation. I found the pattern while reviewing my own event-driven order processing capstone. This is a code review of a learning project, not a claim that a production outage occurred.
The lesson applies well beyond orders. If your API writes data and then emits an event for fraud checks, notifications, inventory, or analytics, a successful HTTP response can tell less of the story than you think.
The four lines that made me stop
My order service saves a new order, then calls a producer:
CustomerOrder order = orderRepository.save(new CustomerOrder(...));
producer.publish(new OrderEvent(order.getId(), ...));
return toResponse(order);
The producer uses Spring Kafka like this:
public void publish(OrderEvent event) {
kafkaTemplate.send("order-events", String.valueOf(event.orderId()), event);
}
That send returns a future. In this implementation, the future is ignored. The method can move on to the HTTP response without observing whether Kafka acknowledged the record. More fundamentally, the database write and Kafka publish are two separate operations. A process crash can occur between them even if you wait for the send result.
The code is perfectly useful for demonstrating an event-driven flow. It is not enough by itself to promise that every saved order will produce a durable event.
Where the gap opens
There are at least two ways to lose the intended handoff:
- Crash after the database commit, before the send. The order exists, but no event was requested.
- Asynchronous send failure after the method proceeds. The producer future completes with an error, but this code does not inspect it.
The reverse uncertainty also matters: Kafka may accept an event while the client never receives the HTTP response. A retry from the client can create another order unless the API has a separate request-idempotency rule.
I already had retries and a dead-letter topic on the consumer side of the capstone. Those help when an event reaches Kafka and processing fails. They cannot recover an event that was never published. This distinction is easy to miss when a demo looks healthy end to end.
First, choose what a successful response means
An API contract should say what has actually happened. For an asynchronous workflow, a successful create response may mean the order and the intent to process it have been stored. It should not imply that downstream processing is complete.
A practical state model might be:
CREATED → PROCESSING → APPROVED or REJECTED
↘ NEEDS_ATTENTION
A GET endpoint lets the caller check the current state. An order stuck in CREATED for too long should be visible to monitoring and operators, not quietly forgotten.
The transactional outbox fix
A transactional outbox makes the event intent part of the same database transaction as the order. The API writes two rows: the business record and an outbox record containing an event ID, aggregate ID, event type, and payload. Either both rows commit or neither does.
A separate relay publishes committed outbox rows to Kafka. That relay can poll the table or use change-data capture. Debezium's outbox event router is one established implementation of the latter approach.
The core application code becomes conceptually:
@Transactional
public OrderResponse createOrder(CreateOrderRequest request) {
CustomerOrder order = orderRepository.save(toEntity(request));
outboxRepository.save(OutboxEvent.orderCreated(order));
return toResponse(order);
}
This is illustrative replacement code, not a feature already implemented in my capstone. The important part is the transaction boundary around both inserts. The relay is responsible for eventual delivery after commit.
The outbox does not turn the whole system into exactly-once delivery. A relay may publish an event and crash before recording that it published it. It will publish again after restart. Consumers therefore need to handle duplicates, usually by recording a stable event ID and making the business effect idempotent.
Two IDs solve two different retry problems
It helps to keep these IDs distinct:
- Request ID: supplied or assigned at the API edge so a repeated Place order request does not create a second order.
- Event ID: stored with the outbox record so repeated event delivery does not apply the same downstream action twice.
Kafka producer idempotence is useful for producer retries, but it does not make a database transaction and a broker publish atomic. Nor does it replace consumer-side deduplication across relay restarts. Each layer protects a different boundary.
The test I would add before calling this production-ready
I would make failure injection part of the acceptance test:
- Create an order while the broker is unavailable.
- Confirm that the order and its outbox event are committed together.
- Restore the broker and confirm that the relay publishes the event.
- Force the relay to retry, then confirm the consumer applies the event only once.
- Check that the order eventually leaves CREATED, or raises an alert if it does not.
In the current direct-send design, a broker failure or crash can leave an order without a durable event intent. The test should make that gap visible. In the outbox design, the same test checks the recovery path rather than just the happy path.
What I would monitor
A reliable event-driven API needs signals beyond HTTP success rate:
- Age of the oldest unpublished outbox row
- Number of publish failures and relay retries
- Consumer lag and dead-letter count
- Duplicate event count
- Orders stuck in CREATED or PROCESSING
- Time from order creation to final state
These metrics turn “the dashboard never updated” into an observable state with an owner and a recovery path.
The takeaway
My capstone taught me a useful production lesson: an event-driven workflow is only as reliable as the boundary between its database and its broker. Consumer retries are valuable, but they start too late to repair a missing publish. Save the business record and the event intent together, deliver at least once, deduplicate downstream, and monitor the gap between creation and completion.
If you have built a database-to-Kafka flow, where do you persist the intent to publish? I would be interested to hear how your team tests that failure window.
References