Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Question

Why publishing an event is only the beginning of building a reliable distributed system.

Building an event-driven application with Spring Boot is relatively straightforward.

You create a Kafka producer, publish an event, implement a consumer, and process the message.

Everything works beautifully during development.

Then production introduces the situations your happy-path tests never covered.

What happens when the database transaction commits, but Kafka is unavailable? What happens when the consumer processes an event successfully but crashes before acknowledging it? What happens when two application instances attempt to process the same message?

These aren't unusual edge cases. They're fundamental challenges of distributed systems.

And solving them requires more than adding a retry annotation.

Let's explore the reliability boundaries behind a production-ready event-driven architecture.

1. The dual-write problem

Imagine a payment service that must perform two operations:

1. Save a payment in PostgreSQL.
2. Publish a PaymentCompleted event to Kafka.

A straightforward implementation might look like this:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);
    kafkaTemplate.send(
        "payment.completed",
        new PaymentCompleted(payment.getId())
    );
}

The code looks reasonable, but there's a serious problem.

The database transaction and Kafka publication do not share a single atomic transaction.

Several outcomes are possible:

• The database commits, but the event is never published.
• Kafka accepts the event, but the database transaction rolls back.
• Publication times out, leaving the application uncertain whether Kafka received the event.

Adding retries doesn't eliminate the fundamental inconsistency.

The problem is not whether Kafka is reliable. The problem is coordinating two independent systems.

The transactional outbox

The transactional outbox pattern changes the transaction boundary.

Instead of publishing directly to Kafka, the application writes an outbox record within the same database transaction as the business operation.

Payment Service
      |
      v
Database Transaction
      |
      +-- Save Payment
      |
      +-- Save Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
    Kafka

Either the payment and outbox event are committed together, or neither is committed.

A separate publisher retrieves pending outbox records and delivers them to Kafka.

This solves the atomicity problem between the business state and the durable intent to publish.

It does not guarantee that an event will be published exactly once.

That distinction matters.

2. Why the outbox still needs idempotency

Consider this sequence:

1. The outbox publisher sends an event to Kafka.
2. Kafka successfully accepts it.
3. The application crashes before marking the outbox record as published.
4. The publisher restarts and sends the event again.

Kafka may now contain two copies of the same logical event.

This is a normal consequence of at-least-once delivery.

The correct response is not to assume duplicates can always be prevented.

Instead, design consumers that can safely tolerate them.

3. The transactional inbox

An inbox provides a durable record of incoming events that have already been processed.

Each event should carry a stable identifier, such as an event ID generated when the logical event is created.

When the consumer receives a message, it checks whether that event has already been processed.

A simplified model looks like this:

Kafka Event
     |
     v
Consumer
     |
     v
Database Transaction
     |
     +-- Claim Event ID in Inbox
     |
     +-- Apply Business Changes
     |
     +-- Mark Inbox Completed
     |
     v
   Commit

The important part is the transaction boundary.

The inbox record and the business changes must participate in the same transaction when using the same transactional database.

Otherwise, a consumer can record an event as completed even though its business operation failed.

A subtle but important case

Suppose a consumer receives PaymentCompleted, updates an invoice, and produces InvoicePaid.

Now the reliability boundary becomes:

Inbox + Business State + Outbox

All three should be committed atomically.

Incoming Event
      |
      v
Database Transaction
      |
      +-- Inbox Claim
      |
      +-- Update Invoice
      |
      +-- Create Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
Outgoing Event

This is one of the most useful patterns for building dependable event-driven workflows.

The inbox protects the incoming side. The outbox protects the outgoing side.

Neither eliminates the need for careful business-level idempotency, especially when external side effects are involved.

4. Retries are a lifecycle, not just an annotation

Retries are often introduced with a simple configuration:

@Retryable(
    retryFor = TransientException.class,
    maxAttempts = 3
)

For short-lived operations, that can be perfectly reasonable.

But what happens when a dependency is unavailable for 30 minutes? What happens when the application restarts? What happens when an event requires operator intervention?

An in-memory retry loop cannot reliably manage every long-running failure scenario.

For durable asynchronous processing, retries should be modeled as persistent state.

A typical lifecycle might look like this:

PENDING
   |
   v
PROCESSING
   |
   +---- Success ----> COMPLETED
   |
   +---- Retryable failure
   |           |
   |           v
   |      RETRY_SCHEDULED
   |           |
   |           v
   |        PENDING
   |
   +---- Terminal failure
               |
               v
             FAILED

Each attempt can record useful information:

• Attempt count
• Next eligible execution time
• Last failure category
• Processing status
• Relevant timestamps

Exponential backoff with jitter helps avoid repeatedly overwhelming an unhealthy dependency.

But retry policies should distinguish transient failures from permanent ones.

Retrying an invalid business request a thousand times won't make it valid.

5. Multi-instance processing changes everything

A background publisher may work perfectly when there's only one application instance.

Then the service is scaled to three pods.

All three publishers query the same pending records.

Without coordination, multiple instances may attempt to publish or process the same work concurrently.

One common PostgreSQL technique is:

SELECT id
FROM outbox_event
WHERE status = 'PENDING'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED allows concurrent workers to skip rows already locked by another transaction.

This can improve coordination between workers, but it's not a complete delivery protocol.

The design must still consider:

• How work is claimed and how long claims remain valid
• Transaction duration and lock ownership
• Recovery after a worker crashes
• Retry scheduling and fairness
• Ordering requirements
• Duplicate delivery

For example, holding database locks across a slow network call can increase contention and create unnecessarily long transactions.

A robust implementation may use short claim transactions, explicit processing ownership, and recovery mechanisms rather than keeping database transactions open while publishing.

Concurrency control and delivery guarantees are related, but they are not interchangeable.

6. Exactly-once is not an end-to-end promise

It's tempting to describe an event-driven system as exactly-once.

But exactly-once semantics depend on the boundaries being discussed.

Kafka transactions can provide guarantees for supported Kafka operations. A PostgreSQL transaction can atomically update records within PostgreSQL.

Neither automatically makes a database update, Kafka publication, HTTP request, and external payment operation one globally atomic action.

A more useful engineering goal is:

At-least-once delivery with effectively-once business outcomes wherever possible.

That typically requires durable event identities, transactional inbox processing, idempotent business operations, and explicit handling of external side effects.

For payment systems, this distinction is particularly important.

Processing the same event twice must not mean charging a customer twice.

7. Observability is part of reliability

A system that retries failures silently can appear healthy while accumulating thousands of unprocessed events.

Operational visibility should answer questions such as:

• How many events are pending?
• How old is the oldest unpublished event?
• How many retries are scheduled?
• Which events are permanently failing?
• Are workers making progress?
• Are duplicates being detected?
• Is event processing latency increasing?

Metrics, structured logs, tracing, and operator-facing failure information are not optional extras for a production system.

They help distinguish a temporary delay from a workflow that has stopped making progress.

A reliable system should make failure visible and recovery understandable.

8. Putting these patterns together

A practical architecture combines several responsibilities:

Business Operation
       |
       v
Transactional Outbox
       |
       v
Durable Publisher
       |
       v
Message Broker
       |
       v
Transactional Inbox
       |
       v
Business Processing
       |
       +-- State Changes
       |
       +-- Optional New Outbox Events
       |
       v
Commit

Around that flow, we need retry policies, concurrency coordination, idempotency, recovery, and observability.

Each component solves a different failure mode.

The outbox prevents losing the intent to publish when business data commits.

The inbox protects consumers from repeated processing.

Retries help transient failures recover.

Concurrency controls coordinate multiple workers.

Observability makes the system operable.

None of these patterns should be treated as magic.

Their value comes from understanding exactly which failure boundary each one protects.

9. Why I built NERV Event

After working with event-driven enterprise applications, I wanted these reliability concerns to be reusable rather than repeatedly implemented in individual services.

That led to NERV Event, an open-source Java and Spring Boot library focused on patterns such as transactional outbox, inbox processing, retries, and idempotency.

The goal isn't to hide distributed-systems complexity behind annotations.

It's to provide reusable infrastructure while keeping the reliability model explicit.

GitHub: https://github.com/czetsuyatech/nerv-event

The library is one possible implementation of these ideas. The architectural principles are useful regardless of which framework or library you choose.

Final thoughts

Publishing an event is easy.

Building a system that behaves predictably when databases, brokers, application instances, and downstream dependencies fail is much harder.

Reliable event-driven architecture begins with a simple question:

What happens if the application crashes at this exact point?

Ask that question at every transaction boundary, every message acknowledgment, and every external call.

The answers will reveal where durability, idempotency, retries, and recovery are actually needed.

I'd be interested to hear how other teams approach this: Do you use a transactional inbox alongside your outbox, or do you handle consumer idempotency directly in your business tables?

---

Originally published on CzetsuyaTech. Explore more articles on Java, Spring Boot, distributed systems, and software architecture there.