When Retries Become the Outage
At 2:17 AM, a payment service started returning errors. Not many. Just enough to attract attention. The dashboards didn't look alarming. CPU utilization was stable. Memory consumption was within normal limits. Infrastructure health checks were passing. If you had opened the monitoring console at that moment, you probably would have assumed the issue was temporary and would resolve itself within a few minutes.
The engineering team began investigating. Meanwhile, the rest of the system was already reacting. API clients started resubmitting requests. Lambda functions retried failed executions. Messages that couldn't be processed returned to queues. Downstream services attempted the same operations again, hoping the next attempt would succeed.
None of these actions were wrong. In fact, every component was behaving exactly as it had been designed to behave. And that was the problem.
What began as a relatively small service degradation gradually evolved into a much larger incident. The original failure affected a limited number of requests. The recovery mechanisms multiplied that traffic until the struggling service was forced to handle far more load than it had seen before the outage began.
The system wasn't collapsing because of the initial error anymore. It was collapsing because everybody was trying to help.
Why Retries Feel Like the Right Answer
Retries are one of the first resilience patterns most engineers learn, and for good reason.
Distributed systems operate in an imperfect world. Networks experience brief interruptions. Services become temporarily unavailable. Databases occasionally take longer than expected to respond. A request that fails now may succeed a few seconds later without any intervention.
Because of this, retries often improve reliability. A customer never notices the timeout. The application recovers automatically. Everybody wins. The trouble starts when we focus on the success story and forget to think about scale. A single retry seems harmless. Ten thousand retries are something else entirely.
Imagine a service receiving 10,000 requests during a busy period. Suddenly, a dependency starts responding slowly. Requests begin timing out. Each client retries three times before giving up.
Without realizing it, the system has transformed 10,000 requests into 40,000. That additional traffic creates more pressure on the dependency. More pressure leads to higher latency. Higher latency generates more failures. More failures trigger more retries. What looked like a recovery mechanism has become a feedback loop. And feedback loops are where distributed systems become dangerous.
The Real Problem Isn't Failure
One of the hardest lessons in cloud architecture is that systems rarely fail where the problem starts.
- A database slows down.
- An API begins timing out.
- A message consumer falls behind.
Those events are usually manageable.
The real challenge comes from the chain reaction that follows. Every service makes a reasonable decision based on its local perspective. Every team implements protections intended to improve reliability. Every retry policy makes sense when viewed in isolation.
Yet when an incident occurs, those independent decisions begin interacting with each other. The result is often a level of complexity that nobody anticipated during design reviews. I've seen incidents where the root cause was resolved within minutes, but the platform continued struggling for hours because retries, reprocessing, and recovery workflows kept generating load long after the original issue had disappeared.
The failure wasn't the hardest part. Recovering from the recovery was.
When AWS Does Exactly What You Asked It To Do
AWS services are designed to help us build resilient systems, which is why many of them include retry mechanisms by default.
Take Lambda and SQS. When a Lambda function cannot successfully process a message, that message doesn't simply disappear. It becomes available again and can be processed later. This behavior protects against transient failures and prevents data loss. Most of the time, that's exactly what we want.
However, if the underlying problem is persistent—a broken dependency, an invalid payload, or a misconfigured service—the same message may be processed repeatedly while consuming compute resources and generating additional pressure on downstream systems. The same principle appears in event-driven architectures.
Services such as EventBridge improve reliability by attempting redelivery when consumers fail. That's an incredibly valuable capability. However, it also means that consumers must be designed with duplicate events in mind. Retries improve resilience. But resilience is never free.
Every retry consumes resources, generates traffic, and increases complexity. The question is not whether retries are good or bad. The question is whether the system can absorb the consequences when retries occur at scale.
Designing Retries That Help
Over the years, I've become less interested in whether a system retries and more interested in how it retries. The difference matters. One of the simplest improvements is exponential backoff.
Instead of retrying every second, the delay increases after each attempt. A request may wait one second, then two, then four, then eight. This gives dependencies time to recover and reduces the likelihood of overwhelming a struggling service.
Even then, another challenge remains. Imagine a million clients all deciding to retry after exactly eight seconds. Eight seconds later, they arrive together. The system experiences another traffic spike.
This is why mature systems often introduce jitter, adding a small amount of randomness to retry timing. The overall volume remains similar, but the requests become distributed over time rather than arriving in synchronized waves.
Another valuable pattern is the circuit breaker. When a dependency is clearly unavailable, continuing to send requests often provides little benefit. A circuit breaker temporarily stops traffic, allowing the dependency to recover while protecting the rest of the system from unnecessary load.
And sometimes, the most responsible decision is not to retry immediately at all.
Sometimes the correct answer is to move the failed message to a Dead Letter Queue, investigate the cause, and replay it later under controlled conditions. That approach may feel less aggressive than continuous retries, but it often prevents a localized issue from becoming a platform-wide incident.
A Different Way to Think About Resilience
Early in my career, I thought resilience was mostly about making systems persistent.
- If something failed, try again.
- If it failed again, try harder.
Experience changed that perspective.
Today, I think resilience is often about restraint. It's about understanding that every recovery mechanism introduces new behavior into the system. It's about recognizing that retries are not free. It's about accepting that the healthiest response to failure is not always immediate action. Sometimes the best thing a system can do is wait. Sometimes it's stop. Sometimes it's fail gracefully.
The most resilient cloud architectures aren't the ones that retry forever. They're the ones that understand the difference between recovering from a problem and amplifying it. Because in distributed systems, the outage doesn't always begin with the first failure. Sometimes it begins with the second attempt.