Retries are one of the easiest resilience patterns to add and one of the easiest to get wrong.
The idea is sound. Transient failures happen. Networks drop packets. Load balancers close idle connections. A database primary moves. A single failed call should not always fail the whole operation.
But retries also create traffic.
During a healthy moment, that extra traffic is small. During an outage, it can be enormous. Every timeout becomes multiple calls. Every slow dependency receives new requests from callers that have already given up on older ones. The dependency is not just handling user traffic anymore. It is handling user traffic plus the system's anxiety about user traffic.
That is how retries turn a partial failure into a larger one.
Retry Storms
A retry storm starts when many clients observe the same failure at roughly the same time.
Without jitter, they retry at roughly the same time too.
If the dependency is overloaded, those synchronized retries make recovery harder. The service needs quiet time to drain queues, rebuild caches, elect leaders, or scale out. Instead it receives a second wave of traffic. Then a third.
The clients think they are being persistent. The dependency experiences persistence as pressure.
Timeouts Come First
Retries without timeouts are not retries. They are a way to accumulate stuck work.
Every retry policy needs a per-attempt timeout and an overall deadline. The per-attempt timeout prevents one slow call from consuming the entire budget. The overall deadline prevents retries from continuing after the user or caller no longer cares.
In Go, that usually means retries must be tied to context.Context, not a loop with time.Sleep.
func callWithRetry(ctx context.Context, attempts int) error {
for i := 0; i < attempts; i++ {
if err := ctx.Err(); err != nil {
return err
}
attemptCtx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
err := callDependency(attemptCtx)
cancel()
if err == nil {
return nil
}
if !isRetriable(err) {
return err
}
if !sleepWithJitter(ctx, i) {
return ctx.Err()
}
}
return errors.New("dependency failed after retries")
}
The exact helper functions matter less than the shape: bounded attempts, bounded time, cancellation-aware sleep, and retry only for errors that are actually retriable.
Retry Budgets
A retry budget limits how much extra traffic retries may create.
For example, a client might allow retries to add no more than 10 percent extra request volume over a rolling window. When the budget is exhausted, the client stops retrying and returns errors faster.
That sounds harsh, but it protects the dependency and the caller. If the system is already failing, unlimited retries mostly buy longer queues and slower recovery.
Retries should be treated as a shared resource.
Not Every Error Is Retriable
Retrying a deterministic failure is just load generation.
Good retry policies distinguish between:
- connection resets
- timeouts
- rate limits
- temporary 5xx responses
- validation failures
- authentication failures
- not found responses
- conflicts caused by stale state
Some of those may deserve a retry. Many do not. A 400 response should usually not be retried. A 401 should not become three 401s. A permanent validation error is not going to become valid because the client waited 100 milliseconds.
The retry policy should be closer to the protocol semantics than to the transport library.
Retries Need Backpressure
Retries and backpressure belong together.
If the dependency returns 429 Too Many Requests or a clear overload signal, callers should respect it. If there is a Retry-After header, callers should use it. If the local queue is already full, the client should stop adding retry work to it.
The purpose of a retry is to survive a transient failure, not to argue with a saturated system.
The Lesson
Retries are useful when they are humble.
They should have a deadline. They should have jitter. They should have a budget. They should understand which errors are worth retrying. They should stop when the caller no longer needs the answer. They should listen when the dependency says "slow down."
The question is not "should we retry?"
The question is:
How much extra pressure are we allowed to create while trying to recover?