Question

Your LLM App Works Until It Doesn't — Handling Rate Limits, Timeouts, and Overloaded APIs in Production

Your LLM feature worked flawlessly for weeks. Then you got your first traffic spike — a newsletter mention, a busy Monday — and the logs filled with 429 rate_limit_error. Your quick fix, a while loop that retries immediately, turned one over-limit request into fifty, which kept you pinned over the limit, which took the feature down harder. Meanwhile a few requests didn't fail at all: they just hung for two minutes and died at your load balancer's timeout, and the user watched a spinner the whole time.

None of this is bad luck. An LLM API is a remote dependency with unusually long response times, strict throughput quotas, and real capacity fluctuations — and most LLM apps ship with none of the resilience patterns we'd consider mandatory for any other dependency like that. Let's fix it layer by layer, with working Python along the way.

First, know your enemies: the four failure classes

Everything below is a response to one of four distinct failures, and they need different treatment:

1. Rate limits (HTTP 429) — you exceeded your requests-per-minute or tokens-per-minute quota. The API is healthy; you are over budget. Providers meter these with token-bucket-style algorithms, and Anthropic's rate limit docs describe separate buckets for requests, input tokens, and output tokens — you can be under one and over another.
2. Overload and server errors (HTTP 529 / 500-class) — the provider is degraded. Anthropic signals capacity pressure with a dedicated 529 overloaded_error. Nothing you did caused it; nothing you send fixes it. Only time does.
3. Timeouts — the connection died, or generation is slower than some timeout between you and the model (yours, a proxy's, a load balancer's). With responses that can legitimately take minutes for long outputs, this class is far more common with LLMs than with typical APIs.
4. Ambiguous failures — the request timed out or errored after the provider may have processed it. This one quietly matters most: it's the difference between "retry safely" and "charge the customer twice."

The cardinal rule that falls out of this taxonomy: retrying is for transient failures, and only with backoff. A 400 (malformed request) will fail identically on attempt ten; a 429 retried instantly is self-sabotage. Match the response to the class.

Layer 1 — Retries with exponential backoff and jitter

The foundation, and the place where most hand-rolled implementations go wrong twice: they retry too fast, and they retry in sync. If your process fires 100 concurrent requests and all get a 429, all 100 retrying after exactly one second produces a synchronized second wave — a thundering herd that keeps you rate-limited forever. The fix is exponential backoff (double the wait each attempt) plus jitter (randomize it, so retries decorrelate). AWS's classic analysis of backoff and jitter showed that adding full jitter dramatically reduces total work and time-to-success under contention — it's not a nicety, it's the mechanism.

Good news: the official SDKs already do the basics. The Anthropic Python SDK retries connection errors, 408, 409, 429 and 500+ automatically with backoff — two retries by default, configurable:

import anthropic

client = anthropic.Anthropic(
    max_retries=4,        # SDK default is 2; raise it for background/batch work
    timeout=60.0,         # seconds; don't let requests hang forever
)

For everything the SDK doesn't cover — your own wrapping logic, other providers, non-SDK calls — Tenacity gives you the same pattern declaratively:

from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception
import anthropic

def is_retryable(e: BaseException) -> bool:
    # Retry transient classes only: rate limits, overload, timeouts.
    if isinstance(e, (anthropic.RateLimitError, anthropic.InternalServerError,
                      anthropic.APIConnectionError, anthropic.APITimeoutError)):
        return True
    return False  # 400s, auth errors, invalid requests: fail fast, fix the code

@retry(
    retry=retry_if_exception(is_retryable),
    wait=wait_random_exponential(multiplier=1, max=60),  # exp backoff + full jitter
    stop=stop_after_attempt(5),
)
def ask_model(prompt: str) -> str:
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return msg.content[0].text

Two details worth stealing from that snippet: the allowlist of retryable exceptions (a retry loop that retries BadRequestError just burns five attempts on a bug), and the cap on backoff (uncapped exponential growth turns attempt six into a five-minute nap).

One more thing while you're here: respect what the API tells you. A 429 often carries a Retry-After header, and providers expose live quota state in response headers (Anthropic: anthropic-ratelimit-requests-remaining, token equivalents, and reset timestamps). Reading "you have 2% of your token budget left" and slowing down before the 429 beats any recovery strategy — client-side throttling against those headers is the cheapest resilience you'll ever add.

Layer 2 — Timeouts and streaming: stop hanging, start flowing

Long LLM responses create a nasty squeeze: a genuinely healthy request generating 4,000 tokens can take longer than the default timeout of some proxy between you and the user. You then kill a request that was working, retry it, and pay for both.

Two moves resolve the squeeze:

Set explicit timeouts everywhere — client, server, gateway — and make the client's the shortest, so you control the failure instead of discovering it via a 504 page. Budget for realistic generation time, not typical API time; 60 seconds is a floor for long generations, not a ceiling.

Stream by default for anything user-facing. Streaming (server-sent events in most LLM APIs) changes the resilience math twice over. First, time-to-first-token becomes your effective health signal — a stream that starts within two seconds is alive, no matter how long the full answer takes. Second, the user experience of failure transforms: a connection that drops at token 3,000 of a streamed answer left the user with 3,000 useful tokens and a "retry" button, instead of 90 seconds of spinner followed by nothing. The SDKs make this nearly free:

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=2048,
    messages=[{"role": "user", "content": prompt}],
) as stream:
    for text in stream.text_stream:
        emit_to_client(text)   # user sees progress from the first second

Wiring streams through your own backend (SSE out to the browser, partial-result handling, resume-on-drop) is unglamorous plumbing — and it's exactly the kind of production detail that separates demos from products, which is why a serious course on advanced LLM integration in production treats streaming and failure handling as core curriculum rather than an appendix.

Layer 3 — The fallback chain: degrade, don't die

Retries handle a bad moment. They do nothing for a bad hour — a provider incident, a sustained capacity crunch. For that you need a fallback chain: an ordered list of "what I'd rather serve than an error page."

For most apps the chain looks like this:

1. Primary model — your quality default.
2. Faster/cheaper same-provider model — e.g., falling back from your top-tier model to a fast one like Claude Haiku 4.5. Latency-sensitive features often lose nothing users notice, and the small model's quota is a separate budget from the big one's.
3. Cached or canned response — for repeated questions, a semantic cache hit beats a fresh generation at infinite reliability and zero cost; failing that, an honest "AI features are briefly degraded" beats a stack trace.

FALLBACK_CHAIN = ["claude-sonnet-5", "claude-haiku-4-5-20251001"]

def resilient_ask(prompt: str) -> str:
    for model in FALLBACK_CHAIN:
        try:
            return ask_with_retries(prompt, model=model)   # Layer-1 wrapper, per model
        except (anthropic.RateLimitError, anthropic.InternalServerError,
                anthropic.APITimeoutError):
            continue                                        # next rung down
    return cached_or_apology(prompt)                        # never raise to the user

Two honest caveats. Different models produce different outputs — if your prompts are tuned to one model's behavior, validate the fallback rung against your test set before the incident, not during it. And cross-provider fallbacks (a whole second vendor) buy real independence but double your integration surface, your prompt-tuning work, and your compliance review; most teams are better served by a same-provider chain plus a cache than by a half-tested second vendor. Systematically checking that every rung of the chain still meets your quality bar is precisely what LLM evaluation and testing discipline is for — an untested fallback is just a slower way to fail.

Layer 4 — Circuit breakers and queues: protect yourself from your own retries

There's a failure mode where Layers 1–3 make things worse: the provider is hard-down, and every user request now ties up a worker for five backoff attempts × up to 60 seconds before falling through the chain. Your retry budget, multiplied by your traffic, has become a resource leak — threads, connections, and your own latency SLO all burning on requests you already know will fail.

The circuit breaker pattern fixes this with three states: closed (normal; count failures), open (failure rate tripped a threshold — fail or fall back instantly, no API call at all), and half-open (after a cooldown, let a few probes through; success closes the circuit). The point is speed of failure: while the circuit is open, users get the cache/apology response in a millisecond instead of after a full retry gauntlet, and the struggling provider gets zero additional load from you — which, in aggregate across all their customers, is part of how providers recover at all. Libraries like purgatory or aiobreaker implement this in a few lines; the concept matters more than the dependency.

The second structural tool: move non-interactive work off the request path entirely. Anything that doesn't need an answer in seconds — enrichment, summarization jobs, embedding backfills — belongs in a queue consumed by workers at a rate you set below your quota. A queue converts "burst of 500 requests → 429 storm" into "500 jobs drain over four minutes," absorbs provider downtime invisibly (jobs wait; users never see it), and gives you a natural place for the one thing everyone forgets: idempotency keys, so the job that timed out ambiguously in failure class #4 can be retried without double-charging, double-sending, or double-writing. If your stack has batch API support, non-urgent work often gets a large discount there too — resilience and cost optimization from the same refactor.

This is the point where "calling an LLM API" quietly becomes real backend architecture — queues, workers, breakers, budgets — the same skill set covered end-to-end in building and shipping a production AI SaaS, because no production AI product escapes it.

Layer 5 — Observability: you can't harden what you can't see

Every layer above has a knob, and without measurement you're turning knobs blind. The minimum viable dashboard for an LLM dependency:

• Error rate by class — 429 vs. 5xx/529 vs. timeout, separately. A 429 spike means raise limits, throttle, or queue; a 529 spike means fallback and wait. One merged "error rate" graph hides the decision.
• Latency percentiles, including time-to-first-token for streams. p50 will lie to you; the p95/p99 tail is where timeouts actually live.
• Retry and fallback counters — how often each rung of the chain actually fires. A fallback that silently serves 30% of traffic is an incident you haven't noticed yet.
• Quota headroom — consumption against your rate limits, from those response headers, before the 429s start.
• Alerting on the breaker state. An open circuit is a page, not a log line.

Treating an LLM provider like any other tier-one dependency — SLOs, dashboards, runbooks — is the mindset shift, and it's the same operational thinking that AI-for-DevOps and SRE practice applies from the other direction.

The checklist

Pin this next to your deploy button:

• [ ] Retries: exponential backoff with jitter, retryable-error allowlist, capped attempts
• [ ] Retry-After and rate-limit headers read and respected; client-side throttle before the cliff
• [ ] Explicit timeouts at every hop; client timeout is the shortest
• [ ] Streaming for user-facing generation; time-to-first-token monitored
• [ ] Fallback chain defined and eval-tested: smaller model → cache → honest degradation
• [ ] Circuit breaker in front of the provider; open state alerts
• [ ] Non-interactive work queued, rate-limited by you, with idempotency keys
• [ ] Errors classed separately in dashboards; percentile latency, not averages

Conclusion

LLM APIs fail in exactly four ways — over-budget, overloaded, too slow, or ambiguously — and every one of them has a known, boring, decades-old remedy: backoff with jitter, header-driven throttling, timeouts and streaming, fallback chains, circuit breakers, queues with idempotency. None of this is AI-specific engineering. That's the reassuring part: the reliability discipline your team already trusts for databases and payment gateways transfers directly — the only thing new is admitting that a probabilistic text API that takes ninety seconds and costs money per token is a tier-one dependency and deserves the same respect.

Build the five layers once, and the next traffic spike is a graph you glance at — not a Monday you remember.

---

Written by the team behind Cursuri-AI.ro, an AI education platform with hands-on English-language courses on production LLM integration, AI agents, evaluation, and shipping AI products.

Sources & further reading:

• Anthropic — API errors and rate limits (error classes, 529, headers, retry guidance)
• AWS Architecture Blog — Exponential Backoff and Jitter
• Tenacity — retrying library for Python
• Michael Nygard — Release It! (circuit breaker pattern, stability antipatterns)

This article is educational content. Provider error semantics, SDK defaults, and model names evolve — verify against current official documentation before shipping.