I want to talk about a problem that sounds simple until it isn't: fetching content from URLs that you don't control.
You build a system that ingests web pages. It works great in development, you point it at a handful of reliable URLs and everything comes back clean. Then you deploy it. Users start throwing real-world links at it: a research paper hosted on a university server that throttles at 2 AM, a company blog that returns 503 for thirty seconds after a CDN hiccup, an API endpoint that drops the TCP connection halfway through a large response. Suddenly your "working" system is silently losing jobs.
This is the story of how we thought through that problem, what we built, and what it taught us about resilient job queues.
The Problem
The surface-level issue is easy to describe: sometimes HTTP requests fail. But the real challenge is more nuanced than that. There are at least three distinct failure modes that look superficially similar but demand completely different responses:
Permanent failures. A 404 means the page is gone. A 410 means it was deliberately removed. Retrying these is pointless, you'll just get the same answer a thousand times.
Transient failures. A 503 means the server is temporarily overloaded. A ReadTimeout means the server started responding but stalled. A RemoteProtocolError means the connection dropped before the response completed. These failures do resolve on their own, the key word being "on their own," not "immediately."
Quota and rate-limit failures. A 429 means you're sending requests too fast. Retrying immediately makes this actively worse. You need to back off and give the server room to breathe.
When we first encountered this in production, our naive approach was: fail the job, surface an error, let the user retry manually. That's honest, but it's also terrible UX. A user who ingests twenty URLs shouldn't have to babysit the queue because three of them hit a momentary server hiccup at 3 AM.
The fix needed to be automatic, and it needed to be smart about the difference between "this will never work" and "this will probably work if you wait a bit."
Classifying Errors Before Retrying
The first piece of the solution was error classification. Not all exceptions are equal, and the retry logic should only apply to genuinely transient conditions. We settled on a concrete list:
- Transient (retry with backoff): connection timeouts, read timeouts, pool timeouts, connection refused, mid-stream read failures, protocol errors (TCP connection dropped), and HTTP 5xx responses
- Fatal (fail immediately): HTTP 4xx responses (except 429), DNS resolution failures for clearly invalid hosts, malformed URLs, parsing errors in the fetched content
The distinction matters. If you retry a 404, you're wasting retry budget on something that will never change. If you don't retry a ConnectTimeout, you're turning a 30-second server hiccup into a permanently failed job.
One thing that surprised us: httpx.RemoteProtocolError — the exception Python's httpx raises when a server closes the connection mid-response — was not in our initial transient list. It was classified as a generic error, which meant it triggered a full stacktrace in the server log and didn't retry. In production, this happened constantly with certain CDN-backed sites that aggressively close idle connections. Getting the classification right was more important than the backoff math.
Why "Just Retry" Isn't Enough
Once you know a failure is transient, the next question is: when do you retry?
The instinct is to retry immediately, or after a fixed short delay. Both are wrong for the same reason: if a server is struggling, hitting it again right away just adds to its load. You become part of the problem.
Exponential backoff is the standard solution, and for good reason. The idea is simple: each retry waits longer than the last, growing exponentially up to some cap. In our case:
delay = min(base × 2^(attempt − 1), max_cap)
With a 30-second base and a 300-second cap, the schedule looks like:
| Attempt | Wait before retry |
|---|
| 1st retry | 30 s |
| 2nd retry | 60 s |
| 3rd retry | 120 s |
| 4th+ retry | 300 s (capped) |
This gives a struggling server time to recover. It also means that a brief hiccup (a server that was down for 45 seconds) resolves on the second attempt rather than burning through all three retries in thirty seconds.
We deliberately left out jitter in this implementation. Jitter — adding random noise to the delay — is valuable when you have many clients hitting the same endpoint simultaneously (it desynchronises their retry waves). With a single-instance queue serving one user's ingestion jobs, the coordination problem doesn't exist yet. We'll revisit this when we get to multi-user support.
The Architecture: Backoff Without Blocking
Here's where it gets interesting. The obvious implementation of "wait before retrying" is to sleep: put the worker thread to sleep for the backoff duration, then try again. This is simple but has a critical flaw. While the worker is sleeping on one job, all other jobs are also blocked.
The solution we used is a pattern worth knowing: store the retry timestamp in the job record, not in the worker.
When a job fails with a transient error, instead of sleeping, we:
- Compute the backoff delay
- Write
retry_after = datetime('now', '+30 seconds') into the job's database record
- Set the job status back to
pending and return immediately
The worker loop doesn't sleep at all. It keeps running, calling dequeue() to pick up the next available job. The dequeue() query just filters out jobs whose retry_after timestamp hasn't passed yet:
SELECT ... FROM jobs
WHERE status = 'pending'
AND (retry_after IS NULL OR retry_after <= datetime('now'))
ORDER BY created_at ASC
LIMIT 1
The result: a job waiting out its 60-second backoff window doesn't block any other job. Other URLs in the queue get processed normally during that window. When the window expires, the job becomes eligible again and gets picked up on the next dequeue() call.
Here's the full state flow:
One subtlety worth noting: when the last retry attempt fails (the job goes dead), the log message should say so clearly. Our initial implementation always logged "will retry" regardless of whether the job was actually going to retry or die. We fixed this by reading the current retry count from the database before calling fail(), so the message accurately reflects what's about to happen. Operational clarity in logs is underrated, it's the difference between an engineer immediately knowing what happened and spending twenty minutes tracing through job IDs.
This Is a Common Pattern
What we built here isn't novel. It's a simplified version of what every serious distributed task queue does. Amazon SQS has visibility timeouts and dead-letter queues. Celery has max_retries and countdown. Sidekiq (Ruby) does exponential backoff with jitter by default.
The pattern has a name: at-least-once delivery with bounded retries. The key properties are:
- A job is retried at least as many times as configured (transient failures don't count as "the job failed" until the budget is exhausted)
- Each retry is bounded by a delay that grows, so you don't cause thundering-herd problems
- The retry window is non-blocking, other work proceeds during the wait
- Dead jobs are surfaced explicitly for manual inspection or operator retry
What's often underappreciated is why you store the retry timestamp in the database rather than sleeping the worker. It's not just about throughput. It's about durability: if the server restarts while a job is waiting out its backoff window, the retry schedule survives. The timestamp is in persistent storage; the sleep would not be.
What Changes with Multiple Concurrent Users
Everything described above works well for a single-user instance with one job queue. When you move to multi-tenant or high-concurrency scenarios, the picture changes in interesting ways.
Fairness becomes a problem. If user A submits 100 URLs right before user B submits 5, user B's jobs sit at the back of the queue. The current FIFO ordering has no concept of per-user fairness. A weighted round-robin or per-tenant sub-queue would fix this.
Per-domain rate limiting becomes important. Right now, backoff is per-job. If ten different jobs all target the same domain and it starts returning 503, each one independently backs off and retries. What you actually want is domain-level rate limiting: once you've seen a 503 from example.com, all pending jobs targeting example.com should pause, not just the one that just failed. This requires a shared rate-limit state (either in the database or in a cache) that all workers can read.
Jitter earns its place. With multiple workers and multiple users, you will get synchronized retry waves. Adding ± 20% jitter to the backoff delay breaks this up. With a single worker it's noise; with ten workers it's meaningful.
The worker concurrency model matters. Our current design is a single async worker loop. Scaling to multiple workers introduces the classic dequeue race: two workers might pick up the same job if the status update and the dequeue aren't atomic. The standard fix is a SELECT ... FOR UPDATE or optimistic locking pattern on the job row. SQLite doesn't support SELECT FOR UPDATE, but the write-ahead-log mode plus careful transaction scoping can achieve the same guarantee.
None of these are insurmountable, they're well-understood problems with well-understood solutions. But it's worth being deliberate about them before they appear in production rather than after.
What We Learned
A few things stuck with us from going through this:
Error classification is the hard part. The math of exponential backoff is trivial. Figuring out which exceptions deserve a retry versus an immediate failure took more thought than everything else combined. The naive instinct is "retry everything," which leads to wasting retry budgets on permanent failures. The opposite instinct — "fail fast on anything unexpected" — turns transient network noise into user-visible failures. The answer is a maintained allowlist of exception types that are genuinely transient.
Log accuracy is operational infrastructure. A log message that says "will retry" when the job is actually dying is worse than no message at all, it sends operators down the wrong path. Writing accurate log messages requires reading state at the moment of logging, not assuming what state will be. It's a small discipline with a large payoff.
Sleeping the worker is the wrong abstraction. The first intuition for "wait before retrying" is almost always to sleep. Don't. Store the timestamp, return immediately, and let the scheduler do its job. The sleep-based approach is brittle (server restarts lose the sleep), non-composable (you can't introspect what a sleeping worker is waiting for), and blocking (everything queues up behind it). The timestamp-in-database approach is none of these things.
Design for observability from the start. synthadoc jobs list showing job status, retry count, and retry_after timestamp turned out to be more valuable than anticipated. When something goes wrong, being able to see exactly which jobs are in which state — and why — is the fastest path to resolution. Bolting observability on after the fact is always more expensive than building it in.
The specific implementation here is for a LLM domain knowledgebase compilation engine, but the underlying patterns — transient/fatal error classification, timestamp-based backoff, non-blocking retry windows, accurate operational logging — apply anywhere you're calling external services over an unreliable network. Which is to say, anywhere.
If you're building something similar and have questions, the discussion is open.
