The agent was twenty minutes into a migration task. It had read the schema, planned eleven steps, executed six of them — two of which wrote to production — and was waiting on the model for step seven when the pod was rescheduled.
Everything the agent knew lived in a Python list called messages, in the memory of a process that no longer exists.
You now have three bad options. Redo the whole run, and steps 1–6 happen a second time, including the two that wrote to production. Abandon it, and leave a migration half-applied. Or have a human reconstruct what happened from logs, which is what most teams actually do, at 11 p.m.
None of these are retry problems. Retries handle a 429 or a 529 inside a process that is still alive — a separate and well-understood layer. This is what happens when the process itself goes away, and the fix is architectural: the run has to live in the database, not in the heap.
Why long agent runs die, and why it is not exotic
A ten-second chat completion effectively never gets interrupted. A twenty-minute agent run gets interrupted constantly, because twenty minutes is long enough to collide with ordinary operations:
- A deploy. You ship twice a day. Any run longer than your drain timeout is killed by your own CI.
- A platform timeout. Lambda caps at 15 minutes. Cloud Run request timeouts, ALB idle timeouts, and Cloudflare's limits all sit below "agent doing real work".
- OOM. Agent runs accumulate tool results — a large file read, a big query result — and the process footprint grows with the transcript.
- A long backoff. You hit a rate limit with a
retry-after of 90 seconds, and something upstream decides the worker is hung.
- A crash in your own tool code. An unhandled exception in step 7 takes the loop down with it, and the loop was holding the only copy of steps 1–6.
The frequency is what surprises people. If a run takes 20 minutes and you deploy twice a day with a 30-second drain, a meaningful fraction of in-flight runs die every single day. At one run an hour that is invisible; at a thousand it is a daily incident.
The model: a run is a state machine, not a for-loop
The mental shift is small and everything follows from it. Today your loop probably looks like this:
def run_agent(task: str) -> str:
messages = [{"role": "user", "content": task}]
while True:
resp = client.messages.create(model=MODEL, messages=messages, tools=TOOLS, max_tokens=8000)
messages.append({"role": "assistant", "content": resp.content})
if resp.stop_reason != "tool_use":
return text_of(resp)
results = [execute(b) for b in resp.content if b.type == "tool_use"]
messages.append({"role": "user", "content": results})
messages is the state, the heap is the storage, and the process is the transaction boundary. Kill the process and the state is gone.
The durable version separates three things that the loop above conflates:
- The run — a row with a status and a task. It outlives every worker.
- The steps — an append-only log of what happened, in order. This is the conversation;
messages becomes a projection of it.
- The worker — a disposable process that leases a run, advances it one step, commits, and repeats. Any worker can pick up any run at any time.
Step 7 crashing now means: the run is still running, steps 1–6 are committed, and the next worker to pick up the lease rebuilds the message list from the log and continues at step 7. Nothing is redone.
The schema
CREATE TABLE agent_run (
id UUID PRIMARY KEY,
task TEXT NOT NULL,
status TEXT NOT NULL, -- pending|running|succeeded|failed|cancelled
model TEXT NOT NULL, -- pin it: see the append-only section
lease_owner TEXT, -- worker id holding this run
lease_until TIMESTAMPTZ, -- heartbeat expiry
attempt INT NOT NULL DEFAULT 0,
result TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE agent_step (
run_id UUID NOT NULL REFERENCES agent_run(id),
seq INT NOT NULL, -- 0,1,2... strictly increasing per run
kind TEXT NOT NULL, -- model_turn | tool_result
payload JSONB NOT NULL, -- the raw content blocks, verbatim
request_id TEXT, -- provider request id, for support tickets
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (run_id, seq)
);
Two constraints carry most of the weight.
PRIMARY KEY (run_id, seq) makes double-writes impossible. If two workers somehow both try to append step 7, one of them gets a unique violation and loses. That is not an edge case to handle later; it is the safety property that lets you run more than one worker.
Steps are append-only. Never UPDATE agent_step. A retried tool call is a new step with a new seq and its own error payload, not an overwrite of the failed one. You lose nothing, you can replay any run exactly, and post-mortems stop being fiction.
payload is verbatim. Store the provider's content blocks as they arrived — do not normalize, do not strip fields you think are noise. The next section is about why that is not merely good hygiene.
Rebuilding the conversation: append-only history is now enforced
Resume means reconstructing messages from the step log:
def rebuild_messages(db, run_id: str) -> list[dict]:
steps = db.query(
"SELECT kind, payload FROM agent_step WHERE run_id = %s ORDER BY seq", run_id
)
messages = [{"role": "user", "content": steps[0]["payload"]["task"]}]
for step in steps[1:]:
role = "assistant" if step["kind"] == "model_turn" else "user"
messages.append({"role": role, "content": step["payload"]})
return messages
Simple — and there is one trap in it that turns a working resume into a hard 400 on the current generation of models.
Anthropic's Fable 5.1 made conversation history append-only. Each thinking block is cryptographically bound to the system prompt, the tool definitions, and the history that preceded it. Modify any of that on the client and you get Invalid "signature" in "thinking" block — a 400, not a degraded response. (Enforcement is implicit for accounts created from 31 August 2026 onward; older accounts can opt in with the thinking-binding-controls-2026-08-01 beta to find out now rather than later.)
Resume is precisely the operation that tempts you to modify history. Three rules keep you on the right side of it:
- Store payloads verbatim. A "cleanup" that drops a field you considered redundant will break the signature. Serialize what the SDK gave you, whole.
- Pin the model on the run row.
thinking blocks are bound to the model that produced them; 5.1 can read blocks from older models, but not the reverse. A worker that picks up an old run and reads MODEL from environment config — which you just bumped — resumes with the wrong model and fails at a distance from the cause.
- Pin the system prompt and tool definitions too, or version them. Deploying a reworded system prompt mid-run invalidates the binding for every in-flight run. Store a
prompt_version on the run and resolve the text from a versioned registry.
That last rule catches teams weeks later: resume worked perfectly in testing, then the day someone edits the system prompt, every run that survives the deploy fails on its next turn. Note that server-side compaction and context editing do not count as client-side modification — this constraint is about what you rewrite.
Idempotency: the step that spends money
Persisting the transcript fixes the "redo the whole run" problem. It does not fix the narrower and more expensive one: a tool call executed, and the process died before the result was committed.
You cannot tell those two apart from the outside. All you know at resume time is that step 7 has no result. The tool may have done nothing, or it may have charged a card.
At-least-once delivery is the only guarantee a crash-safe system can offer. Exactly-once execution is achieved the same way it is everywhere else in distributed systems: at-least-once delivery plus idempotent effects.
import hashlib, json
def idempotency_key(run_id: str, seq: int, tool: str, args: dict) -> str:
canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(f"{run_id}:{seq}:{tool}:{canonical}".encode()).hexdigest()
The key is derived from position in the run, not from time or a UUID. Two attempts at step 7 produce the same key; step 8 produces a different one. Pass it to anything with a side effect — Stripe, your own API, a queue — and the second attempt returns the first result instead of performing the action again.
For tools you own and that have no natural idempotency support, record the effect in the same transaction as the step:
def execute_step(db, run, seq, tool_use):
key = idempotency_key(run["id"], seq, tool_use.name, tool_use.input)
cached = db.fetchone("SELECT result FROM tool_effect WHERE key = %s", key)
if cached:
return cached["result"] # already happened; do not repeat
result = TOOLS[tool_use.name](**tool_use.input)
with db.transaction():
db.insert("tool_effect", key=key, result=result)
db.insert("agent_step", run_id=run["id"], seq=seq,
kind="tool_result", payload=serialize(result))
return result
There is still a window — between TOOLS[...]() returning and the transaction committing — where a crash loses the record of an effect that happened. It cannot be closed without a distributed transaction, and you do not want one. Shrink it instead: for genuinely dangerous tools, write an intent row before calling, so resume finds "we were about to charge this card" and can ask a human rather than guess. High-risk actions are also exactly where a human approval gate belongs, and an approval record is itself a durable checkpoint — the two designs reinforce each other.
Tool classification, done once, at design time:
| Class | Example | Resume policy |
|---|
| Pure read | get_schema, search_docs | Just re-run it |
| Idempotent write | set_status(id, "done") | Re-run; same end state |
| Non-idempotent | charge_card, send_email, post_message | Idempotency key, mandatory |
| Irreversible | DROP TABLE, terminate_instance | Intent row + human confirmation |
The row that gets skipped is the third one, and it is the one that shows up in a customer complaint.
Leases: stopping two workers from running the same run
Once runs are in a table and workers are disposable, the obvious failure is two workers picking up the same run — one that appeared dead but was only in a long backoff. Both advance it, both append steps, and the primary key saves you from corruption but not from duplicate tool calls.
A lease with a heartbeat fixes it:
def claim_run(db, worker_id: str, lease_seconds: int = 60):
return db.fetchone("""
UPDATE agent_run
SET lease_owner = %s,
lease_until = now() + make_interval(secs => %s),
status = 'running',
attempt = attempt + 1
WHERE id = (
SELECT id FROM agent_run
WHERE status IN ('pending', 'running')
AND (lease_until IS NULL OR lease_until < now())
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *;
""", worker_id, lease_seconds)
FOR UPDATE SKIP LOCKED is what makes this safe under concurrency without a separate queue: two workers polling simultaneously get two different runs, never the same one. The worker renews lease_until on every committed step. If it dies, the lease expires and the run becomes claimable — with no supervisor process needed to notice the death.
Keep the lease short (60 seconds) and renew often, rather than long and hopeful. A long lease means a dead worker's run sits idle for the length of the lease. Renew on each step and, if a single step can exceed the lease, from a background heartbeat thread.
Guard against poison runs with attempt: a run that has been claimed eight times is failing deterministically and must go to a dead-letter status. Without that check, one malformed run will be retried by your fleet forever.
Cancellation, for free
A benefit that pays for the whole design on its own: with the run in a table, cancellation is an UPDATE.
# in the worker, checked before each step
if db.value("SELECT status FROM agent_run WHERE id = %s", run_id) == "cancelled":
return
Try cancelling an in-flight while True loop inside an anonymous process. You cannot, which is why so many agent products have no stop button — and why so many have a stop button that only hides the UI while the tokens keep burning.
The same table makes the operational surface trivial: runs by status, oldest pending run, p95 steps per run, attempts distribution. That is a dashboard and an alert, built from data you already have. Pipe run_id into every log line and a support question ("what did it do for this customer?") becomes one query instead of a log-grep expedition. This is the ordinary SRE treatment applied to agents, and it works because you made the agent ordinary software.
When to use a workflow engine instead
Everything above is roughly 200 lines on top of Postgres. Temporal, Restate, and similar durable-execution engines give you the same guarantees with the replay semantics handled for you.
Build it yourself when: you already run Postgres, your steps are minutes not days, and you want the step log in the same database as your application data so you can join against it. The debugging story — SELECT * FROM agent_step WHERE run_id = ... — is genuinely better than an opaque engine's event history.
Use an engine when: runs span days and involve human waits, you need timers and signals, you are coordinating multiple agents that must rendezvous, or you already run one for other workloads. Do not adopt one only for agents; the operational cost is real, and an agent run is a simpler shape than the workflows these engines were built for.
The part that is not optional either way is idempotency. Temporal will happily re-execute your non-idempotent activity during a replay. Durable execution guarantees that the workflow resumes correctly, never that your side effects are safe.
FAQ
Does this apply to a 10-second chat completion?
No. If a run cannot outlive a deploy, an in-memory loop is the right amount of machinery. The threshold is roughly "longer than your drain timeout" — past that, interruption stops being an edge case and becomes a daily event.
Can I just store the messages list as a JSON blob after every turn?
It survives a crash, which is most of the benefit, and it is a reasonable first version. What it does not give you is a per-step primary key (so concurrent workers can corrupt it), a record of failed attempts, or a queryable history. The step table is maybe forty more lines and you stop rewriting the same row under concurrency.
Do I need a queue like SQS or Celery in front of this?
Not for correctness — FOR UPDATE SKIP LOCKED is a perfectly good work queue at moderate volume, and keeping the queue in the same database means claiming a run and committing a step share one transaction. Add a broker when you need fan-out to many worker types or cross-service routing.
What happens to a run whose model was retired mid-flight?
It fails, and it should. Pinning the model on the run row is what makes that a clean, early, obvious failure instead of a confusing one. Drain long runs before removing a model from your rotation, and treat any run still pinned to a dead model as a dead-letter case for a human.
Is streaming compatible with this?
Yes. Stream to the user for responsiveness, and commit the assembled message to the step log before dispatching any tool. The checkpoint boundary is the completed turn, not the token.
Checklist
- [ ] Every model turn and tool result is committed to an append-only step log before the next call
- [ ]
PRIMARY KEY (run_id, seq) — duplicate steps are impossible, not merely unlikely
- [ ] Payloads stored verbatim; model, system prompt, and tool definitions pinned per run
- [ ] Every tool classified: pure / idempotent / non-idempotent / irreversible
- [ ] Non-idempotent tools take a key derived from
run_id + seq + args
- [ ] Irreversible tools write an intent row before executing
- [ ] Leases with
FOR UPDATE SKIP LOCKED and a heartbeat; attempt capped with a dead-letter status
- [ ] Cancellation is a status change the worker checks between steps
- [ ] A
run_id in every log line
- [ ] A test that kills the worker mid-run and asserts the resumed run does not repeat a side effect
That last one is the only way to know any of this works. Run the agent, SIGKILL the worker between steps 3 and 4, start a new worker, and assert the effect count. Do it in CI. Crash-safety that has never been crashed is a hypothesis.
The short version
Agent runs are long enough to be interrupted by routine operations, and an in-memory messages list is not a place to keep work you cannot afford to redo. Move the run into a table, make the step log append-only, rebuild the conversation on resume, and make every side-effecting tool idempotent by position. None of this is AI-specific — it is the same durability discipline as any long-running job. The only genuinely new constraint is that modern models bind their reasoning blocks to the exact history that produced them, so "rebuild the conversation" now means byte-identical, not equivalent.
I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform. If you are taking an agent from a working demo to something that survives a deploy, the production AI SaaS track covers this end to end.
