Multi-agent systems tend to fail quietly. Every agent returns well-formed JSON, every eval suite goes green, and the final output is still wrong - because the failure happened in a handoff nobody inspected. The fix isn't a better model; it's a cheap invariant check between steps.
Schema Validation Isn't Correctness
Most agent pipelines validate structure: does the payload have the right keys, are the types correct, does it parse. That catches malformed output, which is the easy failure. The expensive failure is a payload that passes every structural check and is still semantically false - a summarizer agent that cites a document ID the retriever never returned, a pricing agent whose total doesn't equal the sum of its line items, a planner that outputs three steps when the task requires four.
End-to-end evaluation misses these because it scores the final answer, usually against a small golden set. If the bad handoff happens in an edge case your eval set doesn't cover, the score stays high while production quietly degrades. A watchdog - a small function that runs between agents and asserts relationships the schema can't express - catches the error at the step where it started, not three hops later when it's untraceable.
A Watchdog Between Two Agents
The pattern is unglamorous: pure Python, no model call, runs in microseconds.
class HandoffError(Exception): pass
def guard(step, payload, prior):
if payload["total"] != sum(i["price"] for i in payload["items"]):
raise HandoffError(f"{step}: total != sum(line items)")
if set(payload["cited_ids"]) - set(prior["retrieved_ids"]):
raise HandoffError(f"{step}: cited a doc that was never retrieved")
return payload
Then wrap each transition:
draft = writer_agent(context)
draft = guard("writer", draft, prior=retrieval_output)
final = reviewer_agent(draft)
When guard raises, you have three sane options: retry the step with the error text appended to the prompt, fall back to a deterministic path, or route to a human. Log every raise with the step name - after a week you'll know exactly which agent is your weak link, which is information no aggregate eval score gives you.
The design work is deciding what to assert. Good invariants are relationships between a step's input and its output: nothing cited that wasn't retrieved, no field invented that wasn't in the source, counts and sums that must reconcile. If you're a PM scoping this, that list is a spec question, not an engineering one - you likely already know the three things that must never be true in your product's output.
Key Takeaways
- Structural validation and end-to-end evals both miss payloads that are well-formed but semantically wrong; those are the failures that reach users.
- A watchdog is a plain function between agent steps that asserts input - output relationships a JSON schema can't express - no extra model call needed.
- Log every guard failure by step name. That log tells you which agent is unreliable, which an aggregate score never will.