Your multi-agent system works locally. Three agents coordinate, call tools, pass context, return a clean answer. In production the final output is wrong, and you cannot tell which agent failed, which tool call returned garbage, or where the reasoning chain broke. That gap is the whole problem, and a span tree is what closes it.
Multi-agent systems fail in ways a single agent does not. Agents hand off tasks, share state, call external APIs, and decide independently. When one hallucinates or a tool call times out, the error cascades silently through the rest of the chain while the dashboard stays green. Logs give you fragments. A trace gives you the full path: every decision, every tool call, every token, from the user query to the final response. There is a longer production writeup with the full code if you want to lift it; this is the shorter version.
The four failures that hide from logs
- Tool-calling errors. The agent calls a function with malformed parameters, the tool errors, and the agent either retries wrong or ignores it and makes up an answer. Logs show "tool failed," not why.
- Silent handoff failures. Agent A passes incomplete context to Agent B, B produces a confident wrong answer, no exception is thrown, and the user just gets a bad reply.
- Hallucination that compounds. An agent fabricates a detail in step 2, and every downstream step treats it as fact. The logs show the final output, not where the invention started.
- Latency compounding. Every agent in a chain adds time. A 2-second delay in any one of a planner, retriever, or summarizer can push the whole response past what a user will wait for.
Each of these is invisible unless you have span-level data, and each is obvious once you do.
Model the span tree to match the agents
A trace is one full execution; a span is one operation inside it. For agents the hierarchy is a root span (the whole workflow), agent spans (each agent's turn), and under them the LLM, tool, retriever, and embedding spans. Every span carries its input, output, latency, model, token counts, and status, and each child links back to its parent, so the trace is the full execution tree.
For "what is the status of order #4521," it reads top to bottom:
triage_agent decides to route
chat model call: picks order_lookup_agentorder_lookup_agent
order_api tool: GET /orders/4521chat model call: formats the order into natural language
response_agent
chat model call: composes the final reply
When the answer is wrong, you walk it backward: did the response agent misread the data, did the order API return something stale, did triage route to the wrong sub-agent. The tree is the chain of custody for every piece of information.
Three failures, and how the trace names them
A tool call with bad arguments. A booking agent calls flight_search and it errors on an empty destination. Walk one span up to the model call that produced it, and the input was "somewhere warm next week." The model could not resolve that to a city, so it passed an empty string instead of asking. The fix is a prompt change: tell the agent to ask for a concrete destination when the query is ambiguous rather than calling the tool with a hole in it. The logs would only have said "flight_search failed."
A hallucination in a chain. A research agent retrieves "Acme reported $42M in Q1, up 12% YoY" and then answers "$42M, up 12% YoY, driven primarily by expansion into the European market." Open the retriever span, open the LLM span, compare: the European-market claim is in neither document. In a multi-agent pipeline that fabricated clause gets passed to a downstream analyst agent as fact. You catch it by scoring each LLM span for faithfulness against its upstream retriever span, with a threshold around 0.85 to flag the span for review. The score attaches to the span, so the failing node shows its faithfulness number next to its input and output.
A latency outlier. p95 jumps from 4 to 9 seconds. Sort the spans by duration and one retriever agent's vector query is 5.9 of the 8.8 seconds. Its attributes show an unindexed 2M-document collection queried with top_k=50. The fix is indexing, a smaller top_k, or a metadata pre-filter. Without per-span timing you would only know the pipeline was slow, not that one query ate half the wall time.
Tracing tells you what happened; evals tell you if it was good
The pattern that closes the loop is scoring spans, not just offline datasets. Run the same rubrics on live spans and attach the score to the span as an attribute, so the dashboard can surface failing spans by score, by error rate, and by latency in one view. The metrics worth tracking per agent: task completion, tool accuracy (from span status codes), faithfulness (retriever span vs LLM span), end-to-end latency (root span), cost per query (summed LLM tokens), and handoff success (a custom span attribute).
Then alert on the trends that matter: p95 past your SLA, tool-span error rate above baseline, quality scores drifting down, and token-cost spikes that usually mean an agent is looping.
The reactive version stops there. To make it preventive, put the same eval thresholds at a gateway in front of the model and fail closed, so a bad output never reaches the user instead of being explained after the fact in the next deploy.
A few practices worth adopting early
- Instrument from day one. Adding tracing after an incident is much harder than building it in.
- Name spans descriptively.
research_agent:web_search, not tool_call.
- Separate dev, staging, and prod with distinct project or version tags so test traffic does not pollute production dashboards.
- Trace state, not just inputs and outputs. If agents keep memory between steps, capture the state transitions as attributes.
- Keep one attribute schema across frameworks. If you run LangChain and CrewAI in the same system, both should emit the same attribute names, which OpenTelemetry conventions handle.
- Gate at the boundary. Run the same eval templates live that you run in CI, and block the traffic that fails.
Without the trace you debug blind. Without span-level evals you fly without instruments. Without the gate your fix only lands in the next deploy, not at the boundary. Instrument the agents, build a span tree that mirrors the architecture, attach evals to spans, and gate live traffic on the same thresholds CI uses.
