An MCP-connected agent passes staging and then fails in production. A user sends a query slightly outside the test distribution, the agent picks the wrong tool, passes malformed arguments, chains three unnecessary calls, and returns garbage. The reason static tests do not catch it is structural: an MCP agent discovers its tools at runtime, the tool list changes between requests, and the agent decides which tools to call, in what order, with what arguments, at runtime. You cannot assert "the agent must call this tool" when the tool list itself is dynamic.
That shift creates three evaluation problems that did not exist with fixed-tool agents. Tool selection is non-deterministic, so you score whether the choice was reasonable given the alternatives, not against a fixed answer. Injected context needs validation, because an MCP resource returning stale data makes the agent reason wrong. And a single request fans out into five to ten tool calls across servers, so you score every step and the chain as a whole. There is a longer production writeup with the code if you want it; this is the shorter version.
The five pillars, with targets
Score five dimensions per trace, each with a workable production target.
- Tool selection accuracy. Compare the agent's tool choices against labeled examples, and split it into precision (of the tools it called, how many were needed) and recall (of the tools it should have called, how many it used). High precision and low recall means too cautious; low precision and high recall means over-calling, which burns tokens and slows the chain. Targets: precision above 85 percent, recall above 90 percent.
- Argument correctness. Right tool, wrong arguments is common: an MCP tool expects a
documentId and the agent sends a full URL, or omits a required field. Score JSON schema compliance against the live schema, type correctness, required-field presence, and semantic accuracy (the right document ID for this task, not any valid one). Target: above 98 percent schema compliance, and treat a drop below 95 as outage-level.
- Task completion. The bottom line: did the agent do what the user asked. Perfect tool selection means nothing if the synthesis is wrong, so score it with a judge that reads the full transcript against the original intent. Target above 80 percent.
- Chain efficiency. MCP agents over-call routinely. Track total calls per request, redundant calls (same tool, same arguments, one trace), unnecessary calls (outputs that never fed the answer), and total latency. A ratio of minimum-needed over actual calls above 0.7 is workable.
- Context utilization. Did the agent use the MCP-injected resources or hallucinate past them. Score groundedness and context relevance against the provided context, target above 85 percent.
Trace it, because you cannot score what you cannot see
Instrument with OpenTelemetry so every MCP tool call is a span carrying the tool name, server name, schema version, arguments, response, latency, and status. The spans nest under one parent trace: the root request, the LLM decision span (reasoning and tool choice), one span per tool call, the context-retrieval spans for MCP resource fetches, and the final synthesis span. That tree is what lets you attribute a bad answer to a specific step instead of guessing.
The gateway pattern
The cleanest production shape is a gateway between the agent and the MCP servers it can call. It enforces policy on every call (allowed servers, allowed tools, argument shape, rate limits, budgets), routes with your own keys, captures every interaction for scoring, and applies pre-call guardrails. The point is that the eval signals you use in development become the same ones gating live traffic, from one chokepoint, without touching agent code to rotate keys or budgets.
Sample production traffic, do not score all of it
Do not evaluate every request. Set a 10 to 20 percent sample rate for general traffic and score it async, and raise it for high-stakes flows (refunds, healthcare actions, code merges). Run the deterministic checks (schema validation, type checks, retry-on-same-tool detection) on 100 percent of traces, since those are the cheap-to-detect failures you cannot afford to miss. The pattern is deterministic-on-everything, judge-on-the-sample.
Then alert on the signals that matter: task completion below 80 percent, average tool calls per request above six, schema compliance below 95 percent. Route those to Slack, PagerDuty, or CI to close the loop.
The pitfalls worth naming
- Happy-path only. Dev MCP servers have limited tool sets, so mirror the production server config in your eval environment.
- Scoring calls in isolation. Order affects correctness, so evaluate whole chains, not each call alone.
- Judge-only scoring. LLM judges drift, so pair every judge with deterministic schema checks.
- No baseline. Establish baseline metrics in the first week and track deltas, or you cannot tell degradation from noise.
- No cost tracking. MCP tool calls add up fast, so put token and call cost on every trace and alert on spikes.
Evaluation without action is just monitoring. The full loop is trace every call, score the sampled traces across the five pillars, cluster the failures to find which tool calls and which queries break most, fix the prompts and tool descriptions and server configs, and verify by comparing scores across versions. The teams shipping reliable MCP agents are not the ones with the best base model. They are the ones whose pipeline catches a regression on Tuesday and ships the fix on Wednesday.
