A support agent passes every eval in staging. In production the same agent is reached three ways: from an IDE over MCP, from the web app over the OpenAI Responses API, and from a triage agent over A2A. The dashboard stays green for weeks. Then a customer escalation shows the triage agent handed over a ticket without the customer's region, and the support agent quoted the refund policy for the wrong country. No evaluator flagged it, because no evaluator ever looked at the hop.
The reason is structural. Eval is a property of the path a request took, and the protocol defines the path. Most eval stacks were written against one surface, usually chat completions, so they score whatever still looks like a chat completion and quietly skip the rest. A protocol-neutral eval stack scores the path instead, whichever protocol carried the request.
Every protocol adds seams to the path
A plain chat completion has one path: prompt in, response out. One rubric can grade it. Each protocol you add forks that path into more decision points, and each decision point needs its own check.
- MCP adds a tool catalog. The path becomes prompt, tool descriptor list, tool selection, call arguments, tool result, final answer. Score each tool call against policy, and score the catalog as a whole, because a poisoned descriptor looks fine at call time and only shows up against the full catalog.
- A2A turns the path into a tree. Every hop needs three answers: did the calling agent pass enough context, did the called agent stay in scope, and did the returned artifact match the task that was asked for.
- The Responses API puts three outputs in one envelope. Structured JSON gets graded against a schema, streamed text gets graded as it arrives, and a tool call gets graded against a per-tool policy. One end-to-end pass mixes three things that share nothing except the wrapper.
- Realtime has no final message. Audio streams in both directions, so checks have to run mid-stream, every N tokens or N audio chunks, with a stop or disclaimer action when a rule fires. The end-to-end budget sits below 500ms, which caps how heavy each check can be.
- Inline vision needs a second judge. When images ride in the same message as tool blocks, the text portion and the image portion need different rubrics, and the image rubric needs a judge that can open the image.
- Session state spans turns. On a framework with explicit session state, the evaluator has to read state across turns instead of grading each turn alone.
Trace fidelity caps eval fidelity
An evaluator can only score what the trace can represent. If your trace has no way to express an agent-to-agent hop or a streaming chunk, no rubric downstream will catch a failure inside one.
So the first fix is in instrumentation, before any rubric work:
- Model both sides of a hop. Emit one span kind on the calling agent and another on the called agent, and carry the trace ID across the wire so the whole multi-agent path lands in one trace.
- Give each modality its own span kind. Voice, image and computer-use spans should be distinguishable from text spans, so each one reaches an evaluator built for that modality.
- Record guardrail and evaluator results as spans. When a guardrail decision and an evaluator score sit in the same trace as the model call, you can see which check ran on which step.
The MCP spec helps here. The 2026-07-28 revision documents trace context propagation through _meta (traceparent, tracestate, baggage), so MCP tool calls can join the same trace as the rest of the agent loop.
Normalize the spans, then write each rubric once
The same agent identity shows up differently on every protocol. If you write a groundedness check against raw MCP payloads and another against raw Responses payloads, you now maintain two checks that drift apart.
Normalize first. Map every span to one semantic convention, such as the OpenTelemetry GenAI conventions, before any evaluator reads it. Then write rubrics against the normalized trace. One groundedness rubric can then grade a RAG answer that arrived over the Responses API and a RAG answer that arrived over MCP, and one task-completion rubric can grade the support agent no matter which client reached it.
This is the same move observability teams made with OpenTelemetry. Teams that locked to one vendor's tracing format rebuilt their instrumentation every time they switched vendors. Agent protocols are repeating that pattern one layer up.
Gate releases per protocol
A single overall score hides which surface regressed. A drop in the MCP tool-audit rubric and a drop in the Realtime guardrail are different failures with different fixes.
- Keep a regression dataset per protocol surface. Run the matching rubric set against it on every change.
- Set a threshold per protocol. Fail the release when any one surface drops below its own threshold, even if the blended number looks fine.
- Key each suite on protocol and version. MCP's 2026-07-28 revision removed the
initialize handshake, so every request now carries its protocol version in _meta, and the new deprecation policy gives every deprecated feature a minimum twelve-month window. Tag each regression case with the revision it ran on, and keep the tests for deprecated features until their window closes.
- Keep the rubric sets in the repo as code. Reviewers should be able to read exactly what gets scored on each surface.
A five-step readiness audit
- Inventory every surface in production. MCP servers, A2A endpoints, streaming, voice, vision and computer-use, with the protocol version and client library for each. Most teams find one they did not plan for, usually an MCP server a developer installed last sprint.
- Instrument and pull one real trace per surface. Confirm the hop spans appear on both sides with the propagated trace ID, and that voice and image calls emit their own span kinds. Fix gaps before writing rubrics.
- Write the rubric set for each protocol. Per-tool audit on MCP, per-hop audit on A2A, split rubrics on the Responses API, mid-stream checks on Realtime, an image-capable judge for vision.
- Run a per-protocol regression suite and gate on per-protocol thresholds.
- Cluster production failures by protocol. Failures that look like noise one at a time often form a pattern once you group them by surface.
If you only have half a day, do step 1. The list is usually longer than the team expects, and it tells you which evaluators are scoring the wrong path today.
Pitfalls that keep showing up
- Evaluators hard-coded to the chat completions shape. Every new protocol triggers a rewrite.
- Scoring each agent in isolation. The end-to-end view looks green while a handoff in the middle dropped context.
- Post-call checks only on streaming surfaces. By the time the evaluator sees the output, the risky tokens are already on the wire.
- Accepting tool descriptors from any MCP server without a scan. An untrusted server can change what your agent does, and the eval layer never sees it.
- Scoring voice and computer-use on text alone. Errors at the ASR layer and the vision layer stay invisible.
The span kinds, the rubric set for each of the six protocols and the full audit are in the longer protocol-neutral eval playbook.
New protocols and revisions will keep arriving. Score the path the request took, and each new surface plugs into a trace model and a rubric library you already maintain.