Every MCP server I have worked on is really two products under one binary.
The first is the functional surface: tool descriptions that match what the tools actually do, calls that succeed with the right arguments, results that flow into the next agent turn without breaking the chain. The second is the security surface: a tool catalog that can carry injection payloads, results that can be tampered with, arguments that can escape a sandbox, tenants that can read each other's calls.
Cover one and you ship something that works in the demo and falls over on the audit. Cover the other and you pass the security review and fall over on a real workload. Both gates need to run on every PR, over one trace tree. Here is the workflow I keep coming back to.
The functional side has three layers, and the cheap one earns the most
Tool-description accuracy. The agent reads the tool's name, description, and JSON schema at tools/list time and plans off that text. If the description says "returns the customer's billing history" but the tool returns the last 30 days of invoices, the agent reaches for it on queries it cannot satisfy. I have burned days on this one. It is a documentation regression that reads like a model regression in the logs, so you go looking in the wrong place.
Score it in two passes. First deterministic: validate the JSON schema against the tool's real signature. A required field missing, an integer typed as a string, an enum that does not match the runtime allow-list. Mechanical failures with mechanical fixes, caught before any model call. Then semantic: run a golden corpus of 30 to 60 prompts that should and should not reach for this tool, and score how often the agent picks correctly. A drifted description gives you a recall problem, an overpromised one gives you a precision problem, and both show up as the same number until you split the confusion matrix.
Tool-call success rate. A call succeeds when three things hold: the agent picks the right tool, the arguments validate against the schema, and the server returns 200 inside the per-tool budget. We hold production to 98 percent schema compliance. Below that, an upstream server has drifted and you are roughly three weeks behind knowing it.
Budget per tool, not per chain. On a four-call chain with an 8-second user-facing budget, the per-tool p95 sits around 1.5 seconds. Track p50, p95, and p99 per tool and alarm on p95 growth against a rolling 7-day baseline.
Result integration. A call that returns 200 still breaks the chain if the agent cannot parse the result, drops the right field, or ignores the response and replans from the prompt. Score three signals on the next turn: does the agent reference the returned data, does the chain progress, does the final answer cite the right value. Task completion, step efficiency, tool-selection accuracy, and goal progress cover this from the trajectory.
Two reasons the surface is bigger than people assume. Tool catalogs are part of the prompt, and the supply chain is npm install.
Four checks belong in the gate: description-injection scanning, result tampering, sandbox and permission-escape attempts, and cross-tenant isolation. Two implementation details matter more than the list. The scan layer has to fire at registration and on every tools/list refresh, against the name, the description, the top-level schema, and every nested description and enum value, because that is where a payload hides. And cross-tenant isolation has to be structural rather than classifier-based: per-key allowed tools, namespaced tool IDs, trace-ID scoping on every span. A classifier is the wrong instrument for a boundary that should be enforced.
If you only have one gate today, run the functional one and add security before your next red-team cycle.
A server that works in Claude Desktop and breaks in OpenAI's Responses API is a production bug, and a single-client run will never catch it. It breaks in three places:
- Schema dialect. Claude's planner handles deeply nested
oneOf and anyOf. OpenAI refuses some union shapes. Gemini wants flat parameter lists. A schema that plans cleanly in one client crashes another's router.
- Argument coercion. A tool declares
integer and a client sends "5". Booleans become strings. Dates arrive as ISO 8601 in one client and Unix epoch in another. The schema is the contract, the client is the enforcement.
- Result format. Text versus content blocks, MIME types on file results, image base64 versus URL. A tool returning
{ content: "..." } works everywhere. One returning { items: [...] } fails silently in some clients.
The gate is a fixture suite: 5 to 10 cases per tool against Claude, OpenAI, Gemini, and Cursor configs, asserting the same tool sequence and the same answer up to known reformatting. A diff between clients is a server bug, not a client quirk.
Three numbers decide whether it holds under load
Concurrency. Open parallel sessions calling the same tool with non-overlapping arguments. The break point shows up as 503s, dropped SSE streams, or state leaking between sessions. Run at twice peak production concurrency and hold error rate under 1 percent. State leaks are the failure production eventually finds for you.
Rate-limit behavior. Hit the per-tool limit and confirm you get a structured 429 with a Retry-After header, not a 500 with a stack trace.
Long-tail latency. A 200ms p50 with a 4-second p99 is a chain quietly retrying on the tail and paying for it in tokens. p50 is not the budget.
Two gates, two datasets, one workflow
Pre-merge on the server PR: schema validation across the golden set, hard fail on any regression. Semantic function-calling scoring on 200 to 500 traces sampled from the last week of production, at 95 percent pass for tools with stable descriptions and 99 percent for tools the PR did not touch. The compatibility fixture suite, hard fail on any cross-client diff. The security regression set at recall above 0.95 on the adversarial set and precision above 0.99 on the benign set. The stress run at 2x peak.
Post-merge on the agent PR: frozen prompts through the agent against the candidate server, asserting the same plan, the same tool sequence, and the same final answer as the baseline. Drift in any of the three blocks the merge until the trace explains the diff.
The mistakes I see most often
- Tracing at the model level only. A response-only score misses over-calling, retry-on-same-tool, and dropped context, which are the failures driving your cost. Trace at the tool level.
- Pinning tool names. MCP catalogs are dynamic. Assert on tool families and intent, or the suite breaks on the next
tools/list refresh.
- One budget for the whole chain. A single per-request budget hides the one tool that is regressing.
- Skipping cross-client checks. One client passing is not coverage.
- No stress gate. A server that is fine at 1x concurrency can leak state at 2x.
What actually changed with MCP
MCP moved a meaningful part of your agent's behavior into someone else's release cycle. The tool catalog is prompt content you do not own and it changes without asking you. That is the shift, and it is why one functional gate is no longer enough. Two gates, every PR, one trace tree. Anything narrower ships a server that only works on the happy path.