A support model picks refund_order, fills the schema, validates clean, and pushes $54 to the right account. The user asked about $5,400. Tool selection scored 1.0. The arguments passed schema validation. The number was off by an order of magnitude, and the only signal anything broke was a support ticket forty minutes later.
That failure is the reason I stopped scoring tool use as one number. Tool use is a four-step contract. The model has to decide whether to call any tool at all, pick the right tool from the catalog, build correct arguments against the schema and the user's intent, and integrate the result into its answer without mangling the payload. Get one of the four wrong and the demo still passes. Score it as a single number and you will never know which step broke. There is a good longer writeup of this four-step contract with the rubric code and full CI wiring; this is the shorter argument for scoring each step on its own.
Why one number hides three failures
Four failure modes show up in postmortems, and a single score collapses all of them into "the agent broke."
- Over-call. The model fires
search_orders on a greeting because the prompt was tuned harder on selection. Selection metrics never see it, because the gold was always a tool.
- Wrong tool, right family.
get_order_status versus get_customer_status: same schema shape, different downstream system. F1 catches it only if the eval set has both endpoints.
- Right tool, wrong arguments.
departure_date="next Friday" validates if the field accepts strings. customer_id="me" returns someone else's account. Schema gates the type, not the meaning.
- Right call, wrong narration. The tool returns
amount_cents=4500, the model says "$54.00 is processing," and the user believes it. Almost every public benchmark skips this step.
Score the four separately and the vocabulary changes from "the agent failed" to "step 3 regressed on the date-string judge for the flight-booking path." That is one bisect instead of three days.
Most eval sets only contain rows where the gold answer is a tool call, so the over-call regression ships untested. Reserve 10 to 15 percent of the set for no-call cases: greetings, clarifications, in-model factual questions, refusal-worthy asks, off-topic chatter. Score precision and recall on the binary call-or-no-call decision, and gate on per-class accuracy.
This is the step I see skipped most. A model that scores 0.95 on tool-name F1 and 0.40 on the no-call slice is a model that over-calls, and without the no-call score you only find out in production. Counting calls against the gold count is the easy, deterministic half. The hard case is when both calling and staying quiet are reasonable, which is where a judge that reads the conversation earns its place.
Once the model decides to call, pull the chosen tool name, compare to the gold label, and aggregate F1 per tool so a catalog of 28 endpoints does not hide a regression on one rare endpoint behind the global mean.
The failure that hides here is the near-miss family: get_user versus get_customer, cancel_order versus cancel_booking, refund_order versus void_charge. Similar names, different downstream systems. Exact-match on the name is a cheap deterministic gate. The ambiguous pairs need a judge that gets the tool catalog with descriptions and scores whether the pick maps to the user's stated entity, not just keyword overlap.
Step 3: construct correct arguments
Argument failures come in three buckets, and each needs a different check.
- Schema mismatch (wrong type, missing required field). Deterministic, and it runs first. Validate the model's output against the tool's input schema. Zero LLM cost, and it catches type errors, missing fields, pattern mismatches, and range violations.
- Semantic mismatch (right schema, wrong value).
departure_date="2026-01-01" validates and is still wrong if the user said "next Friday." customer_id="42" is a valid string that refunds the wrong account. This needs a judge with the last few turns of context, scoring whether each argument is grounded in the conversation: identifiers must appear in prior turns, amounts must be within an order of magnitude of any stated value, dates must match intent.
- Edge cases (null, empty array, unicode, type coercion across the model-to-tool boundary). These are private to your registry, so a small per-tool regression suite is the only thing that catches them. Put the time-zone case on every date field and the currency case on every monetary field.
Step 4: integrate the result
The tool returned, the model has the payload, and three patterns go wrong.
The model paraphrases with a number flipped: amount_cents=540000 becomes "$54.00." The model substitutes prior knowledge: a balance tool returns a value below the limit and the model "knows" it is above the $200 threshold, because it never read the result. The model drifts across turns: the right itinerary on turn 1, an invented baggage policy on turn 3 that contradicts the payload from two turns back.
The rubric is groundedness with the tool payload as the context, not a retrieved corpus. Chunk the result into its JSON fields and score whether each claim in the response maps to one. Run it on every model where a tool feeds the answer, and score it across turns, because per-call rubrics never see the turn-3 drift.
Where public benchmarks fit
Two public benchmarks anchor model selection, and neither should gate your production.
BFCL measures function calling across syntactic correctness, an executable track where the call actually runs, and an irrelevance bucket that steps 1 and 2 depend on. The breakdown matters more than the headline: a model that aces syntax and tanks irrelevance over-calls; one that aces syntax and tanks the executable track writes plausible calls that do not run.
τ-bench measures multi-turn behavior with an LLM-simulated user and a domain policy, and its headline metric is the fraction of independent rollouts that all succeed, not retries on one input. It is one of the few public numbers that quantifies how much consistency degrades when nondeterminism stacks, and even strong models fall hard on it over repeated rollouts.
Both tell you whether the underlying model can call tools at all. Neither tells you anything about your registry, your schemas, your error codes, or your business policy. The private eval set is the one that gates production. Build it stratified by tool, by argument edge-case bucket, and by call-versus-no-call gold, at least 30 cases per tool, and promote failing production traces into it weekly.
Keep the contract honest in production
CI is not enough, because the contract has to hold on live traffic. Emit every tool call as a span carrying the tool name, the arguments, the result, and the latency, so per-tool p50, p95, and p99 are one query away. Then attach the same per-step rubrics you ran in CI as scorers that run server-side after export, at no inline latency, so every tool call in production is graded on the same four steps. Rough thresholds to gate on: call-or-no-call precision at 0.95, tool-name F1 at 0.95, argument schema validation at 0.90, argument plausibility at 0.85, and result groundedness at 0.90.
Three tradeoffs worth naming
- Per-step scoring costs more than one aggregate number. Five rubrics per case, not one. The payoff is that a failed run names the broken step instead of starting a triage session. Ship the deterministic layers first (schema validation, name match, call counts) and turn on the judge layers once volume justifies the bill.
- Groundedness on JSON is noisier than on prose. Fields have fewer surface cues than passages, so pin a small human-labeled calibration set and re-tune it monthly.
- The no-call slice feels expensive to build. Labeling "what should the model do here" is softer than a tool-call gold. Build it once, treat it as a fixture, and refresh it quarterly from production turns where the model fired a tool it should not have.
Pick one tool with a real write side effect. Build a 60-case eval set across the four steps: 10 no-call, 20 selection with at least one near-miss family, 20 argument cases including 6 semantic, 10 result integration. Wire name match, schema validation, an argument-plausibility judge, and groundedness against the tool payload into a fixture, gate CI on the five thresholds, then attach the same rubrics as production scorers. One tool, four scored steps, and the next regression tells you which step it broke instead of leaving you to guess. The full four-step writeup has the rubric code and CI config if you want to lift it directly.