I want to start with a trace that still bothers me.
An agent fails to book a flight. The model called search_flights with departure_date="next Friday". The endpoint expected an ISO date, returned a 400, and the agent retried four times with the exact same string before apologizing to the user.
Here is the part that matters. Tool selection was correct. The model picked the right function out of a registry of 28, so tool-selection accuracy logs a clean 1.0. Task completion logs a 0. Neither number tells you which of three things actually broke: the argument was wrong, the model never read the 400 body, or the retry policy looped on the same input.
That gap is the whole problem. If you only evaluate "did the agent call the right tool," you are testing intent, not execution. Tool selection is necessary, not sufficient.
The opinion I have landed on after enough of these postmortems: tool-calling eval is four eval problems stacked, not one.
The four layers
Layer 1, tool selection. Did the agent pick the right tool, or correctly pick none? F1 on the tool name catches most of it. The piece almost everyone drops is the irrelevance bucket: cases where the right answer is no tool call at all, like a greeting, a clarification request, or an in-model factual question. Without those rows in your test set, you cannot catch the regression where a new prompt revision makes the model bolder about calling search on every input.
from fi.evals import evaluate
result = evaluate("function_name_match",
output={"function_name": predicted_tool},
expected={"function_name": ground_truth_tool})
# result.score = 1.0 exact match, 0.0 otherwise
Layer 2, argument extraction. The schema is right, the types are right, and the values are wrong. departure_date="next Friday" schema-validates and still fails the user. customer_id="me" returns someone else's account. Schema validation catches the type class and should run first as a deterministic gate. The semantic class, whether the argument captured what the user actually meant, needs a judge.
from pydantic import BaseModel, Field, ValidationError
class SearchFlightsArgs(BaseModel):
departure_airport: str = Field(pattern=r"^[A-Z]{3}$")
arrival_airport: str = Field(pattern=r"^[A-Z]{3}$")
departure_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
cabin: str = Field(pattern=r"^(economy|premium|business|first)$")
try:
args = SearchFlightsArgs.model_validate(predicted_args)
except ValidationError as e:
schema_errors = e.errors() # emit as span attribute, gate CI
Layer 3, result utilization. This is the layer almost every public post skips. The tool returned correctly, and then the agent ignored the payload. Three patterns show up over and over: it paraphrases the result with a number flipped (says "$54.00" when the payload said 4500 cents), it substitutes prior model knowledge instead of reading the result, or it uses the result correctly on turn 1 and drifts off it by turn 3. The rubric here is groundedness, except you point the context slot at the tool's return payload instead of a retrieved corpus.
from fi.evals import Evaluator
from fi.evals.templates import Groundedness, ContextAdherence, ChunkAttribution
from fi.testcases import TestCase
evaluator = Evaluator()
for tool_call in result.tool_calls:
# context = the actual tool payload, not the retrieved corpus
tc = TestCase(input=ex.user_message, output=result.response,
context=json.dumps(tool_call.result))
scores = evaluator.evaluate(
eval_templates=[Groundedness(), ContextAdherence(), ChunkAttribution()],
inputs=tc)
Layer 4, error recovery. The tool 4xx-ed or timed out, and the next move is the eval surface. Did the agent read the error body and retry with corrected arguments, or send the same broken string again? Did it fall back to another tool? Did it stop at a sensible cap (3 retries is a common floor; 6 usually means the loop guard is missing)? Did it communicate the failure, or fabricate success to hide it? This is a trajectory-level question. Per-call rubrics never see it.
Score those four separately and your vocabulary collapses from "the agent failed" to "the argument extractor regressed on date strings on the flight-booking path." One bisect instead of three days.
from fi.evals.metrics.agents import TrajectoryScore, AgentTrajectoryInput
from fi.evals.metrics.agents.types import AgentStep, TaskDefinition
trajectory = AgentTrajectoryInput(
trajectory=[AgentStep(action=s.action, tool_used=s.tool,
tool_args=s.args, tool_result=s.result,
error=s.error) for s in agent_steps],
task=TaskDefinition(goal=expected_goal, description=user_request),
available_tools=[t.name for t in registered_tools],
final_result=agent_response)
score = TrajectoryScore().compute_one(trajectory)
The math that makes this non-optional
Here is why per-turn green lights lie to you on multi-step agents.
End-to-end success on a k-step agent is roughly the product of per-step success rates. A 95-percent-per-step agent over eight steps lands near 66 percent. A 99-percent-per-step agent over eight steps lands near 92 percent.
Sit with that. Two thirds of sessions can end structurally wrong while every individual step scores green. That is not a hypothetical, it is the default math.
What has helped:
Score the trajectory as a unit, not just per step. The per-step rubric is the gate; the trajectory metric is the truth. Treat any agent longer than five steps as suspect, and push the planner to decompose into shorter sub-agents.
Keep a small consistency slice: take 30 hard cases, run each one 8 times, and track the fraction that pass all 8. When that number moves, the planner regressed, not the tools. On public benchmarks, use BFCL and tau-bench as the floor, not the gate.
BFCL tells you whether a model can call tools at all across syntactic, executable, and irrelevance-detection checks. Tau-bench measures multi-turn reliability across independent rollouts, and even GPT-4o lands below 25 percent at pass^8 on retail, which is a clean read on how much nondeterminism stacks across eight steps.
Neither tells you anything about your tool registry, your argument schemas, your error codes, or your business policy. The private eval set is the one that gates production. Build it stratified by tool, by argument-edge-case, and by error code, and promote failing production traces into it every week.
Three honest tradeoffs
This approach has real costs, and here are the three that matter most.
Per-layer scoring costs more than a single aggregate score, because you are running five rubrics per case instead of one. The payoff is that when CI fails, the failing layer name is the root cause. You can ship the deterministic layers first and turn on the judge layers once volume justifies it.
Groundedness on tool output is noisier than groundedness on a retrieved corpus, because tool payloads are JSON and the rubric reasons over fields, not prose. Pin a small human-labelled calibration set and re-tune it monthly.
The consistency slice is expensive: 30 cases times 8 rollouts is 240 agent runs per gate. Run it on release candidates, not on every PR.
Here is what it comes down to.
An aggregate task-completion score hides which layer broke. Scoring the four layers separately is what tells you what to fix this afternoon, because arguments, result use, and error recovery are each their own failure surface with their own root cause. Picking the right tool is necessary, but it is not sufficient. The other three layers are where the agent actually succeeds or fails.
