A booking agent's eval suite gives a run a trajectory score of 1.0. The run went wrong.
The agent searched twice with the wrong date, then booked with a field the booking tool does not accept. Switch the same check to strict mode and the score drops to 0.33.
The reason is structural. Every trace check makes scoring choices, and those choices fix the verdict before the check reads a span.
The match mode, how calls get paired, how arguments get compared and what the span captured are all choices. Pick them without thinking and the score answers a question nobody meant to ask.
Four modes answer four different questions
Trajectory matching compares the tool calls in a run with an expected path. The mode decides how much freedom the agent gets. Here is one common convention (some libraries flip the last two).
Strict. Same steps in the same order, scored by the matching prefix. Fits regulated or ordered workflows.
Unordered. Same set of steps in any order, scored by overlap. Fits independent lookups.
Subset. Every expected step appears, and extra steps are allowed. Fits agents that may add safe steps.
Superset. Every step taken was expected, and skips are allowed. Fits teams that forbid unexpected tools.
Pick the mode from the workflow. Write down why, next to the check.
Set-based modes cannot see a repeated call
In the common set-based definitions, the three looser modes compare sets. A set has no order and no duplicates, so a repeated or reordered call is invisible to them. Check whether your library counts duplicates.
That is how the booking run scored 1.0. The expected path was search, then book. The actual run was search, search, book.
As sets, both hold the same two tools.
Strict mode compares in order. Search matches search. The second search does not match book, so the prefix stops at one step out of three.
That gives 0.33.
If a duplicate call costs money or hits a rate limit, a set-based mode will not flag it.
Check what subset means in your library
Libraries do not agree on which way subset and superset point. In one library, subset means every expected step appeared. Another library uses superset for that same check.
Move a suite between the two and keep the labels, and a check meant to forbid unexpected tools can start allowing them.
Read the definition in each library's docs before you trust the name.
Partial credit divides by the longer list
Pass or fail hides near misses. Tool-call accuracy gives partial credit instead.
Full credit (1.0). The name and the arguments both match.
Half credit (0.5). Only the name matches.
No credit (0). Anything else.
The total is divided by the longer of the two lists. A duplicate call makes the actual list longer, so it lowers the score even when every expected call happened.
The pairing method matters too. Pair calls by position and the booking run scores 0.17, since only the first search earns half credit across three calls. Match each call to its best unused expected call and the same run scores 0.33.
The trace and the credit rule did not change, but the score did. Record which pairing your scorer uses.
Normalize arguments before you compare them
Plain equality fails JFK against jfk. It also fails a date against the same date with a time attached. Canonicalize case, dates and units first, then compare.
Choose the rules per field. Uppercasing suits airport codes. It can break IDs where case carries meaning.
Confirm the span holds the arguments
Every check above reads tool arguments from the trace. In the OpenTelemetry GenAI span conventions, still in development status, tool-call arguments and results are opt-in attributes.
With capture off, the schema check, the per-call check and the partial-credit score have nothing to read. Turn capture on first.
Then turn on format checking in the schema validator. JSON Schema treats "format": "date" as a note unless the validator enforces it.
Even an enforced format only proves the string is a valid calendar date. In the booking run, the wrong date was still a valid date, so the schema passed it. The per-call comparison against the expected call is what caught it.
Use invariants when there is no single right path
Trajectory checks need a reference path. Open-ended agents often have several valid ones.
Keep the invariants. No forbidden tool, and a step budget. Both hold on any valid route. Let an LLM judge score the route itself.
Check the end state. For tools with side effects, query the result after the run. Confirm one booking exists with the right flight and date. The check is deterministic, and it passes a valid alternative path that trajectory matching would fail.
Refresh the expected path. Update it whenever a tool changes.
Pitfalls that keep showing up
Picking a loose mode without a reason. A set-based mode passes duplicate and reordered calls without a word.
Trusting the label. Subset in one library can mean superset in the next.
Comparing raw strings. Case, dates and units fail equality on calls that were correct.
Scoring traces with no arguments. If capture is off, three of the checks have nothing to score.
Forcing one path on an open-ended agent. Use invariants and a state check where several routes are valid.
The full cost ladder, the runnable checks and the worked trace are in this guide to agent trace evaluation with code-based checks.
Decide the mode, the pairing and the normalization before the first run. Then a score of 1.0 means what you think it means.