How to evaluate a voice agent before launch with simulation
Evaluating a voice agent used to mean listening to recordings, then it meant scripts that replayed transcripts at the agent. Neither catches much.
The approach that works now is to simulate thousands of synthetic callers placing real audio calls, score every turn, and use turn-level localization to find where each failure happened. Done well, a pre-launch cycle that took three months lands in about three weeks.
The loop in five steps
- Define the agent: its job, its limits, when it hands off to a human.
- Pick or write the personas that will call it.
- Auto-generate branching scenarios from a plain-text description.
- Run the whole matrix and score every turn.
- Triage by the exact turn each failure landed on, then wire it into CI.
Personas drive the calls
A scenario is only as realistic as the caller driving it, so personas come first.
A good setup ships around 18 caller archetypes: first-time and confused, repeat and in a hurry, frustrated, hard of hearing, non-native speaker, interrupter, hostile boundary-pusher, and so on. You author your own on top of those.
The custom controls that matter are gender, age range, location, accent, speaking speed, background noise, and language. But the field that actually changes behavior is the free-form instruction. "Has been on hold 20 minutes before reaching the agent" or "has had the same policy for 30 years and gets confused by any process change" makes a persona behave in a specific way you can measure against, instead of a generic caller.
You describe scenarios, you don't write them
You do not hand-write hundreds of conversations. You describe the call in plain text and pick a size.
For an insurance agent, that description might be: "customer calls for an auto quote; some are first-time, some are switching, some renewing; the agent qualifies them, quotes, handles objections, and closes or books a follow-up." Then pick 20, 50, or 100 rows.
The generator turns that into the branching conversation paths, spreads your personas across them, varies the situation (clean record vs violations, one car vs several), and sets a success condition per branch. A 100-row description usually becomes about a dozen distinct paths and a few hundred individual runs.
Check the branch coverage before you run. If the generator leaned too hard on the price-objection branch, rebalance so the other objections get covered too.
Auto-generation is the default, not the only source. You can also seed the same run from real calls uploaded as CSV or JSON, from a synthetic dataset, or from hand-written scripts, which helps when you already have production calls or a fixed script to cover.
Running the matrix
A run is scenarios times personas times rubrics. A hundred scenarios across ten personas and five rubrics is 5,000 scored pairs, which finishes in a few hours at moderate concurrency.
For voice, start with five rubrics: did the call resolve the caller's goal, did the agent complete its task, was the transcription accurate, was the audio clean, and was the tone right (polite, helpful, concise). Add a tool-calling rubric if the agent calls functions.
If your agent is not on a natively supported provider, place the calls over real telephony to its phone number. The codec degradation there affects speech recognition in ways software-only playback does not, which matters when accents are part of what you care about.
Turn-level localization is the payoff
When a simulated call fails, the useful question is where in the conversation it went wrong, not just that it failed. A conversation is a series of turns, each one a caller message and the agent's reply.
Turn-level localization tags every failed call with the turn that caused the failure, then counts those turns across all the failures. That count is the point. In a 5,000-call run that fails 15 percent of the time, you would otherwise be reading 750 failed transcripts by hand, well over a day of work. Instead you see that most failures land on the same one or two turns, and the scorer writes a short reason for each, so you get the where and the why together.
Here is what that looked like on the insurance agent. Across the calls, 76 percent resolved the caller's goal, below the 80 percent bar set for launch. Grouped by the turn where they failed, three patterns covered most of it:
- Turns 5 to 6, 41 percent of the failures: when quoting a price, the agent used jargon like "deductible-aggregate" that first-time shoppers could not follow. Fix: detect a first-time caller and quote in plain English.
- Turns 8 to 9, 29 percent: the agent's only response to a price objection was "this is the best value," and callers pushed past it. Fix: give it a few escalating rebuttals before it gives up.
- Turns 3 to 4, 22 percent: the agent asked more than a dozen qualifying questions and callers hung up. Fix: cut to the six questions that actually change the quote and ask the rest later.
Each fix is specific because the failure was tied to a turn and a reason, not guessed at from reading transcripts. After the three fixes and a re-run of the same calls, resolution rose to 89 percent, past the gate.
Turning it into a CI gate
The last step turns this from a one-off pre-launch exercise into a gate.
A programmatic API lets you run a 50-scenario smoke run on every PR against an 85 percent threshold in about 20 minutes, and block the merge if it drops. Run the full few-thousand-scenario matrix on each release candidate. Run a nightly sample against a rolling baseline to catch drift from a model or dependency change.
Slice the results by persona, by branch, and by failing turn, and you can see whether the agent breaks on a caller type, a conversation path, or a specific step.
Pre-launch quality is a measurable loop now. Describe the call, generate the personas and paths, run at scale, and let the turn each failure lands on tell you what to fix. Start with resolution and task completion on a small matrix, then grow the persona and scenario library as the agent changes.