Most of us here have built a chatbot at some point, and going from text to voice looks like it should just be two more API calls, one on each end. It is not, and the reason is a single latency number that reorganizes the whole pipeline.
A voice agent runs the same loop every turn: capture audio, transcribe it, run it through a model, generate a reply, synthesize speech, play it back. That entire round trip has to land under about 500 milliseconds or the caller hears a gap and assumes the line dropped. A text chatbot can sit there for two or three seconds and nobody blinks, so most of the habits you carry over from text work are suddenly too slow.
That budget is why production voice stacks look the way they do. Speech recognition runs in streaming mode so transcription starts before the caller stops talking, which lets the language model begin working on a partial sentence instead of waiting for a complete one. Modern ASR from Deepgram, AssemblyAI and Google sits under a 5 percent word error rate on clear English, so accuracy is largely a solved problem. The engineering work is all about where the milliseconds go.
The other thing that eats the budget is end of turn detection. Voice activity detection has to decide when you have actually stopped speaking, and that call happens before the model sees a single token. Cut it short and you clip people mid sentence, wait too long and your real time agent feels like a phone tree.
If you want the full pipeline breakdown, the open source options like LiveKit, Pipecat and Vocode, and where the cost actually lands, AI voice agents covers it end to end.
The cost comparison surprised me more than the latency did. A US call center seat runs $25 to $65 an hour once you count benefits, training and management overhead, while a voice agent handling the same calls costs $0.05 to $0.25 a minute with no ramp up and unlimited concurrency. That gap is why enterprise production deployments grew 340 percent year over year through early 2026.
Curious what people here have hit in practice. If you have shipped a voice agent, where did your latency actually go, the ASR, the model, or the TTS?