Most voice AI demos look great. Then you ship to production and discover the system can't handle background noise, interrupts users mid-sentence, or adds latency that kills the experience.
Deepgram's recent acquisition of OfOne, a platform that achieved 95%+ order containment in actual drive-thru environments, offers insight into what it takes to build voice AI that works in the real world. Not in controlled demos. In production, where people order food with screaming kids in the back seat and truck engines idling.
Here's what their engineering team learned, and what it means for anyone building with voice AI.
The Acoustics Problem Nobody Talks About
Drive-thru environments are especially hostile to speech recognition. Car engines, wind noise, HVAC systems, multiple voices bleeding into the same microphone, and customers speaking with wildly different accents and speeds. Most legacy speech systems break down because they were never designed for unpredictable, overlapping, multi-source audio.
The solution isn't better microphones or noise cancellation. It's training models specifically on drive-thru acoustics and continuously adapting them using real operational data. Deepgram's approach couples domain-adapted models with a runtime that hot-swaps them as conditions change.
The system automatically routes audio to the most appropriate model for current conditions while keeping latency low across thousands of concurrent sessions. Generic ASR models trained on clean audio simply can't handle this level of acoustic chaos.
Why Interruptions Are Harder Than You Think
Traditional speech systems treat turn-taking as a separate problem layered on top of recognition. They rely on rules like "if there's silence for X milliseconds, the user is done talking." This works fine for dictation. It fails completely in conversation.
People pause mid-sentence. They think out loud. They restart ideas or trail off before finishing. A silence-based system can't tell the difference between "I want a..." (brief pause while thinking) and "I want a hamburger" (done speaking). The result is either premature interruption or awkward delays.
Flux, Deepgram's conversational speech model, makes turn-taking part of the speech recognition itself. As it transcribes, it evaluates conversational intent using both acoustic signals and language context. Is this a brief pause or the end of a thought? Is the speaker likely to continue or genuinely handing over the turn?
This lets the system prepare a response early when it detects the speaker might be finished, but immediately resume listening if they continue. Only when Flux is confident the turn has actually ended does it finalize the transcript.
The technical challenge is that mistakes are costly in both directions. Interrupt too early and you break trust instantly. Wait too long and the interaction feels sluggish and unnatural.
The Latency Tax of Multi-Provider Systems
Deepgram's Voice Agent API targets sub-500 millisecond end-to-end latency, including perception, LLM reasoning, and speech generation. Real-time voice demands perception in under 200 milliseconds and similarly fast time-to-first-audio-byte on generation.
Building with separate STT, LLM, and TTS providers introduces network hops, buffering delays, and pipeline fragmentation that typically add 1-3 seconds per turn. That's the difference between conversation that feels natural and conversation that makes users interrupt or give up.
Running the entire pipeline—streaming recognition, turn detection, TTS, and state management—within one runtime optimized for real-time workloads eliminates most of that overhead. This isn't about marginal improvements. It's about whether your voice interface is usable or not.
Implementation Reality Check
Most teams can get a proof of concept running within hours using the cloud API. The friction usually appears not in integrating Deepgram, but in surrounding application logic.
Teams often underestimate what's required to manage conversational state, handle interruptions, or orchestrate backend actions triggered by voice. The ones that struggle are typically trying to adapt existing chatbot or IVR codebases to real-time, turn-based audio. Those architectures assume synchronous, text-only interactions.
The adjustment isn't about the speech technology. It's about redesigning workflow to account for streaming audio, real-time decision-making, and fallbacks. Once teams make that shift, adoption accelerates quickly.
When Self-Hosting Makes Sense
Deepgram cloud typically delivers the lowest latency because it runs on optimized GPU clusters. Self-hosted deployments are architecturally identical, but may introduce additional network or hardware constraints.
Customers choose to self-host for compliance, data sovereignty, or isolation requirements—not for performance. It makes sense when audio must stay on-premises, when strict regional data governance applies, or when private VPC isolation is required at scale.
Because the Enterprise Runtime automatically distributes and manages models, self-hosting still benefits from the same adaptivity and resource efficiency as the cloud. Just within a controlled customer environment.
Migration from Existing Providers
Teams using basic STT APIs can typically switch by replacing a single API call and adjusting streaming logic. Deepgram supports open-source models like Whisper in the same runtime, allowing incremental migration—running legacy models during transition while adopting Deepgram models for high-value use cases.
Significant refactoring only occurs when moving from batch or request-response APIs to fully real-time streaming agents. But that refactor is required regardless of provider. Deepgram tends to reduce complexity because consolidating perception, turn logic, and generation into one platform eliminates large chunks of glue code and multi-provider orchestration.
The Domain Adaptation Lesson
Generic ASR models don't hold up in noisy, time-compressed environments where accuracy directly affects revenue. Drive-thru performance validated that domain-specific adaptation isn't optional—it's essential.
This applies equally in healthcare, customer support, and logistics. Models need tuning to specific terminology, acoustic conditions, and conversational patterns for each domain.
Another key insight: even high-accuracy systems need edge case handling. Deepgram's architecture allows seamless human intervention without collapsing the session. In drive-throughs, back-of-house staff can listen and make real-time corrections, just as they do today. Similar hybrid-supervision models are extending into clinical transcription, appointment scheduling, and contact centers.
Common Failure Modes
The most common production failures involve overlapping speech, inconsistent turn-taking, unexpected background noise, and user corrections ("actually, change that"). Traditional ASR models are brittle in these scenarios because they assume clear, single-speaker audio.
Another common failure is latency creep. As teams layer multiple providers, latency quietly inflates until the system feels unresponsive. Running entirely through a single streaming session under one real-time runtime prevents this.
Scalability failures like traffic spikes, concurrency bursts, inconsistent workloads are handled automatically by model hot-swapping and load distribution algorithms.
What This Means for Developers
Voice AI is moving from demos to production. The technical challenges aren't about recognition accuracy in clean audio anymore. They're about handling acoustic chaos, managing natural conversation flow, and maintaining sub-500ms latency under real-world conditions.
The gap between systems that work in demos and systems that work in production is domain-specific adaptation, conversational awareness, and architectural integration. Building with separate providers for each component introduces latency and complexity that breaks the user experience.
Teams shipping production voice AI need to think about acoustics, turn-taking, and latency from the start. Not as problems to solve later.