How to Build a Real-Time AI Avatar Pipeline for the Web: Streaming, State, Latency and Safe Handoffs

Question

Build a responsive conversational avatar with explicit state, typed events, interruption handling, end-to-end telemetry, and a reliable path to human support.

A talking digital human may look like a single interface, but the user is actually interacting with a distributed system. One turn can involve microphone capture, voice activity detection, streaming transcription, retrieval, model reasoning, policy checks, speech synthesis, lip synchronization, character animation, rendering, analytics, and business actions.

The difficult part is not making each component work in isolation. The difficult part is keeping them synchronized while the user changes direction, interrupts, loses connectivity, requests a human, or receives a delayed packet from an older response.

This guide presents a practical real-time AI avatar architecture for the web. It uses TypeScript examples to show how to model avatar state, reject stale events, measure latency, coordinate speech with animation, and transfer a conversation safely to a human team.

Note: The latency values shown in the diagrams and demo are illustrative engineering budgets. They are not product benchmarks or universal targets.

Table of Contents

1. Why a Real-Time AI Avatar Is a Distributed System
2. Reference Architecture for a Browser-Based AI Avatar
3. Choosing WebRTC, WebSockets, or Both
4. Modeling the Avatar as a State Machine
5. Creating a Typed Event Protocol
6. Handling Interruption and Barge-In
7. Measuring the Full Latency Budget
8. Synchronizing Speech, Visemes, and Animation
9. Separating AI Reasoning From Avatar Behavior
10. Designing Safe Human Handoffs
11. Common Production Failures
12. Validating the Reference Implementation
13. Frequently Asked Questions
14. Conclusion

Why a Real-Time AI Avatar Is a Distributed System

A conventional chat interface can wait for a complete text response and render it all at once. A conversational avatar cannot. It has to manage several timelines at the same time:
· The user is speaking.
· The transcription service is producing partial text.
· The agent is retrieving knowledge or calling tools.
· The speech service is producing audio chunks.
· The animation system is receiving visemes and expression cues.
· The renderer is displaying the character at an interactive frame rate.
· A CRM, ticketing system, or human queue may be waiting for the result.

A delay or ordering error in any one of these layers can break the illusion of presence. The avatar may continue talking after the user interrupts. Lip sync may keep moving after the audio has stopped. A late answer may overwrite a newer turn. A handoff may transfer the transcript but lose the customer’s current task.

The architecture therefore needs three properties from the beginning:

1. Explicit state, so every event is interpreted in context.
2. Versioned responses, so stale audio and animation can be rejected.
3. End-to-end observability, so latency and ordering problems can be traced across services.

Tip: Treat the avatar as a coordinated set of media, AI, animation, and workflow services. Do not let one large language model become the implicit controller for the entire experience.

Reference Architecture for a Browser-Based AI Avatar

A maintainable implementation separates the system into four layers: the browser client, the real-time transport, the conversation orchestrator, and business systems.

Figure 1. Separate browser, transport, orchestration, and business-system responsibilities so each layer has clear ownership.

Browser Client

The browser is responsible for the user-facing interaction:
· Capturing microphone input
· Detecting when speech starts and ends
· Playing streamed audio
· Rendering the 3D character
· Applying facial animation, visemes, gaze, and gestures
· Showing connection, listening, thinking, speaking, and handoff states

Keep the browser thin. It should render approved events and collect input, but it should not contain sensitive business logic or broad tool permissions.

Real-Time Transport

The transport layer carries audio, control events, and connection state. WebRTC defines browser APIs for sending and receiving media and application data in real time.1 The WebSocket standard defines a browser interface for bidirectional communication with server-side processes.2
A common pattern is:
· WebRTC media for microphone and synthesized audio
· RTCDataChannel or WebSocket for transcripts, state changes, visemes, cancellations, and telemetry
· HTTPS for configuration, authentication, and non-streaming business actions

There is no universal transport choice. The correct design depends on server topology, media services, browser support, firewall behavior, and operational complexity.

Conversation Orchestrator

The orchestrator owns the session and coordinates services. Its responsibilities usually include:
· Streaming speech-to-text
· Turn detection
· Session state
· Retrieval and tool routing
· Policy validation
· Streaming text-to-speech
· Response IDs and cancellation
· Human handoff
· Logging and metrics

This layer is the authority for whether a response is still valid. The browser may request cancellation, but the orchestrator must propagate it to model generation, tools, speech synthesis, and downstream event queues.

Business Systems

The avatar becomes operationally useful when it can access narrowly scoped business systems, such as:
· Product knowledge
· CRM records
· Scheduling
· Customer support tickets
· Authentication
· Analytics
· Human agent queues

Every integration should follow least privilege. A lead-qualification avatar may need to create a meeting request, but it should not have general write access to the entire CRM.

Choosing WebRTC, WebSockets, or Both

The transport decision should be based on the data you are carrying, not on which technology sounds more “real time.”

Use WebRTC When Media Is the Primary Workload

WebRTC is a strong fit when the browser needs continuous microphone or audio streaming, adaptive media transport, or a data channel coupled to the same session.

Typical WebRTC responsibilities include:
· Upstream microphone audio
· Downstream synthesized speech
· Network adaptation
· Optional data-channel events
· Connection statistics

Use WebSockets for a Server-Centric Control Plane

WebSockets are useful for structured bidirectional messages between the browser and an application server. They are often simpler for:
· State changes
· Partial and final transcripts
· Tool progress
· Response cancellation
· Viseme events
· Handoff updates
· Session telemetry

If you stream binary audio over WebSockets, monitor queue growth and define backpressure behavior. The WebSocket interface exposes bufferedAmount, which reports application data queued but not yet transmitted.

Use Both When the Responsibilities Are Clear

A hybrid design is often practical:

WebRTC:microphone and synthesized audio
WebSocket: JSON control events, state, tools, and telemetry
HTTPS: authentication, configuration, and durable business actions

The risk is not using multiple protocols. The risk is giving them overlapping ownership. For example, do not let both WebRTC and WebSocket messages independently decide when a response has ended. Choose one authoritative event.

Modeling the Avatar as a State Machine

A state machine makes turn-taking visible and testable. Without one, state is spread across audio callbacks, UI components, network handlers, and model promises.

Figure 2. Explicit state and response IDs prevent stale audio or animation from reaching the user.

A useful minimum set of states is:

type AvatarState =
>| "IDLE"
>| "LISTENING"
>| "THINKING"
>| "SPEAKING"
>| "INTERRUPTED"
>| "HANDOFF"
>| "ERROR";

These states answer practical questions:
· Should microphone audio be accepted now?
· Is an incoming audio chunk still valid?
· Can a tool call continue?
· Should the mouth be moving?
· What should the user interface display?
· Is the human team now responsible for the session?

The state machine should have invariants. The most important one is:

A media or animation event is valid only when its > responseId matches the active response.

This rule prevents a late packet from a cancelled response from reaching the user.

Creating a Typed Event Protocol

Events should be explicit, versionable, and serializable. A discriminated TypeScript union gives the browser and server a shared contract.

type AvatarEvent =
| { type: "USER_SPEECH_STARTED"; timestamp: number }
| { type: "USER_SPEECH_ENDED"; timestamp: number }
| {
type: "AGENT_RESPONSE_STARTED";
responseId: string;
timestamp: number;
}
| {
type: "AUDIO_CHUNK";
responseId: string;
sequence: number;
timestamp: number;
}
| {
type: "VISEME";
responseId: string;
value: string;
timeMs: number;
timestamp: number;
}
| {
type: "AGENT_RESPONSE_ENDED";
responseId: string;
timestamp: number;
}
| {
type: "HANDOFF_REQUIRED";
reason: string;
timestamp: number;
};

In production, add these fields to every event envelope:

· sessionId
· traceId
· eventId
· schemaVersion
· responseId, when applicable
· sequence, for ordered streams
· createdAt

The > responseId handles cancellation. The traceId handles observability. The schemaVersion allows the client and server to evolve without silently misinterpreting events.

Rejecting Stale Events

The event handler should accept audio and animation only for the active response:

private isCurrent(responseId: string): boolean {
   return this.activeResponseId === responseId;
 }

 case "AUDIO_CHUNK":
   if (!this.isCurrent(event.responseId)) {
 this.rejectedStaleEvents += 1;
 return;
   }

   this.acceptedAudioChunks += 1;
   return;

Count rejected stale events. A rising count can reveal network reordering, ineffective cancellation, or a service that keeps publishing after it has been stopped.

Handling Interruption and Barge-In

Barge-in occurs when the user begins speaking while the avatar is still talking. It is a normal conversational behavior, not an edge case.

A correct interruption sequence should:

1. Detect new user speech.
2. Stop local audio playback immediately.
3. Stop visemes and gesture playback.
4. Cancel the active response on the server.
5. Cancel associated tool calls when safe.
6. Reject late events for the cancelled > responseId.
7. Return the interface to > LISTENING.

A simple state transition looks like this:

case "USER_SPEECH_STARTED":
   if (this.state === "SPEAKING") {
 this.cancelActiveResponse();
 this.state = "INTERRUPTED";
   }

   this.state = "LISTENING";
   return;

The local client should stop playback before waiting for a network acknowledgment. Otherwise the avatar may continue talking while the cancellation request is in flight.

Caution: Stopping the speaker is not enough. The application must also stop mouth movement, gestures, queued text-to-speech chunks, and any autonomous tool work that no longer applies.

Make Cancellation Idempotent

Cancellation messages may be duplicated or arrive after completion. The server should treat repeated cancellation as safe:

function cancelResponse(responseId: string): void {
   if (responseId !== activeResponseId) return;

   modelStream.abort();
   speechStream.abort();
   toolController.abortSafeOperations();
   activeResponseId = null;
 }

Some business operations should not be blindly aborted. A payment capture, ticket creation, or database write may need a compensating action or durable workflow. Separate conversational cancellation from transactional consistency.

Measuring the Full Latency Budget

“Avatar latency” is not one number. It is a sequence of delays that should be measured independently.

Figure 3. Measure each processing stage and the slow tail, not only the average end-to-end turn time.

A useful trace includes:

speech_started
speech_ended
transcript_final
agent_started
retrieval_finished
policy_finished
first_text_token
first_audio_chunk
audio_playback_started
first_viseme_applied
response_finished

The most useful user-facing metrics are often:
· Speech end to first meaningful audio
· Speech end to visible avatar response
· Barge-in to silence
· P50 and P95 turn latency
· Reconnect recovery time
· Stale-event rejection count
· Human-handoff acceptance time

A small tracing utility can instrument the pipeline:

class LatencyTrace {
   private readonly marks = new Map<string, number>();

   mark(name: string): void {
 this.marks.set(name, Date.now());
   }

   duration(start: string, end: string): number {
 const startTime = this.marks.get(start);
 const endTime = this.marks.get(end);

 if (startTime === undefined || endTime === undefined) {
   throw new Error(`Missing latency mark: ${start} or ${end}`);
 }

 return endTime - startTime;
   }
 }

Use one traceId across browser telemetry, transcription, retrieval, model inference, speech synthesis, and rendering. Without cross-service correlation, teams tend to optimize the service they can see rather than the stage that is actually slow.

Optimize for Time to First Useful Response

Streaming every possible intermediate event is not always helpful. Partial transcripts can reduce delay, but unstable partial text may trigger the wrong tool. Early speech chunks can improve responsiveness, but they are harder to cancel cleanly.

A balanced strategy is:
· Stream partial transcripts to the UI.
· Wait for a stable intent before executing consequential tools.
· Stream speech after the response passes required policy checks.
· Buffer enough audio to avoid gaps, but not so much that interruption feels slow.
· Start neutral listening and thinking animations immediately, then apply semantic gestures only after intent is stable.

Synchronizing Speech, Visemes, and Animation

Audio and facial animation must share a timeline. If audio uses the browser clock while visemes use arrival time, lip sync will drift whenever network conditions change.

Each viseme event should include its intended offset from the start of the current audio response:

interface VisemeEvent {
   responseId: string;
   value: string;
   timeMs: number;
   durationMs?: number;
 }

When playback begins, store a reference time:

const playbackStartedAt = audioContext.currentTime;

 function scheduleViseme(event: VisemeEvent): void {
   if (event.responseId !== activeResponseId) return;

   const executeAt = playbackStartedAt + event.timeMs / 1000;
   faceController.schedule(event.value, executeAt);
 }

Use a Shared Terminal Event

The animation controller should reset on every terminal condition:
· Response completed
· Response cancelled
· Connection lost
· Handoff started
· Pipeline error
This prevents frozen mouth poses, gestures that continue after speech, and facial expressions from leaking into the next turn.

Do Not Drive Every Gesture From the Language Model

A model can suggest intent, but a behavior controller should choose from a limited, reviewed library. For example:

interface AvatarBehavior {
   expression: "neutral" | "friendly" | "concerned";
   gesture?: "greet" | "explain" | "confirm";
   speakingStyle: "brief" | "standard" | "detailed";
 }

This keeps character performance consistent and makes behavior testable. It also prevents untrusted model output from becoming an arbitrary animation command.

Separating AI Reasoning From Avatar Behavior

A production pipeline should distinguish between four kinds of output:

1. Conversational content, what the avatar says.
2. Business actions, what systems the agent may call.
3. Presentation behavior, expression, gaze, and gesture.
4. Safety decisions, whether to answer, refuse, clarify, or hand off.

If one prompt controls all four, a small change can have unpredictable effects. A safer pattern is to validate each output separately.
User input
-> intent and context
-> retrieval and tool proposal
-> policy validation
-> approved response text
-> approved avatar behavior
-> speech and animation streams

For example, the language model may classify the response as “explanation.” The behavior layer can map that to a controlled gesture set. It should not accept free-form instructions such as “perform any animation named by the model.”

NIST’s AI Risk Management Framework is intended to help organizations incorporate trustworthiness into the design, development, use, and evaluation of AI systems.3 For a real-time avatar, trustworthiness includes not only the answer, but also what the agent can do, what data it can access, and how clearly it signals uncertainty or transfer.

Designing Safe Human Handoffs

A handoff is a first-class user journey. It should preserve context and make responsibility clear.

Figure 4. A safe handoff preserves context, stops autonomous work, and makes ownership visible.

Define Handoff Triggers

Typical triggers include:

· The user requests a person.
· The agent has low confidence.
· The question crosses a legal, medical, financial, or policy boundary.
· Authentication is required.
· A high-value opportunity needs a specialist.
· The same issue has failed repeatedly.
· The user shows frustration or distress.

Transfer a Structured Handoff Package

Do not forward only the raw transcript. Send a structured payload:

interface HandoffPackage {
   sessionId: string;
   traceId: string;
   reason: string;
   summary: string;
   transcript: Array<{ role: "user" | "assistant"; text: string }>;
   currentTask?: string;
   completedActions: string[];
   pendingActions: string[];
   authenticatedUserId?: string;
 }

The human team should know what the user wants, what the avatar already did, and what remains unresolved.

Stop Autonomous Work Before Transfer

Before handoff:

· Cancel the active response.
· Stop nonessential tools.
· Prevent new autonomous actions.
· Persist the session summary.
· Notify the user that a transfer is in progress.
· Confirm when a human has accepted ownership.

The OWASP Top 10 for LLM and generative AI applications highlights risks including prompt injection, improper output handling, excessive agency, and sensitive-information disclosure.4 These risks are directly relevant when an avatar can call tools or transfer customer data.

Caution: Do not include secrets, hidden prompts, unrestricted tool output, or unnecessary personal data in the handoff package. Transfer only the information required for the human to continue the task.

Common Production Failures

The following failures are common because each one crosses service boundaries.

Tip: Create automated tests for cancelled responses, duplicated events, out-of-order packets, lost connections, and repeated handoff requests. Happy-path tests are not enough for a streaming interface.

Validating the Reference Implementation

The included TypeScript demo models:
· State transitions
· Active response IDs
· Barge-in cancellation
· Rejection of stale audio
· Handoff state
· Stage-level latency marks

It was compiled in TypeScript strict mode and executed with Node.js. The displayed millisecond values come from deliberately short local delays and are only used to confirm that the instrumentation works.

Figure 5. The TypeScript demo compiled in strict mode and passed cancellation, stale-event, handoff, and latency checks.

The full source file is included with this article package as code/avatar_pipeline_demo.ts.
To validate it locally:

npx tsc --strict --target ES2022 --module CommonJS \
   --outDir dist avatar_pipeline_demo.ts

node dist/avatar_pipeline_demo.js

Expected final line:

All assertions passed.

Note: When posting on CoderLegion, paste code as selectable text and apply the editor’s code formatting button. Do not upload the code snippets as images.

Frequently Asked Questions

1. Should a real-time AI avatar use WebRTC or WebSockets?
   Use WebRTC when continuous media transport is central. Use WebSockets for a server-centric event and control plane. Many systems use both, with clearly separated ownership.

2. How should an avatar handle interruption?
   Stop local playback immediately, cancel the active server response, stop associated animation, and reject every late event carrying the cancelled response ID.

3. Do lip-sync systems need phonemes or visemes?
   The renderer usually needs visemes or blend-shape targets, while the speech system may produce phoneme timing. Use an explicit mapping and schedule the result against the same audio clock.

4. Where should conversation memory be stored?
   Store durable memory on the server under a defined retention policy. Keep only short-lived interaction state in the browser, and avoid exposing sensitive memory to client code.

5. When should the avatar transfer to a human?
   Transfer when the user asks, confidence is low, policy boundaries are reached, authentication is required, repeated attempts fail, or human judgment is more appropriate.

6. What should teams measure first?
   Start with speech-end to first meaningful audio, barge-in to silence, P95 turn latency, stale-event rejection, error recovery, and handoff acceptance time.

Conclusion

A real-time AI avatar is a coordinated media and software system, not a talking model attached to a face.

Use an explicit state machine so every event has a valid context.

Assign each response a unique response ID and reject stale audio or animation after cancellation.

Measure latency by stage, including interruption recovery and slow-tail behavior.

Synchronize speech, visemes, and gestures against a shared playback timeline.

Keep AI reasoning, business actions, avatar behavior, and policy decisions in separate control layers.

Design human handoff as a normal, observable workflow that preserves context and stops autonomous work safely.

References
1 W3C, “WebRTC: Real-Time Communication in Browsers,” W3C Recommendation, March 13, 2025. https://www.w3.org/TR/webrtc/

2 WHATWG, “WebSockets Standard.” https://websockets.spec.whatwg.org/

3 National Institute of Standards and Technology, “AI Risk Management Framework.” https://www.nist.gov/itl/ai-risk-management-framework

4 OWASP GenAI Security Project, “2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps.” https://genai.owasp.org/llm-top-10/

Author Bio

Mimic Minds Editorial Team

Mimic Minds develops conversational AI avatars, embodied digital humans, and real-time intelligent interfaces for enterprise and interactive applications.