Question

Building AI Voice Agents? Here is where your architecture would break.

1. When a user can update the UI using voice

2. User clicks and speaks at the same time

3. Adding voice to legacy applications

4. Asking Agents to make a phone call

You must have observed recently that AI models (Realtime voice models) have improved significantly. 

They sound very natural.

This is because other than understanding words, they now also understand para-lingusic characteristics like

• Tone

• Pitch

• Pauses

However, building AI voice Agents is more than just adding Realtime voice models. Let's talk about the challenges.

Challenge 1 – Taking Action Using Voice

This is where the user is not just having a voice conversation with the AI, but is also asking it to perform actions in the application.

For example, playing a Tic-Tac-Toe game with an agent using voice.

Now the challenge becomes synchronisation.

• The AI Agent can update the UI.

• The application must notify the AI whenever the user acts.

• The user must immediately see every action performed by the AI.

Without proper synchronisation, the AI and the application quickly go out of sync.

Solution: Industry Best Practice 1

First, identify all the actions the AI Agent is allowed to perform in the application.

For example:

• Show the board

• Place a mark

• Reset the game

• Update the user's move

Let AI not invent new actions. It can only choose from this predefined set.

Next, identify all the user actions that the AI needs to know about.

For example:

• User placed a mark

• Board synchronized

Now, whenever one of these actions occurs, the application simply triggers the corresponding function.

This significantly reduces the chances of inconsistent state.

Challenge 2 – User can speak or click or both

Voice means the user has multiple modalities to interact with

So, now the user can:

• Speak: "Place X in the center."

• Click on the board.

• Or both

Best Practice 2 – Use a Single Function

Whether the move comes from voice or from a mouse click, don't write two different pieces of logic.

I

nstead, write a single function. 

Call the same function for both actions

When the user clicks a cell, we call a common function, for example: applyMark().

Similarly, when the AI understands the spoken command, it also calls the same applyMark() function.

So regardless of whether the input comes from voice or a click or both, the application follows the same path.

This keeps the behaviour consistent and avoids duplicate logic.

One Function, One Source of Truth

Check out the complete article with code and images here: https://www.aimletc.com/industry-best-practice-building-ai-voice-agents/