Voice Agents: Latency, Turn-Taking and How to Design the Conversation
A voice agent is AI that talks by audio and takes actions. Its design depends on latency, on spotting when users finish speaking and on handling interruptions.
What happens in each turn?
A voice turn passes through several stages: the network carries the audio, a speech recognition model turns it into text, an LLM generates the reply, and a speech synthesis system turns it back into voice. Each stage adds time. A 2026 Future AGI guide that breaks down the budget places the network at 30 to 50 milliseconds, the first recognition text at 100 to 150, the model’s first token at 200 to 300 and the first synthesized audio at 80 to 150. The sum explains why the target is hard to hit.
To see why the first token weighs so much, review latency and tokens per second.
What latency is acceptable?
These figures come from voice vendors, so treat them as guidance, not norms:
- A Future AGI guide points to an average below 350 milliseconds and a 95th percentile of 500 for sales and support agents, and warns that sub-500 is the wrong target for clinical intake or financial planning.
- A Retell guide claims that below about 600 milliseconds end to end, callers stop noticing they are talking to an AI. It is a vendor claim with no study behind it.
- An Acquaintsoft guide proposes under 700 milliseconds and says that going past 800 or 900 starts to feel like a bad line.
The useful thing is not the exact number but the test: if the silence between turns is noticeable in a real conversation, latency is too high for that case.
How do you design the turn, and what is barge-in?
The central problem is knowing when the user has finished. If the agent replies too early, it cuts the person off. If it waits too long, an awkward silence appears.
Real-time voice APIs expose this setting. OpenAI’s Realtime API, for example, offers two turn-detection modes: server_vad, which decides by silence and lets you tune the volume threshold and the milliseconds of silence (silence_duration_ms), and semantic_vad, which uses a classifier on the user’s words to judge whether they are done and has an eagerness parameter (how soon it replies). Other providers have equivalent controls under other names.
Barge-in is the ability to interrupt the agent while it speaks. A person who says “wait, that’s not what I meant” expects the agent to stop immediately. Future AGI describes a five-part pipeline: voice activity detection, an interruption trigger, flushing pending audio, cancelling the LLM generation and preserving context. It proposes a combined budget under 150 milliseconds to detect the interruption, stop the audio and cancel the LLM.
A reasonable turn design:
- Confirm long utterances with a short pause before replying.
- Allow interruptions at any point in the response.
- If the agent is going to take time, say something brief: “Let me check that.”
- Summarize at the end what you understood, so the person can correct it.
- Offer a route to a human if the conversation gets stuck.
Example
Illustrative example: a clinic’s voice agent checks availability to book an appointment. At first it replies as soon as it hears a 300-millisecond pause, and it cuts off patients who read out numbers with pauses. The team raises the end-of-turn threshold, adds “let me check the schedule” when the lookup takes more than a second, and tests with ten recorded calls. Interruptions drop, and patients confirm the time without repeating themselves. The figures are invented to illustrate the method.
How do you test a voice conversation?
Record ten real or simulated calls with different people: someone who speaks fast, someone who stops to think, someone with background noise and someone who wants to interrupt mid-sentence. For each one, mark three moments: when the agent started replying, whether it cut the person off, and how much silence came before the reply. That list is worth more than an average figure, because it shows the situations where the turn design fails.
Repeat the test after every change to a threshold or a model. A tweak that improves the pause for one person can make interruptions worse for another, and you will only notice if the test covers different cases.
When do you hand the call to a person?
Define the transfer conditions before launch, not during an incident. Three triggers are usually enough: the person asks to speak with someone, the agent fails to understand the intent after two attempts, or the topic is sensitive, such as a medical complaint or a disputed charge. In any of those cases, the agent should say it is transferring, summarize what has already been said so the person does not repeat anything, and pass the context to whoever takes the call.
Measure how many calls are transferred and for what reason. If most arrive through the same trigger, that topic is the first candidate for improvement.
Common mistakes
- Measuring latency only by the average, without looking at the high percentiles.
- Designing turn-taking without testing with people who speak slowly or pause.
- Not allowing interruptions.
- Leaving long silences with no sign that the agent is still working.
- Using voice for information a person needs to see, such as a long list of times.
When voice is NOT the right channel
When the content is long, comparative or needs review, such as a contract or a price list. Also when the environment is noisy or the user cannot speak. In those cases a text interface or a mix of voice and screen works better. For voice interfaces built on language models, review LLM voice interfaces and on-device AI.