LLM Voice Interfaces: Latency, Turns and Errors
A voice interface built on an LLM fails because of silence, not the answer. Principles for designing turns, perceived latency, interruptions and errors.
Why is voice different from a screen?
On a screen, a one-second delay is tolerated. In a conversation, silence means something: the user thinks they weren’t heard, that the line dropped or that they must repeat themselves. Picovoice gives the human reference: the median gap between speakers is roughly 200 milliseconds across 10 languages (Stivers et al., 2009), and agents are judged against that rhythm.
A voice agent chains several stages: audio capture, end-of-turn detection, transcription, model reasoning and speech synthesis. Each stage consumes time, and the user only perceives the sum. That is why latency is budgeted per stage, not solved with a single change. For where a model’s time goes, see latency and tokens per second.
How do you design turns?
Fixed silence timeouts fail in both directions: they cut off a user who paused to think, or they add dead air when the user has already finished. Picovoice’s alternative combines three signals: acoustic (voice activity detection), linguistic (whether the partial transcript sounds like a complete utterance) and behavioral (a tiered confidence policy). At medium confidence the agent starts reasoning speculatively but does not speak; at high confidence it commits. If the user resumes talking, the speculative work is discarded silently.
Salesforce recommends putting the question at the very end of each agent turn, with nothing after it, and yielding immediately when someone interrupts, treating what they say as the new turn. For interruptions (barge-in), Picovoice describes two layers: stop or mute outbound audio and flush the playback buffer, and cancel everything in flight (pending generation and queued synthesis), so a stale answer doesn’t surface after the user has moved on.
How do you handle perceived latency?
There are two kinds of latency: measured (in milliseconds) and perceived (what the user feels). They can be managed separately:
- Overlap stages: start reasoning on partial transcripts and synthesize speech from the first tokens.
- Speak early: a brief, truthful acknowledgment such as “Let me check” while the data loads makes the wait feel conversational.
- Never talk over the user: if they start speaking, the agent stops and listens.
The acknowledgment has to be true. Saying “Let me check” when the system isn’t looking anything up breaks trust on the next turn.
How do you recover from errors?
Salesforce proposes recovering by degrees, using the weakest cue that can locate the problem. First, a general cue asking for a repeat (“Sorry?”). Then, a question that names the trouble spot (“I didn’t catch the date”). Then, offering a specific guess for the person to verify (“Did you mean Tuesday the 14th?”). Don’t repeat the same failed prompt unchanged, and don’t frame the error as the caller’s fault. For names and long numbers, switch to spelling or digit-by-digit entry. And if failures pile up across the call, proactively offer a transfer to a person, passing along the transcript so they don’t have to repeat themselves.
The order matters because voice has limited working memory: the user can’t scroll back or look at a list. That’s also why you should keep lists to about three options (Salesforce suggests headlines first, details on request) and reserve explicit confirmation for hard-to-reverse actions such as a booking or a payment; for ordinary data, echoing it inside the next question is enough.
This ties into handling AI errors and recovery in any interface: the system should say what failed and what the person can do. To decide how much the agent should confirm before acting, see trust calibration and action approval.
What do real-time voice APIs let you configure?
Model platforms now expose turn-taking and interruption decisions as parameters. Two examples from their official documentation, checked in November 2026:
- OpenAI Realtime API. Voice activity detection (VAD) is on by default and is configured with
turn_detection, for example withsemantic_vad. When the user speaks, the API cancels the response in progress and starts a new one. Over WebRTC or SIP the server truncates audio that was never played; over WebSocket, your client must stop playback and sendconversation.item.truncatewith how many milliseconds the person actually heard. For push-to-talk, setturn_detectiontonulland control turns manually. - Gemini Live API. Automatic VAD is configured under
realtimeInputConfig.automaticActivityDetection, with start-of-speech and end-of-speech sensitivity andsilence_duration_ms; the guide recommends 500 to 800 ms, because lower values split utterances and higher values add latency. It can also be disabled so you mark turns withactivityStartandactivityEnd. When it detects an interruption, the server cancels generation and sends aninterruptedmessage so the client stops and clears its audio.
The design lesson is that truncation is not a technical detail: if the client keeps playing stale audio, or the history keeps sentences the person never heard, the agent “remembers” things the user did not hear. Also check parameter and model names against current documentation, because they change between versions.
How do you make it accessible?
A short listening timeout hurts people who speak slowly, search for words or have a speech disability. WCAG 2.2.1 (Timing Adjustable, Level A) asks that, for time limits set by the content, people can turn them off, adjust them to at least ten times the default, or extend them after a warning that gives at least 20 seconds to do so. WCAG does not mention voice interfaces and applies to web content, but the idea works as a design criterion: let users lengthen the wait, warn before cutting off listening, and always offer a text alternative.
Example
Illustrative example: a voice assistant for booking a table takes almost two seconds to check availability. Without design, there is silence, the user speaks again, and that cancels the lookup. With design, the agent says “Looking for tables at eight” in under 300 milliseconds, holds the turn while it checks, and if the user interrupts to change the time, it stops the previous lookup and answers the new request. Before confirming the booking, it repeats the date, time and party size, because a booking is a high-stakes action.
This article covers the interaction principles; for the full agent that carries out tasks by voice, see Voice agents.
Common mistakes
- Measuring only the model’s latency and forgetting end-of-turn detection and speech synthesis.
- Using the same fixed silence for every kind of sentence.
- Talking over the user or ignoring interruptions.
- Reading long lists of options aloud.
- Repeating the same error message without changing the question.
- Confirming everything explicitly, which makes the conversation slow.
- Not offering a human when the agent keeps failing.
When NOT to use it
When the information is long, visual or a complex comparison, such as a contract or a price table. Also when the data is sensitive and the environment doesn’t allow saying it aloud. In those cases, combine voice with a screen, a text link or keypad entry, as Salesforce recommends for comparing three or more options and for data that needs character-exact accuracy. Get the person’s agreement before switching channels and arrive with the fields already filled in.