Voice agents fail at interruption in exactly two ways, and everyone has met both.
The first talks over you. You realise it misunderstood, you start correcting it, and it keeps going to the end of its paragraph as if you were not there.
The second flinches. Someone added local voice detection to fix the first problem, and now a cough, a breath, or a filler syllable cuts the agent off mid-sentence and leaves it waiting for a thought you never started.
These look like opposite bugs and they are the same bug. Both come from an architecture where the component that hears you and the component that is speaking do not share state.
Where the interrupt signal has nowhere to live
The traditional build chains three models. Speech-to-text turns your audio into text, locally or nearby. A language model turns that text into a response, remotely. Text-to-speech turns the response into audio, and playback starts.
Now trace an interruption through that chain.
Your microphone picks up that you started talking. That knowledge exists locally. Meanwhile the language model is somewhere else, streaming tokens, and the speech synthesiser is already voicing tokens it received a moment ago. There is a queue of audio buffered ahead of what you can hear.
So the client can stop playback. What it cannot do is tell the remote model that its sentence did not land, because there is no channel for that and no shared notion of how far the delivery got. The model finishes generating, records in its history that it said the whole thing, and moves on.
The result is a divergence in conversation history, and it is worse than the audible glitch. The user heard seven words. The model believes it delivered forty. Every subsequent turn is built on a transcript that does not match what happened, and the agent starts referring to things it never actually said out loud.
That is why both failure modes appear. Barrel-ahead is the system honouring the model’s view. Flinching is the client aggressively acting on local detection it cannot reconcile with the remote state, so it cuts audio on a cough and desynchronises anyway. Neither side can be right, because the authority over “was this delivered” is split across a boundary.
The fix is to stop splitting it
The architectural answer is a model that owns both directions of the audio stream. You send raw audio in; you receive raw audio out.
What changes is where voice activity detection lives. It runs on the same side as generation, which means the component deciding “the user has started speaking” is the component producing the output. It can halt itself the moment it hears you, and it knows exactly where it stopped.
Then it emits an explicit event down the socket saying it was interrupted. The client trusts that event, drops whatever it had buffered for playback, and both sides now agree on what was delivered.
The generalisation is worth pulling out, because it has nothing to do with audio. Any time you split the detection of a stop condition from the production of the thing being stopped, you need an explicit protocol for reconciling what got delivered — or the two sides will disagree and the disagreement will accumulate. Streaming responses, long-running jobs with cancellation, progressive rendering: same shape, same failure, and the buffered-ahead portion is always where the accounting goes wrong.
How you would actually test this
Barge-in is a race condition, which means the usual approach of trying it a few times by hand tells you nothing. It works when you test it and fails for a user whose timing differs by 200 milliseconds.
Testing it properly means driving the input rather than speaking into it. Inject a fixed audio clip at a controlled offset from the start of the agent’s response, and vary the offset across runs. The offsets that matter are not random:
- Before the first token arrives. Does an interrupt during the thinking gap get handled, or dropped because nothing was playing?
- Mid-word. The most common real case and the one where buffered audio is largest.
- At a sentence boundary. Where a natural pause may look like a completed turn.
- During a tool call. The agent is not speaking and is not idle. This is where most implementations have no defined behaviour at all.
- Immediately after playback ends. Off-by-one on the turn boundary, which shows up as a swallowed first word.
The assertion is the part people get wrong. The obvious check — did the audio stop — passes on systems that are quietly broken, because stopping playback is the easy half.
Assert instead that both transcripts agree. After the interruption, compare what the client actually played against what the model recorded as delivered. If those diverge by more than the buffer, you have the real bug, and it is invisible in a single turn — it only surfaces three exchanges later when the agent references something the user never heard. That property is what belongs in the suite. It is the clearest case of a check worth automating: expensive to verify by hand, needed on every change, and the kind of thing humans check badly because the symptom appears several turns away from the cause.
What breaks after the demo
The prototype works on a laptop for a few minutes. Four things then arrive in order.
The session has a ceiling. Audio sessions are capped, and the underlying connection is dropped by the server before that with a goodbye message. Reconnecting naively means replaying the transcript, which is slow and expensive. The supported route is a resumption handle: store the token the server sends, pass it on reconnect, and the conversation continues across the break. Build this before you need it, because the first time you hit it is in front of someone.
Network jitter starves the speaker. The prototype pulls chunks and pushes them straight to playback, so a 100-millisecond stutter empties the queue and the audio glitches. The fix is a jitter buffer: deliberately delay playback by 50 to 100 milliseconds to hold a small reserve. You are trading a little latency for continuity, and it is the right trade — listeners forgive a slightly slower start and do not forgive dropouts.
Tool latency is bounded much tighter than in text. In a chat, a three-second database query is a spinner. In a conversation, three seconds of silence reads as a dropped call. Anything past roughly 400 milliseconds needs a filler utterance — have the model say it is checking, run the tool in the background, then stream the answer. Which means tool latency is now a product constraint, and a slow internal API that was fine everywhere else is not fine here.
A phone number changes the whole I/O layer. Local audio devices go away, replaced by a media stream from a telephony provider or a real-time transport service. The tool logic and the model configuration survive unchanged; everything touching microphones and speakers is rewritten. Worth knowing up front so the audio layer is kept behind a boundary rather than threaded through the application.
The part to carry away
The interruption problem is not a voice problem and the two visible failure modes are not two problems. Both are what happens when the authority over “how much of this actually got delivered” is split across a network boundary with no protocol to reconcile it.
Fix that by putting detection and generation on the same side, get an explicit interrupted event, and have the client trust it. Then test the thing that actually breaks — not whether the audio stopped, but whether the two sides still agree afterwards.