Latency and Interruptions in Voice Agents

A caller starts talking while the agent is still speaking. The audio stops, but a lookup from the previous turn finishes a second later. Should the agent use that result? What if the lookup was a booking request?

These are ordinary voice-agent problems. A faster model doesn't answer them; the application needs to know which work belongs to which turn.

Account for the whole delay

The pause a caller hears includes speech detection, transcription, retrieval, model output, synthesis, and audio transport. A slow end-of-turn detector can erase the benefit of a faster model.

Record timestamps for each stage under the same turn ID. Look at p95 as well as the median: occasional long silences matter in a phone call. Measure how quickly playback stops after an interruption too.

There isn't one silence threshold for every utterance. Someone reading an account number may pause between digits. Cutting them off early costs another turn, even if the first response looked fast in a trace.

Cancellation needs to reach the work

When the caller interrupts:

1. Stop playback and discard queued audio from the old turn.
2. Cancel model, synthesis, and read-only tool requests where possible.
3. Mark any remaining results with their original turn ID.
4. Track what was actually played, so the conversation doesn't assume the caller heard the rest.

A late result may still be useful, but it shouldn't silently become the answer to a different question.

Actions need special handling. If a booking request was already sent, cancelling the local task doesn't necessarily cancel the booking. Keep its request ID and check the outcome before trying again. Otherwise, an interruption can produce a duplicate reservation.

Say only what is known

Streaming text straight into speech can commit the agent too early. “Your appointment is booked” must wait for the booking result. A brief “I'm checking” can fill a pause without claiming success.

Send speakable phrases to synthesis, with dates, names, and numbers handled consistently. Keep tool responses structured so the agent can distinguish success, failure, and an unknown outcome.

The failure response should match the problem. Missing audio calls for a repeat. An uncertain name calls for confirmation. A failed service needs another route or a human handoff. Repeating the same apology after every error doesn't help the caller proceed.

Test the awkward calls

A useful test set includes interruptions, long pauses, background noise, corrected names, slow tools, and disconnects. Replay the same scenarios after orchestration changes.

Check whether the task completed, whether actions ran more than once, and whether speech continued after cancellation. Also check the handoff: the next person should receive the caller's request and confirmed results, not a summary of what the model hoped had happened.