- Callers do not grade an AI phone agent on how human the voice sounds. They grade it on the gap before it answers, and human conversation sets that expectation ruthlessly low.
- The round trip has five parts. Only three of them are the vendor's to tune, and the two nobody mentions are endpointing and the network.
- Streaming at every stage rather than completing each one in turn is the single difference between a system callers tolerate and a demo.
- Number and name capture deserves a read back every time, whatever confidence score the transcriber reports, because one wrong digit voids the entire call.
- From 2 August 2026 the EU AI Act requires people to be told they are interacting with an AI system, and California has required a clear bot disclosure in commercial contexts since 2019.
- Containment rate is the metric every vendor reports and the one that flatters worst, because the calls an agent cannot handle are the calls that were worth money.
You missed forty calls last month. You know this because the phone log says so, and you know roughly what a booked call is worth, so you have a number in your head that makes an AI answering service look obvious. Then you sat through the demo, and it was flawless. Of course it was. The demo caller spoke clearly, in a quiet room, and asked a question the system had been built to answer.
The calls you are actually worried about are the ones from a van with the window down, from someone whose accent the transcriber has met less often, asking a question three quarters of the way through your booking flow. Working out whether an agent survives those calls means looking at a single number that vendors rarely put on a slide, and it is not accuracy. It is delay.
Why does the pause decide everything?
Because human conversation runs on a timing system that everybody has internalised and nobody can describe. A study of ten languages, from small indigenous communities through to major world languages, found clear evidence in every one of them for a general avoidance of overlapping talk and a minimisation of silence between conversational turns. The languages differed in their average gap between turns, but only within a range of 250 milliseconds around the cross language mean. Turn taking is close to a human universal, and it is fast.
The practical consequence is that a caller does not decide to be annoyed. They notice a gap, they assume the line dropped or the machine did not understand, and they start talking again. Now the system has to deal with speech arriving while it is speaking, which is the hardest case in the whole design. Latency does not just feel bad. It manufactures the failure mode that breaks the rest of the call.
This is also why voice sits so much higher on a difficulty curve than chat. We laid out that progression in the piece on the rungs of the conversational AI ladder, and the phone is the rung where every weakness below it becomes audible.
Where does the delay actually go?
Five stages, in order, every single turn. The table below is an allocation rather than a measurement: it splits a budget you choose across the stages that consume it, so you can see which stage is eating your headroom and ask the vendor for their measured figure on that one line rather than for a vague total.
Set your own ceiling first. Pick the longest silence you would accept from a receptionist mid sentence, in milliseconds, and treat that as the number every stage has to share. Then read across.
| Stage | What it does | Share of the budget | What actually cuts it |
|---|---|---|---|
| Endpointing | Decides the caller has finished speaking | Large, and usually the biggest single slice | Semantic endpointing that considers whether the sentence sounds finished, not just how long the silence has lasted |
| Transcription | Turns speech into text | Small if streamed, large if not | Streaming recognition that emits partial text as the caller speaks, so this stage is nearly finished when they stop |
| Reasoning | Plans the reply, looks up your calendar or stock | Highly variable, and the one that spikes | Fewer tool calls per turn, cached lookups, a smaller model for routine turns |
| Speech synthesis | Turns the reply into audio | Small if streamed from the first clause | Starting playback on the first few words rather than waiting for the full sentence to render |
| Network and codec | Carries audio between caller and system | Fixed, and outside your control | Nothing you can buy. Budget for it and stop trying. |
The row that surprises people is the first one. Endpointing is not part of the AI at all in the popular sense: it is the decision that the caller has stopped. A system that waits a fixed silence before responding pays that silence on every single turn, and if it is set short enough to feel quick it will interrupt anybody who pauses to think. This is why the same vendor can sound instant on a scripted demo and sluggish on a real call. The demo caller finished their sentences cleanly.
What separates a usable system from a demo?
Streaming, at every stage, rather than completing each stage before starting the next. If transcription waits for silence, then the model waits for the full transcript, then speech synthesis waits for the full reply, you have serialised five delays that could have overlapped. The same five components arranged to stream can start speaking before the model has finished thinking.
The research field has started naming this properly. A 2026 survey of full duplex spoken dialogue systems proposes an architectural hierarchy from L0 to L3, an interaction ontology, and a decision state machine with states named IDLE, LISTEN, SPEAK, WAIT and DUAL. The survey's most useful observation for a buyer is what it calls the realisation gap: many systems support full duplex operation in principle, while their actual behaviour stays limited by the interaction patterns present in their training data. Translated for a purchase decision, a datasheet claiming full duplex tells you what the architecture permits, not what the system does when somebody talks over it.
Which is exactly what the benchmarks now try to measure. The ICASSP 2026 HumDial challenge built a benchmark from a dual channel dataset of real recorded human conversations, aimed at systems handling interruptions, overlapping speech and feedback. Dual channel matters: the caller and the system are recorded separately, which is the only way to see who was talking when. If you ever get a vendor to run a structured evaluation, that is the shape it should take.
Ask for one recording of a call the system handled badly. Not a highlight reel, one failure. A vendor who cannot produce a single bad call either does not record them or has not looked. Both answers tell you what you need to know before you route your main line into it.
The four failures real callers hit
Latency is the constraint that shapes the design. These are the things that go wrong inside it, in the order they cost you money.
| Failure | What the caller experiences | The control |
|---|---|---|
| Barge in | They interrupt to correct something and the agent keeps talking over them | The system must stop speaking within a few hundred milliseconds of detected speech, and must treat what it heard as the new turn rather than resuming its script |
| Noise and accent | The agent mishears, then confidently acts on the mishearing | Test on your own recorded calls, not on the vendor's samples. Route low confidence turns to a human rather than to a clarifying question loop. |
| Number and name capture | One digit of a phone number or one letter of a surname is wrong, so the confirmation never arrives | Read it back, every time, whatever the confidence score says. This is the only place where adding delay is correct. |
| Escalation | They ask for a person and get another loop of the agent trying to help | An explicit path out on the first request, with the transcript passed across so the caller does not repeat themselves |
The second row deserves an extra sentence, because it is where accuracy claims and reality diverge most. Recognition quality is not uniform across the people who call you. Research on accent related disparities in speech recognition is an active area precisely because the gaps are large and persistent, and the practical version for a small business is simple: the average word error rate on a vendor benchmark tells you nothing about the specific customers you serve. Your own recordings do.
The third row is the one that voids whole calls. A booking with a wrong digit is not a partial success, it is a lost customer plus a slot held for nobody. Reading the number back costs two seconds and removes the failure mode entirely. Whatever your latency budget, spend it here.
Must you tell the caller it is a machine?
In more places than most operators realise, and the answer is trending towards yes everywhere. Under the EU AI Act, Article 50 requires providers to ensure that people are informed they are interacting with an AI system, unless that is obvious to a reasonably well informed person, and those transparency obligations apply from 2 August 2026. In California, Business and Professions Code section 17941 has since 2019 made it unlawful to use a bot to mislead somebody about its artificial identity in order to incentivise a purchase, with a safe harbour for a disclosure that is clear, conspicuous and reasonably designed to inform.
Read the California safe harbour carefully, because it is the more useful drafting guide. The disclosure has to be designed to inform, which is a higher bar than merely present. A rushed sentence at the start of a call, spoken faster than the rest, is present. Whether it informs is a different question.
None of this is legal advice and your jurisdiction may add its own duties on top. The operational point is that disclosure is now a design constraint rather than an optional courtesy, and it belongs in the first turn, said at normal speed, in words a caller understands.
What happens to the recordings?
Every one of these systems records, because transcription requires it and quality review depends on it. That makes your phone line a data collection point, and the questions are the ordinary ones: what is stored, where, for how long, who can play it back, and what happens when a caller asks you to delete it. A vendor who answers those in a sentence each has thought about it. A vendor who says the data is secure has not.
Two specifics worth settling before you sign. First, whether recordings or transcripts are used to improve the vendor's models, and whether you can opt out without losing features. Second, what your retention period actually is, as opposed to what their storage defaults to. The same discipline applies here as anywhere else you connect a tool to your business, which is why an AI policy that names data classes rather than tools outlives any individual vendor decision.
What should you measure once it is live?
Every vendor reports containment rate: the share of calls handled without a human. It is the right metric to watch and the wrong metric to celebrate, because it moves for two opposite reasons. It rises when the agent genuinely resolves calls, and it also rises when callers hang up rather than fight it. Those look identical in the summary.
So measure three things beside it. The rate at which callers ask for a human, which tells you what containment cost. The rate at which contained calls produce the outcome you wanted, a booking made or an order placed, rather than merely ending. And the abandonment rate in the first thirty seconds, which is where a caller decides the thing on the line is not worth talking to.
Then look at the calls that escalated, one by one, for the first month. That set is small enough to read and it contains your entire product decision: whether the calls the agent cannot handle are trivial or whether they are your best customers with an unusual question. If it is the second, containment is a vanity number and the agent is a filter on your revenue.
A test you can run in an afternoon
Before you route your main line anywhere, spend two hours generating the evidence a demo cannot. Take ten recordings of real calls you already have, or if you have none, ring the trial number yourself ten times from different situations. In a car with the window open. Standing next to the extractor fan. With somebody talking in the background. Ask for something slightly wrong, then correct yourself mid sentence, which is what people do constantly and scripts never do.
Score each call on four things and nothing else. Did it interrupt you when you paused to think. Did it stop when you interrupted it. Did it get your number right without being asked twice. Did it get you to a human when you asked for one. Four yes answers out of ten calls is a system you can put on an overflow line. Anything that fails the second or the fourth is not ready for a customer, whatever it did on the other eight.
Write down the delay you felt, in whole seconds, rather than trying to measure it. The perception is the product. If two systems both feel instant, the one with more headroom in the reasoning stage will hold that feeling when your calendar lookup is slow on a busy morning, so ask which of them is doing the work before the caller stops talking.
The sentence a vendor will not say
An AI phone agent is very good at the calls that were nearly free to handle and worst at the calls that were worth the most. Somebody ringing to check your opening hours is contained trivially. Somebody ringing with a complicated order, an unusual date, a complaint they have not decided how to phrase yet, that is the call the system escalates, and it is also the call where a human converts.
That is not an argument against buying one. It is an argument for buying it as overflow rather than as a replacement, and for measuring it on the calls it hands back rather than the calls it keeps. The forty missed calls that started this were worth something precisely because nobody answered them. An agent that answers thirty five of them and hands you five with a transcript has done the job. An agent that answers all forty and loses three of your best has not, and containment rate will report that as an improvement.
The same asymmetry shows up wherever people expect software to close a sale, which we went through in detail in the piece on what an AI sales agent will not do for you. Voice makes it louder because the customer is right there, listening to a pause and deciding what it means. If you would rather see how the rest of that customer conversation gets handled once the call is over, our customer support setup covers the follow up side of it.
One last practical note. The technology underneath this is moving quickly, and the parts that improved most recently are exactly the ones in the latency budget above. We covered that shift when real time full duplex speech models arrived. If you evaluated a phone agent more than a year ago and rejected it on the pause, the honest advice is to run the test again on your own recordings, because that specific complaint is the one the field has spent its effort on.