- Conversational AI is not one thing. It is five rungs, and vendors on rung two and rung five use identical language to describe what they sell.
- The rung is identifiable from outside: ask what happens when the user asks something nobody anticipated, and what happens when the answer requires reading live stock or issuing a refund.
- Containment rate is the number everyone quotes and the least portable. It is measured on the vendor's traffic mix, and your traffic mix is what decides your result.
- The model never executes anything. OpenAI's function calling documentation is explicit that your application runs the function and returns the result, which is where accountability actually sits.
- Cost per resolved conversation only beats cost per human ticket above a containment rate you should compute for your own volume before signing anything.
- Four failures survive every rung: state across channel switches, authoritative price and stock reads, admitting ignorance, and anything a regulator would call a decision.
Somebody has told you to add conversational AI to support, or to sales, or to both. Three vendors have since demonstrated something that looks the same in a slide and behaves nothing alike in production. One of them is a decision tree with a language model writing the sentences. One of them can read your documentation. One of them can cancel an order. All three call themselves an AI agent.
What follows is a ladder rather than a list, because the differences between these systems are differences of degree in one specific dimension: how much of the world the system is allowed to read and change. Everything else, the voice, the widget, the tone, is presentation. Voice adds one constraint the other channels do not have, and we treated it separately in the piece on the latency budget an AI phone agent has to fit inside. This page is built to be updated rather than rewritten, and the dated specifics are marked as such.
What is conversational AI, defined usefully?
A system that holds a turn taking exchange in natural language and resolves a request. The definition that matters is not about language at all. It is about resolution: whether the exchange ends with the user's problem in a different state than it started.
By that test, a chatbot that answers questions beautifully and then tells the user to email support has resolved nothing. It has performed conversation. Most disappointment with these systems traces to buying performance and expecting resolution, and the ladder below exists to make the difference purchasable.
The five rungs
Rung one: scripted flows
Buttons, menus, branches. The system cannot be wrong about anything because it never generates anything. It also cannot handle a sentence its author did not anticipate, which in practice means the majority of real messages.
What it can do: route, collect structured information, deflect the ten questions that make up half your volume. What breaks it: any input off the path. What it costs: engineering time to write and maintain the tree, which grows without limit as the business changes.
Rung one is unfashionable and it is still the correct answer for a shop whose top five questions are where is my order, do you ship here, what is your returns window, is this in stock, and how do I change my address. Four of those five are lookups. None needs a language model.
Rung two: intent classification with generated replies
A model reads the message, sorts it into one of a known set of intents, and a scripted or templated response follows. This is where most products sold as conversational AI actually sit, and it is a genuine improvement on rung one because the user can write like a person.
What breaks it first: an intent nobody defined, and compound messages that carry two intents at once. The classic production failure is a message that says the parcel arrived damaged and asks whether the replacement can go to a different address. Rung two picks one of those and ignores the other, and the customer experiences being half heard.
Rung three: retrieval grounded answering
The system searches your own material, your policy pages, your help centre, your product data, and answers from what it finds. This is the first rung that can handle a question nobody anticipated, provided the answer exists somewhere in writing.
What breaks it first: material that is wrong, stale, or contradictory. Grounded answering does not fix a bad help centre, it publishes it more efficiently. The second failure is confident synthesis across two documents that disagree, which produces an answer that appears in neither and is wrong in a way no reviewer catches, because the sentence looks like your writing.
Rung four: tool calling that changes real state
The system can look up an order, check live stock, start a return, apply a credit. This is where the value is and where the risk moves from embarrassing to expensive.
The mechanics are worth understanding precisely, because they determine who is responsible when it goes wrong. OpenAI's function calling documentation states plainly that the model never executes your function. It returns a structured request naming the function and its arguments, and your application executes it and returns the result. The docs also describe a strict mode that forces the arguments to adhere to your schema rather than being best effort.
Read that again as an accountability statement. The model proposes; your code disposes. Every guardrail that matters lives in your execution layer, not in the prompt: refund ceilings, address change rules, whether an order past a certain status can be modified at all. A vendor who cannot tell you where those checks run is selling you rung four with rung three engineering. We went through the plumbing of this in the piece on making a model's output reliable enough to act on.
Rung five: agent loops with memory
The system plans, acts, observes the result, and decides what to do next, across several steps and often across sessions. It can chase a missing tracking number through three systems and come back with an answer nobody scripted.
What breaks it first: compounding error. Each step is probably right, and probably right five times in a row is meaningfully less than right. The second failure is memory, which is a feature until a customer's stated preference from four months ago overrides what they just said. Our comparison of the frameworks that manage these loops covers the mechanical differences; the operational point here is that rung five needs containment engineering that rung three does not.
The ladder as a table
Each row states what the user experiences, what fails first, who is accountable for that failure, and what drives the bill. Assembled from vendor documentation and from the failure patterns above rather than from any single source.
| Rung | What the user experiences | What fails first | Who is accountable | Cost driver |
|---|---|---|---|---|
| 1. Scripted flow | Menus and buttons that always work | Any sentence off the path | Whoever wrote the tree | Maintenance hours as the business changes |
| 2. Intent classification | Types freely, gets a canned answer | Undefined intents, two intents in one message | Whoever curated the intent set | Per conversation, plus labelling effort |
| 3. Grounded answering | Real answers to unanticipated questions | Stale or contradictory source material | Whoever owns the documentation | Tokens, scaling with retrieved context length |
| 4. Tool calling | Things actually happen to the order | A wrong argument passed to a real action | Whoever wrote the execution layer checks | Tokens plus the cost of every reversal |
| 5. Agent loop with memory | Multi step problems resolved end to end | Compounding error across steps | Whoever set the loop and stop conditions | Tokens per step, multiplied by steps taken |
The accountability column is the one to take to a vendor meeting. Every row names a human. If the answer to who is accountable for a wrong refund is the model, the deployment is not ready, and the framing NIST uses in its risk framework is the same: roles, responsibilities and communication lines have to be documented, with named responsibility for disengaging a system whose behaviour has drifted from its intended use.
What do containment, latency and cost per resolution actually mean?
Three numbers decide whether a build is worth doing, and all three are routinely quoted in a form that cannot be transferred to your business.
Containment rate is the share of conversations resolved without a human. A vendor quoting 70 percent measured that on their own customer base, whose traffic mix determines the result almost entirely. A shop whose inbox is 80 percent where is my order will contain far more than a shop whose inbox is 80 percent sizing advice, with identical software. Deciding which ticket types belong on the automated side in the first place changes the containment number more than any software choice does. Ask any vendor for the traffic mix behind their number, and treat an unwillingness to give one as the answer.
Escalation path latency is how long it takes, and how much context survives, when the system gives up. This is the number nobody publishes and the one customers feel. An escalation that arrives at a human with the full transcript and the order already loaded costs the customer nothing. An escalation that dumps them into a queue to explain everything again has made the experience worse than no automation at all, and it converts a contained conversation into an angry one.
Cost per resolved conversation is the only figure that compares to a human ticket, and it must count the unresolved ones. If a system handles 1,000 conversations and resolves 400, the cost of all 1,000 is spread across those 400. Vendors sometimes quote cost per conversation instead, which flatters the number by exactly the failure rate.
A worked example at 10,000 conversations a month
Numbers below are a model, not a quote. The point is the shape of the arithmetic, so substitute your own figures and the conclusion will hold or reverse on its own.
Take 10,000 conversations a month. Call the fully loaded cost of a human handled ticket H, and the cost of an automated attempt A, where A includes tokens, retrieval, platform fees and the amortised build. At containment rate C, the monthly cost is 10,000 times A, plus 10,000 times one minus C times H. The automated attempt is paid on every conversation, including the ones that escalate, which is the detail that breaks most business cases.
Set A at one fiftieth of H, which is generous to the machine, and the arithmetic is easy to hold in your head. At 30 percent containment you pay 0.02H plus 0.7H, so 0.72H per conversation against 1.0H for the all human baseline: a 28 percent saving. At 60 percent containment you are at 0.42H, a 58 percent saving. At 80 percent, 0.22H.
Now make A one fifth of H, which is what a rung five agent with a long context and several tool calls per conversation can easily cost. At 30 percent containment you pay 0.2H plus 0.7H, which is 0.9H, a 10 percent saving that will not survive its own project management. At 60 percent you reach 0.6H. The same containment rate produces a saving worth the project or a rounding error depending entirely on the per attempt cost, and the per attempt cost is set by which rung you bought.
| Containment | Cost per conversation at A = H/50 | Cost per conversation at A = H/5 | Escalated conversations per month |
|---|---|---|---|
| 30 percent | 0.72 H | 0.90 H | 7,000 |
| 50 percent | 0.52 H | 0.70 H | 5,000 |
| 60 percent | 0.42 H | 0.60 H | 4,000 |
| 80 percent | 0.22 H | 0.40 H | 2,000 |
The last column is the one operations teams forget. At 60 percent containment on 10,000 conversations, 4,000 escalations still arrive, and they are the hard ones. Automation does not reduce your staffing to 40 percent of what it was, because the residue is harder per ticket than the average was before. Plan for handle time on escalated tickets to go up, not down.
Which rung should a small business actually buy?
Rung three, in most cases, and rung four only for the two or three actions that account for most of your volume.
The reasoning is about the cost of being wrong. Rung three's worst failure is an unhelpful answer, which a customer forgives. Rung four's worst failure is money moving or an order changing, which a customer does not. Buying rung four for the whole surface means engineering guardrails for every action; buying it for order status lookup and returns initiation means engineering two, and those two probably cover more than half of what people write in about.
Rung five is worth building when a single request genuinely spans several systems and the sequence cannot be predetermined. That is rarer in a small business than in the demos, and each additional step multiplies both the cost and the failure surface. The orchestration patterns are the same ones we set out in the piece on how multi step AI work is actually coordinated.
What does conversational AI still do badly, at every rung?
Four failures survive the whole ladder. They are not implementation defects and no vendor has solved them.
State across a channel switch. A customer who starts in web chat, replies to an email, then sends a message on a social channel is three conversations to almost every system in production. The identity is the problem, not the language model. Until your systems agree on who the person is, no amount of memory helps.
Anything requiring an authoritative price or stock read. A cached number is a wrong number as soon as it is quoted with confidence. This has to be a live call at rung four or a refusal to answer at rung three. The middle option, answering from indexed content, is how customers end up holding you to a price you no longer offer.
Refusing to answer when it does not know. Models are poor at declining, and grounded systems are worse because retrieval always returns something. The mitigation is structural: a confidence threshold enforced in your code, with a scripted handover below it. Nobody gets this right in the prompt alone.
Anything a regulator would treat as a decision. Credit, eligibility, refusal of service. A conversational surface can gather and explain; it should not decide. This is where the NIST framing earns its keep, because the framework treats documented human oversight and named accountability as prerequisites rather than as maturity.
How should this be monitored once it is live?
Like a production system, which is not how most of these are run. IBM's overview of what observability means for language model applications names three pillars: system performance, resource use, and model behaviour, with latency, throughput, error rate and token counts alongside correctness and factual accuracy. The point it makes about scale applies immediately here. Manual review of transcripts works at fifty conversations a day and fails at five hundred.
The practical minimum for a small deployment: log every conversation with its outcome, sample the contained ones weekly rather than only reading the complaints, and track containment as a trend rather than a headline. A containment rate that rises while satisfaction falls means the system is closing conversations rather than resolving them, and that pattern is invisible if you only look at the headline number.
What about adversarial use?
A conversational surface is an input your customers write, which makes it an attack surface. Microsoft's team published lessons from red teaming more than 100 generative AI products in January 2025, and two of them transfer directly to a small deployment. Attacking these systems does not require any gradient computation or model access, which means the barrier for someone probing your support bot is a text box and patience. And securing them is never finished, because the system, its documents and its tools all keep changing.
For a shop at rung four, the concrete version is short. Assume that anything reachable by the tool layer will eventually be requested by someone testing your limits, and put the limit in code. A prompt instruction not to issue refunds above a threshold is a suggestion. A check in the refund function is a rule.
What changed in the last year
Kept short deliberately, and dated, so this section can be replaced without touching the rest of the page. Tool calling moved from a differentiating feature to a default, with schema enforcement available rather than best effort argument matching. Speech moved in the same direction, toward voice models that handle real time full duplex speech. Model costs per token continued to fall, which moves the arithmetic in the worked example toward automation at the same containment rate. Nothing changed about the four persistent failures, and nothing in the last year made containment rates portable between businesses.
The buying checklist
- Establish which rung is being sold by asking what happens on an unanticipated question and on a request that changes an order.
- Ask for containment measured on a traffic mix resembling yours, and treat a refusal as a data point.
- Compute cost per resolved conversation, not per conversation, using your own volume.
- Ask where the guardrails execute. If the answer is the prompt, the answer is no.
- Design the escalation before the automation, including what context travels with it.
- Decide the four persistent failures in advance: identity across channels, live price and stock, the refusal threshold, and which decisions stay human.
If you are building the storefront and the support surface together rather than bolting one onto the other, the tool layer is the part worth owning outright, because that is where your rules live. Our MCP server documentation covers the interface a model uses to reach real business actions, and it is the same interface whichever rung you eventually settle on.