A customer messaged a small kitchenware shop at eleven at night asking whether a pan was induction safe. The assistant on the site answered in one second, with a specification, a temperature range and a confident yes. None of it was in the product data. The pan was not induction safe. The refund, the return postage and the review took three weeks to work through.
The shop owner's first question was the right one, and almost nobody answers it honestly: was that a bug that someone will fix, or a property of the thing she had installed?
- AI hallucination is better understood as an incentive problem than a knowledge gap. Models are graded like exam candidates, and under that grading a confident guess beats an admission of uncertainty.
- LLM hallucination is two different failure modes wearing one name. Factuality hallucination contradicts the world. Faithfulness hallucination contradicts the document or instruction you supplied. Retrieval fixes one of them and does nothing for the other.
- Published rates are real but almost meaningless without the task attached. HaluEval measured about 19.5 percent fabricated responses on the topics it probed, while a summarisation leaderboard puts the best models under 4 percent.
- No mitigation drives the rate to zero. Every honest option buys a reduction at a cost, and a system designed on the assumption of zero is the actual defect.
- A tribunal has already held a company responsible for what its chatbot told a customer, and rejected the argument that the bot was a separate entity.
- The useful decisions are product decisions: where to show sources, where to refuse, where to require a human, and where a wrong answer is cheap enough to accept.
Why does an AI invent an answer instead of saying it does not know?
Because saying it does not know scores zero, and guessing sometimes scores one. The question of why LLMs hallucinate has that short an answer, and it comes from the people who study how these systems are graded rather than from anyone selling a fix.
In a 2026 paper titled Why Language Models Hallucinate, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang put it in terms any shop owner who has sat an exam will recognise. Like students facing hard questions, they write, models guess when uncertain, producing plausible but incorrect statements instead of admitting uncertainty. The comparison is not decoration. It is the mechanism.
Think about how a benchmark works. A model is asked a few thousand questions and scored on the percentage it gets right. An answer of "I do not know" earns nothing. A confident guess earns nothing when wrong and full marks when right. Across thousands of questions, the strategy that maximises the score is to always answer. Every model that survives to release has been selected, repeatedly, by that rule.
The authors are explicit that this is where the behaviour persists rather than where it begins. It begins earlier, in what they describe as errors in binary classification: if false statements cannot be reliably distinguished from true ones in the training data, some false ones will be produced through ordinary statistical pressure. But the reason it survives every round of polish afterwards is the grading.
Their recommendation follows from that and it is unusually blunt for an academic paper. The fix they argue for is not another hallucination benchmark. It is changing the scoring of the mainstream benchmarks that already dominate leaderboards, so that a calibrated admission of uncertainty earns partial credit instead of nothing. Until that happens, every vendor is optimising toward a target that pays for bluffing.
For a shop owner the practical translation is short. The confidence in that induction answer was not a malfunction. It was the behaviour the system was selected for.
The two kinds of wrong, and why the fix differs
Conflating them is the most expensive mistake in this area, because it sends people to buy a solution for a failure they are not having. The survey literature separates them cleanly.
A survey on hallucination in large language models by Lei Huang and colleagues splits the term in two. Factuality hallucination is output that contradicts real world knowledge or cannot be verified against it, and it subdivides into factual contradiction, where an entity or a relation is wrong, and factual fabrication, where the claim is simply unverifiable. Faithfulness hallucination is different: the output is unfaithful to the context or instruction that you yourself supplied, and it subdivides into instruction inconsistency, context inconsistency and logical inconsistency.
That distinction has teeth. If your assistant invents a specification that appears nowhere, that is factuality, and grounding it in your real product data helps. If your assistant is handed the correct specification and still writes something that contradicts it, that is faithfulness, and adding more retrieval to the prompt does nothing at all, because the right document was already there.
| What went wrong | Typical trigger | What actually helps | What people reach for instead |
|---|---|---|---|
| Factual contradiction, wrong entity or relation | The answer is outside the data the model was given | Retrieval grounding against your own catalogue | A stricter system prompt |
| Factual fabrication, unverifiable claim | A question with no answer in any source | An abstention threshold and a refusal path | A larger or newer model |
| Context inconsistency, contradicts your document | Long context, buried or conflicting fields | Shorter context, one authoritative field, a verifier pass | More retrieval, which adds to the noise |
| Instruction inconsistency, ignores the rule you set | Competing instructions, format pressure | Constrained decoding and structured output | Repeating the instruction in capitals |
Read the last column, because it is where the money goes. Three of those four wrong turns cost real budget and none of them touches the failure they were bought for. We wrote about the ceiling on the first of them in our piece on how guardrails give you a rate rather than a guarantee, and the same arithmetic applies here.
The same survey organises the causes into three stages: data, covering misinformation, bias and knowledge boundaries; training, covering pretraining and the alignment stages; and inference, covering decoding strategy, overconfidence and reasoning failure. That ordering is useful because it tells you which stage you can touch. You cannot touch training. You can absolutely touch what you put in the context and what you accept out of it.
How often does it actually happen?
Often enough to plan for, and the honest answer to the frequency question is that a single percentage is close to useless without knowing the task. Two published measurements make the point better than any argument.
The first is HaluEval, a hallucination evaluation benchmark from Junyi Li and colleagues. Its empirical result is that the model it probed was likely to generate hallucinated content on specific topics by fabricating unverifiable information in about 19.5 percent of responses. Roughly one answer in five, on the topics chosen to stress it.
The second measures something else entirely and gets a very different number. Vectara's public hallucination leaderboard asks models to summarise a document using only what the document says, then scores factual consistency with its HHEM evaluation model. Updated on 11 May 2026, its leading entries sit far lower: Antgroup Finix S1 32B at 1.8 percent, an OpenAI GPT-5.4 nano build at 3.1 percent, Google Gemini 2.5 Flash Lite at 3.3 percent, Microsoft Phi-4 at 3.7 percent.
Those two numbers are not in conflict and neither is wrong. One asks open questions where the model must supply knowledge. The other asks it to stay inside a document it was handed. The gap between roughly 19.5 percent and under 4 percent is the value of grounding, measured. It is also the reason that quoting a hallucination rate without the task, the domain and the abstention policy attached tells a buyer nothing.
What a shop should take from the pair is not a number to trust but a shape to expect. Constrained, grounded, summarising work is the low end. Open ended questions about things not in your data are the high end. Your product assistant does both in the same conversation, and its overall rate is whatever mix of the two your customers happen to ask for.
How do you measure your own rate?
By reading fifty of your own answers against your own data, which takes an afternoon and is worth more than every published benchmark combined. No leaderboard knows your catalogue, your customers or the questions they actually ask, and those three things are what determine the rate you will live with.
The method is deliberately unglamorous. Export the last few hundred assistant answers from your customer service tool or chat widget. Take a sample of fifty, weighted toward the question types that carry money: compatibility, stock, delivery, returns, price. For each one, open the product record or the policy page and mark the answer supported, unsupported or refused. Supported means every factual claim in it traces to a field you can point at. Unsupported means at least one claim does not. Refused means the assistant declined, which counts as a success and not a failure.
Two rules keep the exercise honest. Score the answer, not the outcome, because an unsupported claim that happened to be correct is still a system that guessed. And score against the field rather than against your own knowledge, because you know the catalogue and your customer does not, so an answer you can mentally repair is still wrong on the page.
What comes out is not one number but a split, and the split is the actionable part. In practice the unsupported answers cluster hard into a few question types, almost always the ones asking about something your product data does not contain. A shop that has never recorded whether a pan is induction safe will produce unsupported answers about induction every time somebody asks, no matter which model it runs. That is not a hallucination to be mitigated. It is a missing field, and the fix costs one afternoon of catalogue work rather than a platform migration.
Re-run the sample after any change worth the name: a new model version, a prompt rewrite, a retrieval change, a catalogue import. Fifty answers is a small enough sample that the number moves around, so treat a shift of a point or two as noise and a shift of ten as real. The point of the exercise is not precision. It is that you own a measurement nobody can sell you, and it stays valid when every published figure in this article is out of date.
Keep the dated scores in a single sheet. When a vendor next quotes you an accuracy figure, you will have the only number that answers the question you were actually asking, which is what happens on your site, to your customers, about your products.
Which mitigations actually move the rate?
All of them help, none of them finishes the job, and each buys its reduction with something. The ladder below runs roughly from cheapest to most expensive, which is also roughly from least to most effective.
Retrieval grounding. Put your real catalogue in front of the model and require the answer to come from it. This is the single highest return step and the measurement above is the evidence. What it does not touch: faithfulness failures, where the right document was supplied and contradicted anyway. It also introduces a new failure, because retrieving the wrong document grounds the model confidently in the wrong thing.
Constrained decoding and structured output. Force the answer into a shape with fields that must be filled from known values, so the model picks from your specification list rather than composing prose about it. This converts a fluent paragraph into a lookup, which is exactly what an induction safe field should have been. The cost is that it only works where the answer is a value rather than an explanation. Our piece on forcing a model to hold shape when it returns almost valid JSON covers the mechanics.
Self consistency sampling. Ask the same question several times and compare. Agreement across samples correlates with correctness, and disagreement is a useful signal to escalate. The cost is linear: three samples cost three times the tokens and three times the latency, which on a customer facing chat is felt.
A verifier pass. A second call checks the first against the source before it reaches the customer. This catches the contradiction cases the first pass produced. The cost is another call, another failure point, and a verifier that is itself a language model with its own rate.
Abstention thresholds. Decide the confidence below which the system stops answering and hands over. This is the only item on the list that addresses fabrication directly, because it is the only one that permits the model not to answer. It is also the one almost nobody configures, for the same reason the benchmarks do not reward it: refusing looks like failure until you price the alternative.
Put honestly: the ladder takes a rate from high to low and never to zero. A design that assumes zero, such as publishing an answer straight to a customer with no source shown and no path to a human, is not a system with a hallucination problem. It is a system with a design defect that hallucination merely reveals.
Who pays when the bot is wrong?
You do, and this is settled rather than arguable. The case every shop owner should know by name is Moffatt v. Air Canada, 2024 BCCRT 149, decided by the British Columbia Civil Resolution Tribunal in February 2024.
A passenger asked an airline chatbot about bereavement fares and was told he could apply retroactively within ninety days. Another page on the same website said the opposite. He booked, applied, and was refused. McCarthy Tetrault's summary of the decision records the airline's defence, which is the part worth memorising: it argued that the chatbot was a separate legal entity responsible for its own actions.
The tribunal rejected that outright. A chatbot is interactive, it said, but it remains part of the company website, and the format the information arrives in changes nothing about who is answerable for it. The sum involved was small. The principle is not, and it maps onto the kitchenware shop exactly: the assistant's confident yes was the shop's statement, made in the shop's name, on the shop's site. We went further into how those promises bind in our piece on what happens when a tribunal treats your chatbot's promise as binding.
What should a small shop actually do?
Four decisions, and they are product decisions rather than technical ones. None requires an engineer and all of them are cheaper than one refund cycle.
Decide where sources are shown. Any answer that states a specification, a price, a delivery window or a policy should carry the field it came from, visible to the customer. This is not decoration. It converts an assertion into a citation the customer can check, and it makes the failure visible before the sale rather than after the return.
Decide where the system refuses. Write the list of questions your assistant must not answer, and make refusal the configured behaviour rather than an accident. Compatibility, medical or safety claims, legal terms, anything about a third party product. A refusal that offers a human is a good customer experience. A confident wrong answer is not.
Decide where a human confirms. Anything that creates an obligation should pass a person first: a refund promise, a discount, a delivery commitment, an exception to a policy. This is the cheapest insurance available against the Moffatt problem, because the thing that binds you is the promise, not the technology that made it.
Decide where you can accept being wrong. This is the decision people skip, and it is what makes the other three affordable. A draft product description you will edit, a suggested tag, an internal summary of a support thread: the cost of an error is a few seconds of your attention. Ship those with no ceremony at all. Our walkthrough of which support tickets to automate first is the same triage applied to a queue.
If you are choosing tooling rather than retrofitting it, the ability to see and edit what sits behind an answer is worth more than any published accuracy figure. That is a large part of why merchants building on an AI store builder whose generated code and data stay in their hands can inspect the field an answer came from instead of trusting a vendor's number.
Will this be fixed?
Not soon, and the reason is structural rather than technical. The people who identified the incentive problem also identified who has to solve it, and it is not the shop owner or even, on its own, the lab.
The recommendation in the Kalai and Nachum work is a socio technical one: change the scoring of the benchmarks that already dominate leaderboards so that calibrated uncertainty earns credit. That requires the organisations who publish those leaderboards to agree, and it requires vendors to accept a lower headline number in exchange for a more honest one. Both of those are hard in a market where the headline number sells.
Meanwhile the rate keeps falling on constrained tasks, which is the good news, and the leaderboard above is the evidence. Summarising a supplied document is close to solved. Answering an open question about a pan you did not describe is not, and will not be, because the model has nothing to be faithful to.
Which returns to the shop owner's original question. It was not a bug and nobody is going to fix it for her. It is a property, the rate is knowable, the mitigations are real, and the design decision she actually controls is which answers reach a customer without a source, a refusal or a human in between.