- "Trained on your data" is a sales phrase covering four different mechanisms with different costs, different failure modes and different exit routes.
- Most shop assistants are retrieval systems. The model is not changed at all; your catalogue is looked up and pasted into the prompt at question time.
- Retrieval updates the moment your catalogue does. A genuinely fine tuned model is a snapshot that goes stale and has to be rebuilt.
- The question that separates serious vendors from the rest is what happens to the asset when you leave, and the answer differs sharply by mechanism.
- If the thing gets facts wrong about your own products, the cause is almost always retrieval quality, not the model.
A vendor offers to build you an assistant trained on your catalogue. It will know your products, your policies and your delivery zones, and it will answer customers in your voice. The demonstration is good. The phrase doing the work in that sentence is "trained on", and it can mean at least four different things, three of which involve no training whatsoever.
This is not pedantry. The four mechanisms have different monthly costs, behave differently when your prices change on a Tuesday, and leave you in very different positions if you want to move to another supplier in a year.
What are the four things a vendor might mean?
In order of how little they change the underlying model: putting your data in the prompt, looking it up at question time, adjusting the model's weights, and occasionally nothing at all. Only the third is training in any technical sense.
Pasted into the prompt. Your returns policy and a product list are included as text in every request to the model. Cheap, instant to update, and limited by how much text fits. For a shop with thirty products and a one page policy this is often genuinely sufficient, and some vendors charge a great deal for it.
Retrieved at question time. Your catalogue sits in a searchable store. When a customer asks something, the system searches for the relevant few items and puts those in the prompt. This is retrieval augmented generation, and it is what most shop assistants actually are.
Fine tuned. The model's own parameters are adjusted using examples of your data. This is the only one that is training, it is the most expensive, and for catalogue facts it is usually the wrong tool.
Nothing of the sort. A keyword search over your help pages with a model writing a friendly sentence around the result. Sometimes perfectly adequate. Rarely described honestly.
Why is retrieval almost always the right answer for a catalogue?
Because your catalogue changes and model weights do not. Retrieval reads the current state of your data every time somebody asks; a fine tuned model knows what was true on the day it was built.
Amazon's explanation of the approach puts the general case plainly, describing retrieval augmented generation as a way of extending a model to an organisation's internal knowledge base without the need to retrain the model, and listing the problems it addresses: presenting false information when the system does not have an answer, giving out of date or generic responses when something specific and current was expected, and drawing on non authoritative sources.
Read that list as a shop owner and it is a description of every complaint you would get. A customer asking whether something is in stock, what the current price is, or whether you ship to their region is asking about the present tense. Google's material on the same technique makes the related point about factual grounding, that supplying facts as part of the input is what keeps generated answers tied to something true, and that the whole difficulty lies in making sure the most relevant facts are the ones supplied.
That last clause is where shop assistants actually fail, and it is worth dwelling on because it changes what you should complain about.
| Mechanism | Updates when your catalogue does | Typical cost shape | What you keep if you leave |
|---|---|---|---|
| Pasted into the prompt | Immediately | Per message, slightly higher | Your text, which you already had |
| Retrieved at question time | Immediately, if reindexed | Per message plus storage | Your data, if it is exportable |
| Fine tuned | No, needs rebuilding | Large up front, cheaper per message | Usually nothing you can run elsewhere |
| Search with a friendly wrapper | Immediately | Low | Your pages, which you already had |
When the answers are wrong, what is actually broken?
Nine times out of ten, retrieval. The model received the wrong three products and wrote a fluent, confident answer about them, which is precisely what it is built to do with whatever it is given.
This matters because the instinct is to blame the model and ask for a better one, which costs more and changes nothing. If your assistant tells a customer that a discontinued item is available, the model did not hallucinate it; something handed it a record that should not have been in the index. We went through how this plays out on a real shop in the case of a chatbot inventing a product that was never in the catalogue, and the distinction between a retrieval fault and a generation fault is the first thing to establish in any support conversation with a vendor.
There is a genuine generation failure mode too, and it is worth being able to tell them apart. A model asked a question whose answer is not in what it retrieved will often produce a plausible one rather than saying it does not know, which is a property of how these systems work rather than a bug in your setup. That mechanism is the subject of its own piece on why an assistant invents a policy you never wrote, and the practical defence is the same in both cases: insist the system can cite which of your records an answer came from.
What should you ask a vendor to show you?
One thing, and it settles most of the argument: a wrong answer, with the sources it used displayed next to it. Any serious system can show you which records were retrieved for a given question.
If it can, you can debug your own assistant forever. You look at the bad answer, see that it pulled the 2024 shipping page instead of the current one, delete the old page, and move on. If it cannot show you, you are reduced to complaining about vibes, and so is the vendor, which is why these relationships sour.
The questions that actually separate offers
Price per month is the least informative thing on the quote. Five questions do more work, and none of them require you to understand the architecture.
What happens when I change a price at nine in the morning? You want a specific answer with a time in it. "Within fifteen minutes of your next catalogue sync" is a real answer. "It learns from your data" is not. This single question separates retrieval from fine tuning faster than any technical discussion.
Can it tell me which of my records it used? Covered above, and the one to insist on.
Is my data used to improve anything beyond my own instance? Ask for the clause, not the reassurance. The contractual position on what a vendor may do with what you upload varies enormously, and the general question of what is safe to hand over is one we set out in what you may paste into an AI tool and what you may not.
What do I take with me if I cancel? For a retrieval system the honest answer is your source data in a usable format, since the index can be rebuilt anywhere. For a fine tuned model the answer is usually nothing portable, and that loss of portability should be priced into the deal rather than discovered later. The broader problem of being moved underneath by a supplier you do not control is the subject of what happens when the AI tool you depend on is switched off.
Which model is underneath, and who decides when it changes? Most vendors resell access to a general model. That is fine and usually sensible, but it means your assistant's behaviour can change because somebody else shipped an update. You want to know whether you are told in advance.
How much of this can you do yourself?
More than vendors imply, and less than the enthusiastic corner of the internet suggests. The honest middle is that assembling a decent retrieval assistant over your own content is now a weekend rather than a project, and keeping it correct for two years is the part that has not got easier.
The weekend version works like this. Export your product data and your policy pages as plain text. Put them somewhere a tool can search. Connect a general assistant to that store. Most platforms now expose this as a configuration screen rather than as code, and for a catalogue of a few hundred items the result is frequently indistinguishable from a paid product in a demonstration.
Where the paid product earns its money is everything after the demonstration. Reindexing when your catalogue changes without you remembering to press a button. Logging what customers asked so you can see what the assistant could not answer. Handling the case where two of your documents disagree. Keeping working when the underlying model changes. None of that is visible in a comparison of answer quality on launch day, which is why answer quality on launch day is a poor basis for choosing.
The sensible test is therefore not "is the paid one better today". It is "which of the maintenance jobs above am I realistically going to do myself". For most one person businesses the honest answer is none of them, and that is a legitimate reason to pay somebody. It is a very different reason from the one on the sales page, and it leads you to ask about operations rather than about intelligence.
Does the size of your catalogue change the answer?
Substantially, and it is the first thing to establish about your own situation. Under roughly fifty products with stable descriptions, your entire catalogue may fit in a prompt, and the retrieval layer everyone wants to sell you solves a problem you do not have.
At a few hundred products, retrieval starts to matter because the search step is doing real work in narrowing to the relevant handful. At several thousand, with variants, the quality of that search becomes the dominant factor in whether the assistant is any good, and the model underneath matters less than almost anything else in the stack.
Knowing which band you are in tells you what to interrogate. A small seller being sold a sophisticated retrieval pipeline is buying machinery for a scale they do not have. A large catalogue being handed a prompt stuffing approach will see the assistant quietly stop knowing about most of the range, with no error message to mark the point where it stopped.
One more practical consequence of catalogue size concerns what you should measure. With a small range, the useful test is exhaustive: write down the twenty questions customers actually ask, run them all, and read every answer. With a large range, exhaustive testing is impossible and you need sampling plus a log of unanswered questions, which is an operational habit rather than a one off check.
When is fine tuning the right purchase?
When you are buying behaviour rather than facts. Adjusting a model's weights is good at teaching it how to sound and how to handle a format; it is a poor and expensive way to teach it what you currently sell.
The technique itself is well established and has become much cheaper, with methods that adjust a small number of additional parameters rather than the whole model. The published description of the approach and its low rank variants explains why the cost came down, which is also why vendors now offer it to businesses that would not have been candidates three years ago.
Two situations genuinely justify it for a small business. The first is a consistent house voice that prompting keeps failing to hold across hundreds of outputs, where examples work better than instructions. The second is a narrow repetitive task with a strict output shape, such as turning supplier emails into a fixed set of fields, where a small tuned model can be cheaper and faster than a large general one.
Neither of those is your product catalogue. If a vendor proposes fine tuning so the assistant knows your stock, ask how a price change propagates, and listen to whether the answer involves rebuilding something.
The part nobody mentions: your data has to be worth retrieving
Every one of these mechanisms inherits the quality of what you feed it. A retrieval system over a catalogue with inconsistent attributes, three overlapping shipping policies and a returns page last updated two years ago will produce confident contradictions, and no model choice fixes that.
This is the uncomfortable finding in most of these projects. The work is not the AI, it is discovering that your own documentation disagrees with itself. Two pages say different things about international delivery. The product data says one material and the description says another. A human reading your site resolves those silently by picking the one that sounds more current; a retrieval system picks whichever scored higher on the search, which is arbitrary from your point of view.
The useful sequence is therefore unglamorous. Decide which document is authoritative for each category of question, delete or archive the others, make the product attributes consistent, and only then connect anything. A shop that does this ends up with a better website regardless of whether the assistant ever ships, which is a reasonable hedge.
It also changes the buying decision. If the bulk of the value comes from having one authoritative version of your own facts, then the tool that holds those facts natively, already connected to your orders and stock, starts from a stronger position than a general assistant pointed at your public pages. That is the argument for thinking about assistants that live inside the shop rather than beside it, and it is worth weighing before you sign anything that treats your catalogue as an external document to be scraped.
A short translation guide
Vendors are not usually lying. The vocabulary is genuinely muddled, and the person selling to you may have inherited the phrase from their own marketing without a clear idea of which mechanism sits underneath.
So treat "trained on your data" as a prompt for a conversation rather than as a claim to accept or reject. When you hear it, ask the price change question. The answer tells you the mechanism, the mechanism tells you the cost shape and the exit, and those are the two things you are actually deciding between. Everything else on the quote is detail.