- Fluency and accuracy used to travel together, so reading a sample told you about the whole. They have come apart, and every legacy way of judging translation assumed they had not.
- Neural metrics such as COMET correlate far better with human judgement than word overlap scores, and they are reported to be unreliable on exactly the errors that cost a shop money: deviations in entities and numbers.
- That is why the older lexical metrics have not gone away. They are weak overall and good at catching a changed price or a swapped SKU.
- Choose the method by purpose. Automatic scores for regression between builds, structured human error annotation for absolute claims, quality estimation without references for deciding what a person needs to read.
- The routing decision is where automation actually pays, and almost nobody sets it up. Everything else is a report nobody acts on.
- Language coverage is uneven. Quality on widely spoken languages tells you nothing about the smaller market you are about to open.
Machine translation is now good enough that reading a sample no longer tells you whether it works. That is the whole problem, stated in one sentence, and it inverts the way most businesses check this work.
The old assumption was reasonable. A bad translation used to read badly. Clumsy word order, wrong register, sentences that did not parse. If you read ten product descriptions and they flowed, the rest were probably fine. Systems that write fluent, natural, confident sentences that are also wrong break that inference completely, and they break it in a specific direction: the errors that survive are the ones that read best.
If you sell across borders this is not an academic problem. CSA Research surveyed 8,709 consumers across 29 countries in 2020 and found that 76% of online shoppers prefer to buy products with information in their native language, and 40% will never buy from websites in other languages. The same study reports that 75% say they are more likely to buy the same brand again if customer care is in their language. Translating the catalogue is not optional. Knowing whether the translation is right is the part that gets skipped.
What do the automatic scores actually measure?
Two different things, and the difference explains most of the confusion.
The first generation compared surface overlap. A metric like BLEU counts how many word sequences your system's output shares with a reference translation produced by a human. It is cheap, it is reproducible, and it punishes any legitimate rephrasing exactly as hard as it punishes an error, because it cannot tell the difference between a synonym and a mistake.
The second generation compares meaning. COMET, published in 2020 by Rei, Stewart, Farinha and Lavie, is a neural framework that uses cross lingual pretrained language modelling to predict human quality judgements, trained on direct assessments, post editing effort and structured error annotation. It reads the source, the output and a reference, and it scores adequacy rather than overlap. On the WMT 2019 metrics shared task it reached new state of the art correlation with human judgement, which is why it displaced overlap scores in serious evaluation within a few years.
| Approach | What it compares | Strong at | Blind to |
|---|---|---|---|
| Lexical overlap, for example BLEU | Word sequences against a reference | Catching a changed number, name or code, because the surface changed | Legitimate paraphrase, which it penalises as if it were an error |
| Neural, for example COMET | Meaning, using pretrained representations | Overall agreement with what a human reviewer would say | Certain critical errors, reported to include deviations in entities and numbers |
| Quality estimation, no reference | Source against output only | Scoring live content where no human translation exists | Some hallucination types, which the research finds are hard to detect |
| Human error annotation | Output against a defined error typology | Absolute quality claims you can defend | Nothing much, except that it costs money and does not scale |
Why do the best metrics miss the worst errors?
Because they were calibrated on the errors that existed when they were built, and confident fabrication was rare in that data.
The finding to hold on to comes from work combining the two families. Glushkova, Zerva and Martins report that neural metrics such as COMET, despite strong correlation with human judgement, are sometimes unreliable in detecting certain phenomena that can be considered critical errors, such as deviations in entities and numbers, while lexical metrics catch those specific patterns better. Their proposal is to combine the two rather than replace one with the other.
Read that list again from a commerce chair. Entities and numbers. That is a price, a weight, a voltage, a dosage, a size, a model number, a delivery window. Those are not edge cases in a product catalogue, they are the catalogue. They are also the reason the price and delivery terms should be localised before any product prose, since a swapped digit there stops a sale outright. A metric that scores meaning well and misses a swapped digit is optimised for the opposite of what a shop needs.
This is the practical case for keeping a cheap overlap check alongside a neural one, which sounds like a step backwards and is not. Run both, and treat a large disagreement between them as the signal. A segment that scores well on meaning and badly on overlap has usually had something specific changed, and that something is often a number.
Hallucination is the harder cousin of the same problem. The comprehensive study by Guerreiro, Voita and Martins at EACL 2023 annotated over 3,400 sentences of real in domain data and concluded that previously used detection methods were largely inadequate, with sequence log probability performing best and on a par with reference based methods. Their framing is the useful part for a buyer: detection performance depends on the type of hallucination, so a single number claiming to catch them is not describing the problem accurately.
Which method for which purpose?
Three jobs, three answers, and mixing them up is why translation quality programmes produce reports nobody acts on.
| The question you are asking | What to use | What it costs | The mistake to avoid |
|---|---|---|---|
| Did quality change between last month's build and this one? | Automatic metrics on a fixed test set, tracked over time | Almost nothing once set up | Reading the absolute number. Only the movement means anything. |
| Is this good enough to publish, as an absolute claim? | Human error annotation against a typology with severities | Real money, per segment | Substituting an automatic score because the human version is expensive |
| Which of these 40,000 segments needs a human? | Quality estimation, which scores without a reference | Moderate, and it pays for itself | Skipping it and either reviewing everything or nothing |
The third row is where the money is, and it is the one most teams never build. Reviewing everything means you are not automating. Reviewing nothing means you are gambling. Routing means a person reads the 8% of segments that are risky or uncertain, and that is a workload a small business can actually staff.
How do you decide what a wrong sentence costs?
By content type, before you look at any score. The tolerance is not a property of the translation, it is a property of what happens when it is wrong.
| Content | Cost of a fluent error | Sensible policy |
|---|---|---|
| Safety, dosage, compliance and legal text | Severe, and possibly not recoverable | Human translation or full human review. Never route this automatically. |
| Product specifications, sizing and pricing | High. Returns, chargebacks and complaints | Automatic translation plus a targeted check on numbers and entities |
| Marketing and brand copy | Moderate, and mostly reputational | Transcreation or human editing. A correct but flat sentence still fails here. |
| Support articles and FAQs | Moderate. A wrong answer generates the ticket it was meant to prevent | Route by confidence, review the top viewed articles by hand |
| User reviews and community content | Low. Readers expect imperfection | Translate automatically, label it as machine translated |
The counterintuitive row is marketing. It has the lowest factual risk and the highest failure rate, because an accurate translation of a slogan is usually a bad slogan. Fluency metrics will pass it and customers will not.
What about terminology?
This is the most reliable quality gain available to a small catalogue and it requires no evaluation apparatus at all. Build a glossary of the terms that must always translate the same way and the terms that must never be translated: your brand, your product line names, your material names, your size conventions. Feed it to whatever system you use, and check compliance with it as a separate pass from any quality score.
Terminology consistency is a mechanical property. Either the target term appears or it does not, and you can check that with string matching rather than judgement. It is also the failure customers notice fastest, because a product called three different things across a catalogue reads as carelessness in any language.
The same discipline applies to the register question. Formal and informal address is a single decision per language that then has to hold across ten thousand strings, and it is invisible to a segment level score because each individual sentence is defensible. Decide it once, write it in the glossary, and check it as a rule rather than a judgement.
Does the sentence carry enough context?
Often not, and this is the structural limitation nobody solves with a better metric. Translation systems mostly work sentence by sentence, while the information needed to translate a sentence correctly frequently sits in a different one. A pronoun whose gender depends on a noun three sentences earlier. A product name that decides whether "light" means weight or colour. A yes that answers a question the model cannot see.
For a catalogue this bites hardest on short strings. A button label reading "Free" or a filter called "Small" has no context at all, and any system will guess. The fix is not evaluation, it is supplying context: keep source strings whole rather than concatenating fragments, and attach a note to any string whose meaning is ambiguous in isolation. If you are choosing tooling, this is worth more than a two point difference in a benchmark. Understanding why context is the binding constraint is easier once you know how these systems process text at all, which we covered in the explainer on how a language model gets from text to an answer.
Does quality hold across languages?
No, and the gap is larger than most planning assumes. A large evaluation of GPT models across eighteen translation directions found that they achieve very competitive translation quality for high resource languages while having limited capabilities for low resource languages, and that hybrid approaches combining these models with other translation systems improved results further.
The practical consequence for market entry is direct. Your experience translating into Spanish or German tells you nothing reliable about the quality you will get in a smaller market, and the smaller market is usually the one where you have no in house speaker to catch problems. Budget human review inversely to how well supported a language is, which is the opposite of how budgets are usually allocated.
This is also why the hybrid finding matters commercially. Where a general model is weak, a dedicated translation system may be stronger, and the sensible architecture uses both rather than standardising on one. That trade off between a general model and a specialised one recurs across every AI decision a business makes, and we worked through the general version of it in the guide to how to evaluate a language model for your own task.
What Google expects from a multilingual shop
Since most of this content exists to be found, the search side has its own requirements and they are unrelated to quality scores. Google's documentation on managing multi regional and multilingual sites states that it uses the visible content of a page to determine its language rather than any code level signal such as a lang attribute or the URL, that you should use hreflang annotations when different languages live on different URLs, and that you should avoid automatically redirecting users between language versions because that can stop both people and crawlers from seeing all of them.
It also warns against translating only the boilerplate while leaving the body in one language, on the grounds that the same content appearing repeatedly with different navigation creates a bad experience. That is exactly what a half finished catalogue translation looks like, and it is a common state to be stuck in for months.
A programme a small business can actually run
Six steps, in order, and none of them needs a localisation department.
One. Classify your content by what a wrong sentence costs, using the table above. This decides everything downstream and takes an hour.
Two. Write the glossary. Brand terms, product names, sizes, materials, and the formality decision per language.
Three. Fix a small test set, a few hundred segments with human translations you trust, and score every build against it. Watch the movement, ignore the absolute value.
Four. Add a numbers and entities check as a separate pass. Extract every digit, unit, code and proper noun from source and target and compare the sets. This is scripting rather than machine learning, and it catches the class of error the best metric is weakest on.
Five. Route by confidence for anything above the lowest risk tier, and staff the review queue realistically. A queue nobody drains is the same as no queue.
Six. Sample published content monthly, with a human reading twenty real pages. Not to compute a score, but because sampling catches the failures your process was not designed to look for.
Then treat the whole thing as a copy problem rather than a translation problem, because that is what it becomes once accuracy is handled. A correct sentence that does not sell is still a bad product page, which is the argument we made about generated English copy in the piece on how much editing AI copy needs before it ships. If you are building the storefront that has to carry all of this, doing it on infrastructure that treats language versions as first class rather than as a plugin saves an enormous amount of retrofitting, which is one of the reasons our approach to building a commerce site assumes multiple locales from the start.
The measurement that beats every score
If you employ or contract anybody to review translations, the single most informative number in the whole system is how much they change. Post editing effort, measured as the edit distance between what the machine produced and what the reviewer published, is a direct measure of usable quality expressed in the currency you actually pay in.
It has three advantages over every metric above. It needs no reference translation, because the reviewer's output is the reference. It is impossible to game, because it is the record of work that really happened. And it converts immediately into money, since a segment that needs thirty percent rewriting costs a knowable fraction of a from scratch translation.
Track it per language and per content type and the picture that emerges is more actionable than any leaderboard. One language where post editing effort is falling month on month is a language you can push more volume through. One where it is flat despite model upgrades is telling you the problem is your source text or your glossary, not the translation system. And a content type where reviewers rewrite most of what they receive is one that should have been routed to human translation in the first place.
The catch is that you need reviewers producing edits inside a system that records them. If your process is a spreadsheet emailed back and forth, this measurement does not exist and cannot be reconstructed. That is a reason to choose tooling that captures the edit, and it is worth more than any feature on a comparison chart.
How do you re evaluate a vendor without redoing everything?
Keep a frozen test set and never let it leak. A few hundred segments with human translations you trust, drawn from your real catalogue and covering every content type in your risk table, held somewhere it will not be published, sent to a provider, or used as training data.
That set is the only asset in this article that appreciates. When a vendor claims an improvement, run it. When a new system appears, run it. When your own model changes, run it. The comparison is like for like because the inputs never moved, and the whole exercise takes an afternoon rather than a procurement cycle.
Two rules keep it honest. Refresh perhaps a fifth of it each year so it stays representative of what you now sell, retiring segments for discontinued products rather than replacing the whole set, which would break your history. And never send it to a provider as a sample, because a test set a vendor has seen is a test set that no longer measures anything, which is the same contamination problem that makes public benchmarks unreliable.
Everything else in a translation programme is replaceable. The frozen set, the glossary and the risk classification are the parts you should still be using in five years, whatever the systems underneath have become.
What to re examine each year
Three things, and only three. Whether your language coverage assumptions still hold, because model quality on smaller languages moves faster than on well served ones. Whether your risk classification still matches what you sell, since a new product line can move a whole category up a tier. And whether the review queue is actually being drained, which is the failure that hides longest because the dashboard keeps reporting that segments were routed.
The metrics will keep changing. The structure above does not depend on which one is current, and that is deliberate. Decide what an error costs, check the numbers separately, route what is uncertain to a person, and measure movement rather than absolutes. Any metric that arrives next year slots into that shape without a rewrite.