- Almost every modern voice runs a two stage pipeline: an acoustic model turns text into a mel spectrogram, then a vocoder turns that into sound.
- Tacotron 2 reported a mean opinion score of 4.53 against 4.58 for professionally recorded speech, which is where the "indistinguishable" claims came from.
- Mispronounced names are almost never a model quality problem. They are a text normalisation or lexicon problem, and a better voice will not fix them.
- The original WaveNet generated audio one sample at a time and was too slow for production. The parallel version DeepMind shipped is over 1000 times faster.
- Mean opinion score is a subjective listener rating standardised by the ITU, and P.800.1 exists because people kept comparing scores that measured different things.
- Long inputs drift and repeat, which is why production systems chunk text at sentence boundaries rather than sending a page.
- Judge a voice on your own worst sentences, not on the demo, because the demo was chosen to avoid exactly what will break yours.
Why does one synthetic voice sound like a person reading and another sound like a machine assembling words, when both are described as neural, natural and human like?
The difference is not one thing. It is a pipeline with five places to go wrong, and most of the problems that make a voice unusable in a real product happen in the stages nobody markets. Once you know which stage produces which symptom, choosing a voice stops being a matter of taste and becomes a matter of testing the right sentences.
What are the stages between text and sound?
Five, and the two in the middle are the ones vendors talk about.
Text normalisation turns written forms into spoken ones. The string 1994 becomes nineteen ninety four in a date and one thousand nine hundred and ninety four in a quantity. The abbreviation Dr becomes doctor or drive depending on what follows. This stage is rules and lookup tables, not intelligence, and it is where a startling share of embarrassing output originates.
Grapheme to phoneme converts spelling into sounds. English is hostile here: read rhymes with reed or red depending on tense, and no amount of context helps with a surname the system has never seen. Systems keep a pronunciation lexicon for known words and fall back to a learned model for the rest.
Prosody prediction decides the music: where the stress lands, whether the pitch rises at the end, how long each pause runs. This is what separates reading from reciting, and it is the hardest stage to evaluate because it fails softly.
The acoustic model produces an intermediate representation of the sound. The vocoder turns that representation into an actual waveform you can play.
What is a mel spectrogram, in plain words?
A picture of sound. Time runs left to right, frequency runs bottom to top, and brightness shows how much energy sits at that frequency at that moment. The mel part means the frequency axis is spaced the way human hearing works rather than evenly, so the detail is concentrated where your ear is sensitive and thinned out where it is not.
It matters because it is a compact, well behaved target. Predicting a spectrogram is far easier than predicting a raw waveform, since a second of audio contains tens of thousands of samples but only a few hundred spectrogram frames. Splitting the job in two is what made the modern pipeline tractable.
Where did this two stage design come from?
From a pair of papers that still define the shape of the field.
The first is WaveNet, a generative model for raw audio. It generates waveforms autoregressively, predicting each audio sample conditioned on all the previous ones, and the paper reports human listeners rating it significantly more natural than the best parametric and concatenative systems in both English and Mandarin. That was the moment synthetic speech stopped sounding synthetic.
The second is Tacotron 2, natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. It pairs a sequence to sequence network mapping character embeddings to mel spectrograms with a modified WaveNet acting as the vocoder. The paper notes that using spectrograms as the bridge allows a significant simplification of the WaveNet architecture, and it reports a mean opinion score of 4.53 against 4.58 for professionally recorded speech.
That 4.53 is the number behind a decade of marketing copy. It is also a number obtained on read sentences in a controlled test, which is a caveat worth carrying into every vendor demo you hear.
Why are some voices fast and others slow?
Because autoregressive generation is sequential by construction, and sequential is expensive when you need tens of thousands of samples per second.
DeepMind's own account of shipping this is unusually candid. Its write up on high fidelity speech synthesis with WaveNet states that the original model synthesises one sample at a time with each sample conditioned on previous ones, that this sequential generation was too slow for production environments, and that the parallel version is more than 1000 times faster, which is what made deployment in the Google Assistant possible.
Every vocoder designed since negotiates that same trade. Autoregressive designs sound excellent and generate slowly. Parallel and flow based designs generate quickly and historically paid for it in artefacts. The metallic, buzzy or watery quality people describe in cheap voices is usually a vocoder running fast.
The failure table
This is the part you cannot get from a vendor page. Match the symptom to the stage and you will stop trying to fix pronunciation with a better model.
| What you hear | Which stage caused it | What actually fixes it |
|---|---|---|
| Names, brands or product codes said wrong | Text normalisation and grapheme to phoneme | A custom lexicon or phoneme override, not a different voice |
| Numbers, dates or currency read oddly | Text normalisation | Pre format the text yourself before sending it |
| Emphasis on the wrong word, statements rising like questions | Prosody prediction | Restructure the sentence, or use markup if the vendor supports it |
| Pauses in strange places, or none at all | Phrasing, driven by punctuation | Punctuate for the ear rather than for the page |
| Metallic, buzzy or underwater texture | The vocoder | A different voice or tier, this one is genuinely model quality |
| Drift, repetition or collapse late in a long passage | The acoustic model losing the plot over length | Chunk the input at sentence boundaries and concatenate |
The first row is the one that costs businesses the most time. A shop hears its brand name mangled, concludes the voice is bad, and shops for a better one. The new voice mangles it too, because the problem was never the voice. It was that a made up word has no entry in any lexicon, and the fix is a pronunciation override that takes two minutes.
How should you evaluate a voice?
On your own content, with your own worst sentences, before you care about the demo.
Build a script of about thirty lines drawn from what you will actually publish. Include every product name, every abbreviation you use, prices in your currency, a long sentence, a question, and something with a proper noun a stranger would stumble on. That script is the test, and it takes half an hour to write once.
Then listen for the failure modes in the table rather than for a general impression. A voice that sounds slightly less warm but pronounces your catalogue correctly is the better business decision, and a general impression will never surface that trade.
What does a mean opinion score actually mean?
It is the average of subjective ratings from human listeners, on a five point scale, and it is standardised because the industry kept comparing numbers that were not comparable.
The ITU maintains the terminology. The summary of Recommendation P.800.1 on mean opinion score terminology explains that the revision exists to avoid misunderstanding as to whether specific values relate to listening quality, talking quality or conversational quality, and whether they come from subjective tests, objective models or network planning models. It adds identifiers for bandwidth and for the type of interface.
Read that as a warning. A quoted score means little without knowing what was rated, by whom, on what material, under which of those definitions. It is the same failure that makes a transcription accuracy figure uninterpretable on its own, which we took apart in the piece on what a quoted word error rate hides from a buyer. Aggregate scores hide distributions, in both directions of the audio pipeline.
How much control do you actually get over delivery?
More than most people use, and less than they want.
Speech synthesis markup gives you explicit handles on the stages that misbehave. A phoneme tag overrides pronunciation for a single word, which is the correct fix for a brand name and takes precedence over anything the model would infer. A break tag inserts a pause of a stated length, which solves phrasing problems that punctuation cannot express, such as the beat you want before a price. Emphasis and prosody tags nudge stress, rate and pitch.
Support is uneven and this is worth checking before you commit. Some vendors accept the full markup vocabulary, some accept a subset, and some accept only plain text with a separate lexicon file. A product plan built on fine grained markup control will not survive a vendor that ignores the tags.
What markup cannot do is repair a sentence that is hard to say. If a line reads badly aloud, no amount of tagging rescues it, and rewriting takes less time than fighting the model. Reading your copy out loud before you synthesise it catches most of these, which is an unglamorous technique that outperforms every setting.
What changed after Tacotron and WaveNet?
The pipeline stayed. The pieces got faster, and the conditioning got richer.
On the vocoder side, the field moved away from purely autoregressive designs toward approaches that generate a whole waveform in one pass, trading a little fidelity for enormous speed gains. That is what made real time synthesis on ordinary hardware normal rather than exotic, and it is why the artefact profile changed: modern cheap voices rarely sound robotic in the old parametric sense, they sound subtly smeared or metallic instead.
On the acoustic side, attention based sequence models gave way in many systems to designs that predict duration explicitly rather than learning alignment implicitly. The practical payoff is stability. The failure where a voice repeats a syllable or trails off mid sentence was largely an alignment failure, and predicting durations directly makes long inputs far better behaved, though not so well behaved that you should stop chunking.
The larger shift is that speaker identity, style and emotion became conditioning inputs rather than properties of a trained voice. One model can now serve many voices, which is why vendors offer hundreds of them and why a new voice can appear without a new model. It is also why voice cloning became a product feature instead of a research project.
What does it cost, and what drives the bill?
Characters, almost always, which has consequences for how you write.
Hosted synthesis is typically billed per character or per million characters, sometimes with a premium tier for the better voices. That pricing shape rewards concision and punishes generated verbosity: a description model that produces four hundred words where two hundred would do has doubled your audio bill as well as your reader's time.
It also makes caching valuable in a way that is easy to overlook. Product names, category labels, standard greetings and error messages are spoken repeatedly and change rarely. Rendering them once and storing the audio removes them from the bill entirely and improves latency at the same time. For a catalogue of any size, caching the stable fragments and synthesising only the variable ones is the single largest cost reduction available, and it costs nothing but a little plumbing.
Running the model yourself changes the shape from per character to per hour of hardware, which wins above a volume threshold you can calculate from your own traffic. Below it, hosted is cheaper and considerably less work.
What about voice cloning and custom voices?
Same pipeline, different conditioning. A cloned voice supplies speaker information to the acoustic model so the output carries a particular timbre. The stages before it are unchanged, which has a consequence people find surprising.
Cloning your own voice does not fix pronunciation. If the system says your surname wrong, a model trained on your speech will still say it wrong unless the lexicon is corrected, because the error happened two stages earlier. Cloning changes who the voice sounds like, not what it decides to say.
The other consequence is legal rather than technical. A voice is closer to a likeness than to a recording, and consent for one use does not extend to another. If you are cloning a real person, including yourself for a business that might later be sold, write down the scope. The same discipline applies to synthetic media generally, and the pattern of platforms attaching consequences to undisclosed synthetic personas is one we tracked when Instagram put a reach penalty on unlabelled AI profiles.
Does it have to run in the cloud?
Increasingly no, and the calculation turns on latency and privacy rather than cost.
Small vocoders and compact acoustic models run acceptably on modern phones and on ordinary server hardware without an accelerator. What you give up is the very top of the quality range and the widest voice selection. What you gain is no round trip, no per character billing and no audio leaving the device, which matters when the text being spoken is a customer's order or address.
The general trade is the one we laid out in what actually fits on device, and speech is one of the better cases for it, because the models are small relative to language models and the latency benefit is immediately audible.
Where latency really bites
In anything conversational, and the number that matters is not total generation time.
For a spoken reply, what a listener perceives is time to first audio. A system that starts speaking in 200 milliseconds and streams the rest feels responsive even if the full utterance takes two seconds to render. A system that renders the whole thing in 800 milliseconds and then plays it feels slower, despite being faster on paper. Streaming synthesis exists for exactly this reason.
That perception effect is why voice products are budgeted stage by stage rather than end to end, an exercise we worked through for how a phone agent spends its latency budget. Speech synthesis is usually not the largest term in that budget, and teams optimise it first anyway because it is the most visible.
For non interactive uses, none of this applies. An audiobook or a product video voiceover is rendered once and played many times, so quality wins and generation time is irrelevant. Deciding which category you are in is the first architectural choice, and it changes which voice is correct.
Where a small business actually uses this
Four places, with different requirements in each, and confusing them wastes money.
Product video voiceover is a rendering job. Quality first, custom lexicon for your catalogue, no latency concern at all. This is the easiest win and the one most shops skip, because recording a human is slow and hiring one is slower.
Accessibility audio on a storefront is a rendering job too, though the text is dynamic. Pronunciation of product names is the whole ballgame here, since a shopper hearing your item names mangled will not trust the page. Anyone assembling the storefront and the media together has an advantage, which is part of why we treat content and structure as one job in the ecommerce website builder rather than two.
Phone lines and IVR are conversational and latency bound. Choose for time to first audio and for graceful behaviour on the words your callers will say.
Multilingual content adds the compounding problem: a translation error becomes a confidently spoken error, and the voice makes it more persuasive rather than less. The quality question there is upstream of synthesis, which is the case we made about what actually decides dubbing quality.
The two stage pipeline described here is the organising architecture, not a universal law. End to end designs that go from text to waveform without an explicit spectrogram exist and are used in production. They collapse the stages rather than removing them, so the same failure modes appear, and the same fixes apply to the text side.
A short buying checklist
Can you supply a pronunciation lexicon or phoneme overrides, and how? If the answer is no, and your catalogue has invented names, that vendor is wrong for you regardless of how it sounds.
Does it stream, and what is the time to first audio on your network? Ask for the number under load rather than the best case.
What happens on a long input? Send a thousand words and listen to the last paragraph. If it drifts, you will be chunking, so find out now.
What are the terms on the audio you generate, and on any voice you clone? This is a contract question, and the answer varies far more between vendors than the audio does.
The short version
A synthetic voice is built by normalising text, mapping it to sounds, predicting the melody, drawing a picture of the audio and then turning that picture into a waveform. Quality claims describe the last two stages. Most of the problems that make a voice unusable live in the first three.
So test on your own sentences, fix pronunciation with a lexicon rather than a subscription upgrade, and decide early whether you are rendering or conversing, because that single question decides which voice is right and it is not the question the demo answers.