When Microsoft unveiled its MAI-Thinking 1 reasoning model in 2026, it went out of its way to make one claim: the model was trained on licensed data and was not distilled from GPT or Claude. That single detail says a lot about how modern AI actually gets built, because the reason Microsoft had to deny using the technique is that so many models quietly rely on it. Model distillation is the process of teaching a small, cheap model to imitate a large, expensive one, and it has become one of the most important and most contested methods in the field. Understanding it explains why capable models keep getting smaller, why a Chinese lab could shake the industry on a modest budget, and why the biggest AI companies now guard their model outputs like trade secrets.
- Model distillation trains a small student model to copy the behavior of a large teacher model, transferring most of its skill at a fraction of the size and cost.
- The idea was formalized by Geoffrey Hinton and colleagues in 2015, using the teacher's full probability distribution rather than just its final answers.
- Real distilled models like DistilBERT and DeepSeek's R1 distillations show the technique keeps most of a teacher's ability while slashing compute.
- Distillation now sits at the center of legal and competitive fights, since copying another company's model outputs can violate its terms of service.
The appeal is easy to state. Training a frontier model from scratch costs a fortune and demands enormous compute, yet most of that capability can be poured into a much smaller vessel afterward. Distillation is how labs get a model that runs fast and cheap without paying the full price of building intelligence from nothing. It is also, increasingly, how one company can benefit from another's investment without asking permission.
What model distillation actually is
The technique was formalized in a 2015 paper titled Distilling the Knowledge in a Neural Network, written by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. As the Hugging Face writeup on distillation explains, their insight was that a trained model knows more than its final answer reveals. When a large model classifies an image or predicts a token, it does not just pick a winner. It assigns a probability to every option, and the pattern of those probabilities carries information the single correct label throws away.
Hinton called this extra signal dark knowledge. Suppose a teacher model looks at a picture of a dog. It might assign 90 percent to dog, but also a small slice to wolf and an even tinier slice to cat. That distribution encodes something real: dogs resemble wolves more than they resemble cats. A hard label of simply dog erases that relationship, while the full probability spread preserves it. The original 2015 paper argued that a student trained to match this richer distribution learns faster and generalizes better than one trained on correct answers alone.
The setup is always a teacher and a student. The teacher is the large, capable, expensive model you already have. The student is a smaller network you want to make nearly as good. Instead of training the student only on the ground-truth labels in a dataset, you train it to reproduce the teacher's outputs, so the student inherits the teacher's judgment rather than rediscovering it. Done well, the student captures most of what made the teacher smart while being a fraction of the size.
Soft targets and the temperature trick
The mechanism that makes dark knowledge usable is a knob called temperature, applied to the softmax function that turns a model's raw scores into probabilities. The Hugging Face explainer describes temperature as a way to control how confident or uncertain a model's predictions look, adjusting the sharpness of the distribution. At a temperature of 1, the softmax tends to be sharp, pushing almost all the probability onto the single best option and producing what the field calls hard targets.
Raising the temperature above 1 softens the distribution. The probabilities spread out, so the smaller slices assigned to plausible-but-wrong options grow large enough to matter during training. These are the soft targets, and they are where the teacher's nuanced knowledge lives. By heating up the teacher's predictions this way, distillation exposes the student to the teacher's full sense of how similar different answers are, not just which one it would ultimately choose.
The whole trick rests on a simple idea. A hard label tells the student the right answer. A soft target tells the student how the teacher thinks about all the answers, including why it rejected the wrong ones. That second lesson is far richer, and it is what lets a small model punch above its size.
The student then trains against a blend of two goals. It tries to match the teacher's soft targets, learning the teacher's reasoning texture, while also staying anchored to the true labels so it does not simply inherit the teacher's mistakes. Balancing those two pulls is the practical art of distillation. Lean too hard on the teacher and the student copies its flaws; lean too hard on the hard labels and you lose the dark knowledge that made the whole exercise worthwhile.
The flavors of model distillation
Distillation is not a single recipe but a family of them, and the Hugging Face overview groups the variants along two axes. The first axis is what knowledge you transfer. Response-based distillation, the classic form from the 2015 paper, matches the teacher's final output probabilities. Feature-based distillation goes deeper, training the student to reproduce the teacher's intermediate layer representations rather than just its answers. Relation-based distillation is subtler still, teaching the student to preserve the relationships the teacher sees between different data samples or between its own layers.
The second axis is when the teaching happens. In offline distillation, the most common approach, the teacher is fully trained first and then held fixed while it instructs the student. In online distillation the teacher and student improve together during the same training run. Self-distillation is the strangest of the group, where a model effectively teaches itself, using its own deeper layers or earlier checkpoints as the teacher for a leaner version. Each variant trades complexity for a different slice of the teacher's knowledge, and real systems often combine several.
What unites them is the goal of compression without collapse. The point is never just to shrink a model, which any crude pruning can do, but to shrink it while keeping the capability that justified building the large model in the first place. That is the line distillation walks that simpler compression methods cannot.
Model distillation in the wild
The clearest proof that distillation works sits in models people use every day. The canonical example is DistilBERT, released in 2019 as a compressed version of Google's BERT language model. According to the Hugging Face figures, DistilBERT retained about 97 percent of BERT's language understanding while running in roughly 60 percent of the time and shedding around 40 percent of the model size. Losing three percent of capability to gain that much speed and shrinkage is the kind of trade that reshaped how models get deployed.
The technique scaled up with the models. Microsoft distilled its large Llama 3.1 405B model down to 70B and 8B students that retained comparable accuracy to the much larger teacher, and even Amazon Alexa leaned on the idea, using a teacher that processed a million hours of speech to train a smaller acoustic model fit to run on device. The same pattern repeats far beyond language, showing up in vision networks and on-device speech systems alike: a heavyweight teacher trains a nimble student that keeps most of the intelligence at a small share of the cost.
The example that rattled the industry was DeepSeek. As technical writeups on DeepSeek-V3 describe, distillation formed the final phase of the model's training pipeline, transferring the sophisticated reasoning of the DeepSeek-R1 model into the more efficient V3. DeepSeek also released a line of openly distilled models, fine-tuning open systems like Qwen and Llama on roughly 800,000 high-quality examples generated by R1. The R1 reasoning traces, with their long chains of step-by-step thinking, taught the smaller students to break hard problems into manageable steps, and models like DeepSeek-R1-Distill-Qwen-7B went on to outperform much larger systems on reasoning benchmarks. The lesson landed hard across the field: a small model fed the worked reasoning of a strong teacher could reach a level that had seemed to demand far more parameters, which is precisely why the release forced every major lab to rethink its cost assumptions overnight.
Why labs cannot resist it
The economics explain the enthusiasm. A frontier model is expensive along every dimension that matters: the compute to train it, the memory to hold it, plus the latency and energy of every single query. Distillation attacks the second and third of those costs directly. A distilled student needs less hardware to serve, answers faster, and can sometimes run on a laptop or phone where the teacher would never fit. For any company paying per token or per GPU-hour, a model that keeps most of the quality at a fraction of the footprint is close to free money.
This is exactly the pressure driving the industry's current cost-cutting turn. When a merely good model can be distilled to run for a tenth of the price of a frontier one, the case for routing every routine request to the most expensive system collapses. Distillation is one of the main levers that makes cheaper in-house and open-weight models viable substitutes for premium API calls on high-volume, everyday work, which is why the technique keeps surfacing in stories about companies trimming their AI bills.
There is a subtler benefit too. Distillation can transfer not just raw accuracy but specific behaviors, like a reasoning style or a safety posture, from a carefully tuned teacher into many downstream students. That lets a lab do the hard alignment and reasoning work once, on one large model, then propagate the result across a whole family of smaller ones without repeating the effort each time. For teams building a lineup of models rather than a single flagship, that reuse is a large part of the appeal.
Where distillation runs out of road
For all its power, distillation cannot work miracles, and its limits are as instructive as its wins. The most basic ceiling is that a student rarely surpasses its teacher on the knowledge being transferred. Distillation copies what the teacher already knows, so a distilled model inherits the teacher's blind spots along with its strengths. If the teacher is wrong about something in a confident way, the student tends to learn that confident error too, since the soft targets faithfully encode the teacher's mistaken certainty. Garbage in the teacher becomes garbage in the student.
There is also a quieter loss that shows up at the edges. A student trained to match a teacher's typical behavior often gives up a little robustness on the rare, hard cases the teacher handled through sheer capacity. On the bulk of everyday inputs the gap is invisible, which is why headline benchmark numbers stay high, but on unusual inputs or adversarial ones the smaller model can crack where the teacher would have held. DistilBERT keeping 97 percent of its teacher's ability is remarkable, yet that missing three percent is not spread evenly. It concentrates in exactly the difficult territory where a model's judgment matters most.
Distillation can also drift when the student and teacher are too far apart in size or shape, since a very small student simply may not have the capacity to hold everything a giant teacher knows, no matter how good the training signal. Push the compression too hard and the student stops tracking the teacher and starts flattening into a cruder approximation. Knowing when to stop compressing, and accepting the capability you forfeit at each step, is the unglamorous discipline that separates a useful distilled model from a hollow one.
The fight over who owns a distilled mind
The same power that makes distillation useful makes it legally fraught, because the technique does not care whether you own the teacher. If a model will imitate any teacher whose outputs it can observe, then a competitor's model, accessed through a public API, can in principle serve as an unwilling teacher. That is the accusation at the heart of one of the industry's tensest disputes. As CNBC reported in early 2025, OpenAI suggested DeepSeek may have distilled knowledge out of OpenAI's own models, using their outputs to train a rival at a fraction of the cost, a claim DeepSeek has not accepted.
Whatever the truth of that specific case, it exposed a structural problem. Most frontier providers now write terms of service that forbid using their model outputs to train competing systems, precisely because distillation makes such copying technically easy. The outputs a company sells access to are also the raw material a rival would need to clone its behavior. Enforcing those terms is hard, since a distilled model rarely announces its lineage, and detecting whether a given system learned from a particular teacher remains an open research question. A whole subfield has sprung up around protecting models against unauthorized distillation.
This is the context that makes Microsoft's denial so telling. When the company insisted, as reported by The Decoder, that MAI-Thinking 1 was trained on licensed data and not distilled from GPT or Claude, it was pre-empting exactly the suspicion that hangs over every efficient new model. In an environment where distillation is cheap, effective, and hard to prove, saying we built this ourselves has become a competitive and legal statement, not just a technical footnote. The denial is the tell that the technique is everywhere.
A quiet engine behind the whole field
Model distillation rarely makes the headline, yet it shapes almost everything that does. The steady march of capable models onto phones and laptops, the sudden arrival of cheap reasoning models that rival costly ones, the strategic scramble by big companies to reduce their dependence on frontier APIs: all of it runs partly on the ability to pour a large model's knowledge into a small one. A decade after Hinton's paper, the idea has gone from a clever training trick to a load-bearing pillar of the AI economy.
The tension baked into distillation is not going away. It is simultaneously the most democratizing technique in modern AI, letting small teams build strong models without frontier budgets, and one of the most destabilizing, letting anyone with API access potentially harvest the fruits of a competitor's investment. Those two faces are inseparable, because they come from the same mechanism. As models keep getting cheaper to copy and harder to protect, the question of who is allowed to learn from whom will only grow sharper. For readers tracking how that plays out across the industry, our ongoing coverage follows the releases and the fights around them. Distillation is the reason both keep coming.