An experienced teacher doesn't just give the right answer. They show their hesitation, explain why a given option was almost right, and point out the traps. A student learning from a teacher progresses faster than one learning alone from a mark scheme, because they receive far more information than just the solution. Distillation applies exactly this principle between two models.
We mention this term regularly without ever having explained it. Yet it is one of the techniques that makes possible what we often describe: useful models that fit on an ordinary machine.
The principle
You have a large, highly capable model, called the teacher. You want a much smaller model, the student, that performs almost as well.
The naive approach would be to train the small model on the same data as the large one. That works poorly: the small model lacks the capacity to absorb as much, and it learns a degraded version.
Distillation works differently. You put the teacher to work, and you train the student to reproduce not its answers, but its full probability distribution.
Why it is far richer
Here is the key point. When a model produces a word, it doesn't simply pick the best one: it computes a probability for every possible word, as we explained in our article on LLMs.
Take the phrase "the cat drinks". The teacher might assign 70% to "milk", 20% to "water", 5% to "tea", and almost nothing to "concrete". This distribution contains a huge amount of information: it says milk and water are both plausible, tea is marginal but conceivable, and concrete is absurd.
Classical training would only learn "milk". Distillation conveys all the nuance. The student learns not just the right answer, but the structure of the problem as the teacher perceives it.
This is the technique behind the compact models we often discuss. Meta's agentic model that fits on a single graphics card is distilled from a larger model. Without this method, you would have to choose between an inaccessible giant model and a mediocre small one. Distillation offers a third path: a small model that has been trained by a large one.
The difference from compression
It shouldn't be confused with quantization, which we have explained separately.
Quantization keeps the same model and reduces the precision of each parameter: the same boxes, written more coarsely.
Distillation creates a different model, with far fewer parameters, trained to imitate the first. It isn't compression; it's a transfer.
The two can also be combined: you first distill into a smaller model, then compress that one.
The limitations
The teacher's ceiling. A student doesn't surpass its master. Distillation also propagates flaws: if the teacher has a bias or a gap, the student inherits it.
Loss on rare cases. A small model reproduces frequent behaviours well and degrades on unusual situations. That is precisely where the gaps show: on long reasoning, novel problems, complex chains.
The legal question. Distilling from a model you didn't build, by querying its programming interface en masse, is explicitly prohibited by the terms of use of most providers. It is a recurring source of litigation in the sector, and far from settled.
What to remember
Distillation explains something that often surprises: why small models improve so quickly. They don't just progress thanks to better data or better architectures. They progress because large models progress, and they learn from them.
This has a consequence that is poorly appreciated. Every advance at the top eventually trickles down, with a lag of a few months, to what can be run at home. The race for giant models, which can seem excessive, indirectly fuels the democratisation that is otherwise called for.