TechnicalAI glossary
Model distillation
A technique where a large "teacher" model is used to train a smaller "student" model: the student learns to imitate the teacher's responses and recovers a good portion of its capabilities for a fraction of the cost. When practised without authorisation on a competitor's model, it amounts to theft of know-how, as in the case pitting Anthropic against Alibaba.
In other words
Learning to cook by tasting and recreating a great chef's dishes, rather than by putting in ten years at catering school yourself.
Alyriaz articles that mention it
- Claude Sonnet 5.5: nearly Opus, at half the price
Across 44 professions, Claude Sonnet 5.5 nearly matches Opus 5.5 at half the price. What it changes for you, and where the top model keeps its edge.
- When a newer model scores worse than the one it replaces
A new generation of model can post lower scores than the previous one on certain tests. We explain why, what that reveals about design trade-offs, and how to choose accordingly.
- Can a model's intelligence be stolen?
Model extraction involves reconstructing a system's capabilities by querying it at scale. We explain the mechanism, the possible defences, and why none of them really works.
- Washington accuses Chinese labs of siphoning intelligence from American models
US agencies accuse Chinese labs of extracting the capabilities of American models at scale. We explain the technique in question and why it is hard to prevent.
- What is distillation? Fitting a large model into a small one
Distillation transfers the capabilities of a large model to a much smaller one. We explain the principle, why it works better than direct training, and its limitations.
- Stealing an AI without hacking anyone: the Alibaba case shaking up the AI world
Anthropic claims Alibaba used 25,000 fake accounts to query Claude nearly 29 million times and train its own Qwen model on the responses. Here's what distillation is, why it's nearly impossible to prevent, and why not everyone is convinced by the accusation.