Skip to content

AdvancedLesson 4 of 59 min read

Mixture of Experts, distillation, quantization: the engineering of efficiency

A course from the Academy: Under the hood of LLMs

Go to the quiz ↓

Models are getting bigger, but so are the electricity bills. A whole engineering discipline has grown up around squeezing maximum intelligence out of every euro spent. Three techniques dominate, and you'll come across them in every model announcement.

MoE: only activating the useful specialists

A Mixture of Experts model replaces certain layers with dozens of "expert" sub-networks. For each token, a router activates only the few relevant experts. The result: a model with hundreds of billions of parameters in total, of which only a small fraction computes at any given moment. This is the architecture behind recent giants, including large open Chinese models like DeepSeek or Kimi: the intelligence of a colossus, the compute cost of a mid-sized model.

Distillation: the big one trains the small one

Distillation teaches a small model ("the student") to imitate the responses of a large one ("the teacher"). The student picks up a large share of the know-how for a fraction of the running cost. That's why today's "mini" models outperform the big models of two years ago.

Quantization: compressing the numbers

Parameters are numbers stored with a certain precision. Quantization reduces that precision (from 16 bits to 8, 4, sometimes less): a model 2 to 4 times smaller and faster, often with minimal loss of quality. This is what makes it possible to run real LLMs on a laptop, or even a phone.

Analogy. A hospital doesn't call in all 200 of its doctors for every patient: the front desk (the router) sends you to the two right specialists (MoE). The professor trains interns who do 90% of the work for far less money (distillation). And files are archived in a compact version rather than glossy paper (quantization).

Key takeaways

  • MoE: many parameters in total, few active per token.
  • Distillation: the knowledge of large models compressed into small ones.
  • Quantization: less precision per number, lighter and faster model.

Key concepts

Complete the lesson

A 4-question quiz: 3 correct answers are enough, plus an exercise to practise. It opens in the Academy, on this same lesson.

Take the quiz and complete the lesson Free. With an account, your progress is saved and leads to the course certificate.

The course: Under the hood of LLMs

  1. The Transformer: the architecture that changed everything10 min
  2. From raw text to assistant: fine-tuning and RLHF10 min
  3. Embeddings and RAG: connecting AI to your data10 min
  4. Mixture of Experts, distillation, quantization: the engineering of efficiency9 min
  5. Evaluating an AI: benchmarks, limits, and critical thinking9 min

The whole Academy →The AI glossary