TechnicalAI glossary
Mixture of Experts
"Mixture of Experts": the model is split into specialists and only activates the most relevant ones for each request. Huge capacity, reduced cost, the trick behind DeepSeek or Kimi.
In other words
An expert firm where only the 2 relevant specialists stand up for your question.
Alyriaz articles that mention it
- Mistral unveils Large 4, its 1-trillion-parameter giant
Mistral unveils Large 4, a 1-trillion-parameter model trained in Europe, soon to be openly available. A genuine leap, and a gap that remains.
- Six hundred billion parameters at a dollar per million tokens
StepFun has launched Step 5 Preview, a 600-billion-parameter model with a one-million-token context window, priced at 1 dollar for input. We put that rate into context within the current landscape.
- Dense model or mixture of experts: two philosophies, two use cases
A dense model uses all its parameters for every request, while a mixture of experts only activates a fraction of them. We explain why this architectural choice determines the possible use cases, particularly on-device.
- What is a "Mixture of Experts"? The architecture that explains why giant models don't cost that much
Mixture of Experts, or mixture of experts: this architecture lets a model have trillions of parameters while only activating a small fraction per request. We explain the mechanism, why it now dominates, and what it means when you read a spec sheet.
- What does "1.5 trillion parameters" mean? The figure we brandish without ever explaining it
Grok 4.5 and its 1.5 trillion parameters, LongCat and its 1.6 trillion: the parameter count is brandished as a mark of power. We explain what a parameter really is, why it matters, and above all why a larger number does not guarantee a better model.
- DeepSeek V4: The ultra-fast software architect
DeepSeek's new model reshuffles the global AI deck. Multimodal, open-weight, and conceived as a genuine "senior architect" rather than a mere assistant.