A benchmark is the AI equivalent of a national exam. Like the baccalauréat, it asks all candidates the same questions and compares the scores. And like the baccalauréat, it has its limits: a good score doesn't guarantee you're intelligent in real life, and some people mainly learn to pass the exam rather than master the subject. Keep this image in mind — the whole article revolves around it.
Every time a new AI model comes out, you're hit with an avalanche of percentages. "91.9% on Terminal-Bench." "82.4% on GPQA Diamond." "First on the Artificial Analysis index." These figures decide headlines, stock prices, and which model companies will adopt. But what do they really measure? And do they deserve the trust we place in them? Let's dive in.
Simple definition
A benchmark is a standardised set of questions or tasks given to different AI models to compare their performance fairly. The idea is the same as in any competition: to compare runners, you need them to run the same distance, on the same track, with the same stopwatch.
Without benchmarks, every lab would simply say "our AI is the best" with no comparable evidence. Benchmarks provide a common, quantified, reproducible ground. That's what allows us to say one model is "better" than another on a specific skill, instead of relying on marketing.
The main benchmark families
There are hundreds of them, but they can be grouped by what they measure. Here are the most cited ones in 2026.
| Benchmark | What it measures | Example question |
|---|---|---|
| MMLU | General knowledge and reasoning across 57 subjects | University-level multiple-choice questions in law, medicine, history... |
| GPQA Diamond | PhD-level scientific reasoning | Physics or chemistry problems a non-expert can't solve even with Google |
| SWE-bench | Ability to fix real software bugs | Fix a real issue from an open-source GitHub project |
| Terminal-Bench | Autonomy in a terminal (command line) | Complete a full task without help, step by step |
| HumanEval | Correct code generation | Write a function that passes a series of automated tests |
You'll notice an important nuance: no benchmark measures "intelligence" in general. Each one tests a specific skill. A model can crush GPQA (science) and be average on SWE-bench (code), as we saw recently when Gemini dominated science while others stayed ahead on code.
How to read a score
A technical detail that comes up often: "PASS@1". This means the model completed the task on the first try, without a second attempt. A score of 70% in PASS@1 means that out of 100 tasks, the model succeeded on 70 right away. It's more demanding than "PASS@5", where it gets five tries and the best result is kept.
Another concept: Elo, borrowed from chess. Instead of fixed questions, models face off in pairs and humans vote for the best answer. The Elo score goes up or down based on wins. This is useful because it measures real user preference, not just correct answers on a multiple-choice test.
Why benchmarks lie (often)
Here's the part marketing doesn't tell you. Benchmarks have deep flaws.
Data contamination. Models are trained on billions of texts scraped from the internet. And the questions from many benchmarks also float around on the internet. As a result, a model may have "seen" the answers during training and recite them without understanding anything. It's like a student who memorised the exam key: their score doesn't reflect their understanding.
Test optimisation. When a benchmark becomes the metric that drives sales, labs are tempted to train their models specifically to shine on it. The score goes up, but real-world ability doesn't always follow. Economists call this Goodhart's law: "when a measure becomes a target, it ceases to be a good measure."
Scores announced by the seller. Most launch figures are provided by the lab itself, with no independent audit. That's why on Alyriaz, when we cite a launch benchmark, we always note that it hasn't been verified by a third party. A vendor's number is an indication, not a truth.
When you see a benchmark score, ask yourself three questions. Who measured it (the vendor or an independent third party)? What exactly does this test measure (and does it match your actual use case)? And could the model have seen the answers during training? A good score on the right benchmark, measured by a third party, remains valuable information. A good score from a press release deserves a raised eyebrow.
So, what's it for?
Despite their flaws, benchmarks remain irreplaceable. They're the only common language that allows models to be compared beyond gut feeling. The best rankings, like those from Artificial Analysis, aggregate several tests and favour independent evaluations, which reduces bias.
The right way to use them is as a compass, not as absolute truth. A benchmark tells you which direction to look. But the only real test of a model is the one you run yourself, on your own tasks. Like the baccalauréat: the score opens doors, but it's in real life that you see what someone is really worth.