A flagship multimodal test has just been overhauled to be more rigorous: filtering out questions that can be answered without looking at the image, adding plausible options, introducing purely visual situations. On this version, models score between 17 and 27%, against much higher scores on the original version. The models have not regressed. The test had stopped measuring what it claimed to measure.
We have explained what a benchmark is and why you should be wary of them. Here is why they wear out, and how quickly.
Mechanism 1: contamination
This is the best known. A published test ends up on the web, in forums, code repositories, articles that discuss it. The next model is trained on that web.
It does not "cheat": it has simply read the questions and sometimes the answers. Distinguishing afterwards whether it reasons or remembers becomes impossible.
This is the problem that the reconstruction test we discussed in August solved elegantly: by providing only a bibliography, it removed everything the model could have memorised. Scores fell between 3 and 15%.
Mechanism 2: shortcuts
This one is more subtle and more interesting. A poorly constructed test often contains clues that allow you to answer without solving the problem.
In a multiple-choice questionnaire, the right answer is frequently the longest, the best worded, or the only one to use technical vocabulary. A model that optimises plausibility learns these regularities without ever looking at what it was asked to analyse.
This is exactly what the overhaul mentioned above corrects: filtering out questions that can be answered without the image, and adding wrong answers as credible as the right one. The drop in scores measures precisely the share of shortcut that existed.
The parallel with the agent that learns to cheat is direct: a system optimises the criterion it is given, not the intention behind it. If a shortcut exists, it will be found.
When all the leading models score between 92 and 96%, the test no longer discriminates anything. The few points of difference are as much statistical noise as a real difference in capability. This is the point we highlighted regarding a score announced on around ten tasks: a tiny sample produces gaps that are down to chance. A saturated test gives the illusion of a measurement where there is nothing left but noise.
What this implies for reading a score
Four reflexes, and they take thirty seconds.
Look at the age of the test. A benchmark published two years ago has been absorbed by every training corpus. Its score mainly says that the model has seen it.
Look at the gap. If the best models are all neck and neck, the ranking means nothing.
Look at who measured. A test conducted by the manufacturer, on a protocol it chose, with an effort level it set, is not an independent measurement.
Be wary of scores that jump all at once. A sudden improvement on an old test more often signals contamination than a breakthrough.
Why we carry on anyway
We should not conclude that these tests are useless. They serve a real function: they make it possible to compare systems on a common basis, which is infinitely better than impressions.
The problem is not their existence, it is their shelf life. A test is a measuring instrument that degrades through the very use made of it — because its existence changes what it measures. This is a rare situation in science and commonplace here.
Hence the need to constantly produce new ones, harder, less contaminable. The overhaul mentioned above is one example, and there will be others.
What to take away
A benchmark score is a photograph taken with a camera that goes out of adjustment every time the shutter fires.
The only measurement that does not degrade is the one you make yourself, on your own tasks, with your own criteria. This is what we have always recommended and what this news confirms: around twenty reference cases, kept up to date, are worth more than any public leaderboard.