A new benchmark, published in August 2026, asks a difficult question: from the bibliography of a scientific paper alone, can a model recover the idea that paper was defending? The protocol deliberately removes the full text, author information, and any signal post-dating publication. In other words, it removes everything the model could have memorised. Reported scores fall between 3 and 15%, and a multi-agent setup does not exceed around 42%.
We have often stressed that a good score on a test proves little if the model could have seen the answer during training. Here is a serious attempt to get around that problem, and the result is instructive.
The problem this test addresses
The central difficulty in evaluating models is contamination. A model trained on a huge slice of the web has probably read the scientific papers it is asked about, their abstracts, their commentary. When it produces the right answer, you cannot tell whether it is reasoning or remembering.
This is the limitation we highlighted in our article on benchmarks, and it makes many scores hard to interpret.
The protocol here is elegant: by providing only the bibliography, it gives the model exactly what the researcher had before having their idea. It no longer tests retrieval; it tests generation.
What the results suggest
Scores between 3 and 15% for a single model indicate that recovering an idea from its prerequisites alone remains very difficult. The jump to around 42% with a multi-agent setup confirms what we have observed elsewhere: having several instances work in parallel on different angles improves results, as in mathematical proof production.
But 42% on a problem whose answer exists and is verifiable is still a ceiling. The authors see it as a sign that current models struggle with hypothesis generation as soon as they are denied the chance to retrieve the information.
One should be wary of drawing a general conclusion like AI cannot have ideas. This test measures the ability to reconstruct a specific idea already had by someone else, from its premises. A model could produce a different, original and relevant hypothesis, and score zero because it does not match the paper's. In other words, the protocol measures convergence, not creativity. This is a recognised limitation of this kind of evaluation, and it calls for caution in interpretation.
The link with what we know about how it works
This result is consistent with the mechanism we regularly describe. A model produces the most probable continuation, as we explained in our article on LLMs.
Yet a new scientific idea is, by definition, an improbable continuation. It breaks with what the corpus contained. A system optimised for plausibility is structurally ill-suited to producing what was not plausible before it was thought.
That does not rule anything out: the improbable exists within distributions, and a model tuned to explore more, as we explained regarding temperature, produces more unexpected outputs. But the usefully unexpected and the absurdly unexpected look very similar from inside the system. Sorting the two requires a judgement the model does not have.
What to take away
This test adds a useful piece to a debate often conducted with anecdotes. It does not say AIs are useless in science, which their ability to modernise scientific code largely contradicts.
It suggests something more precise: these systems are extraordinarily powerful at exploiting what is already known, and markedly weaker at producing what is not. The distinction between exploiting and discovering, familiar to researchers, is here measured.
Which leaves a fairly precise role for humans in the chain: not the one who computes, nor the one who compiles, but the one who has the intuition to look elsewhere. It is a small thing, and it is exactly what makes a discovery.