Hallucination, an error that training encourages

The Beginner level established the observation: a model produces the most likely continuation, and the likely is not always the true. The engineering question remains: why doesn't it simply say "I don't know"? The answer lies partly in the way it is trained, and the way it is scored.

Rare facts are not learned

A model retains well what recurs thousands of times in its data: the capital of Italy, the formula for the area of a disc. A fact that appears only once or twice (the date of birth of a little-known researcher, the exact title of a report, the number of a minor article of law) leaves no reliable trace. The model knows that the answer has the form of a date or a title; it produces a plausible date or title. This is why hallucinations cluster around precise details, references and little-known people, and much less around general knowledge.

Guessing pays better than abstaining

In September 2025, researchers at OpenAI published an analysis that became a landmark: most evaluations score a correct answer 1 and a wrong answer 0, exactly like "I don't know". In this system, guessing is always profitable, as in a multiple-choice test with no negative marking. A model optimised to pass these evaluations therefore learns to answer with confidence rather than to acknowledge its ignorance. Their conclusion: as long as the tests do not penalise a confident error more than abstention, models will remain incentivised to bluff.

What can be done about it

• Reward abstention, in training and in evaluations: a model can learn to say "I am not sure", at the cost of answering less often.
• Provide the source rather than relying on memory: web search, supplied documents. The model no longer has to remember the rare fact, it reads it.
• Measure separately the rate of correct answers, the rate of errors and the rate of abstention. A single "accuracy" score hides everything.

Key points

• Hallucinations cluster around rare facts: dates, references, precise details.
• Scoring an error and an abstention the same way pushes models to guess.
• Remedies: reward abstention, provide the source, measure errors and abstentions separately.