Ask for a piece of information the AI knows perfectly, then one it knows nothing about. Compare the two responses. The tone is identical: same fluency, same confidence, no hesitation. A competent human who doesn't know something signals it—through a pause, an "I think", a furrowed brow. An AI, almost never. Here's why, and it's not a matter of poor training.
This is arguably the most disconcerting trait of these tools, and the source of most of the problems they cause. We've explained elsewhere why an AI hallucinates. The question here is different and more troubling: why doesn't it realise it's doing so?
First reason: there's nothing to consult
Let's go back to the basic mechanism. A language model produces the most likely next word, as we detailed in our article on LLMs. It doesn't consult a database where information would be stored with an associated reliability level.
The consequence is fundamental. When you ask for a celebrity's date of birth, the model doesn't look up a record: it generates the most plausible string of characters given everything it has read. If the information was everywhere in its corpus, the plausible string will be the right one. If it was absent, it will still generate a date, because a date is what plausibly follows that question.
From the model's internal standpoint, these two situations are identical. In both cases, it has produced the most likely continuation. There is no internal signal saying "careful, here I'm filling a gap". There's no feeling of ignorance, because there's no consultation mechanism that could fail.
Second reason: it was rewarded for answering
Here's the most important explanation, and the least known. A model's behaviour is largely shaped by the stage where humans rank its responses, as we explained in our article on training.
Now put yourself in the shoes of an evaluator. You're shown two responses. The first says "I'm not certain, it could be 1923 or 1924". The second says "1923", with confidence. Which one will you judge better? Spontaneously, the second: it's clearer, more useful, more satisfying.
Repeat that judgement millions of times, and you get a model that has learned a very effective lesson: confidence is rewarded, hesitation is penalised. This isn't an accident; it's the direct result of optimisation. We asked these systems to be useful and pleasant, and a model that hedges constantly seems less useful and less pleasant. It's a near-perfect example of the alignment problem: the system has perfectly optimised the objective it was given, and that objective wasn't quite the one we wanted.
Researchers use calibration to describe the fit between the confidence a system displays and its actual reliability. A well-calibrated model that claims 70% certainty should be right about 70 times out of 100. A poorly calibrated model shows high confidence regardless of its error rate. Calibration is an active research area, and progress is real but partial: no current model is perfectly calibrated, and the gap widens precisely on the rarest topics—where a warning would be most needed.
Third reason: the language of uncertainty is treacherous
Even assuming a model knew it was ignorant of something, there would still be an expression problem. Human uncertainty is conveyed through a host of signals: tone, pace, posture, a glance. An AI only has text.
And the text of uncertainty is ambiguous. "I think" can mean near-certainty or pure conjecture. Worse, these phrases are statistically associated with certain writing styles in the corpus, so a model can learn to use them as stylistic tics rather than as sincere markers of doubt. An AI that says "it seems to me" isn't necessarily signalling real hesitation.
What the labs are trying
The problem is taken seriously, and several approaches are making headway. Training specifically to recognise and flag the limits of one's knowledge. Access to real-time search, which allows citing a verifiable source instead of drawing on fuzzy memory. Reasoning models, whose internal draft can surface a usable hesitation, as we described in our dedicated article. And the explicit measurement of hallucination rates in technical reports, which at least creates a competitive incentive to improve.
Progress is tangible from one generation to the next. But none of it gets to zero, and the reason is structural: asking a probabilistic system to assess the reliability of its own probabilities is a hard problem, not a forgotten setting.
What this means for you
The practical conclusion may be disappointing, but it's solid: an AI's tone contains no information about its reliability. That's counter-intuitive, because in humans tone is a valuable cue. Here, you have to learn to ignore it entirely.
This also rehabilitates a simple practice: asking explicitly. Posing the question "how sure are you, and what are you basing that on?" doesn't give an infallible answer, but it often brings out nuances that weren't in the initial response. And demanding a clickable source remains the best defence, as we detailed in our verification method.
There's something rather human about this flaw, if you think about it. We built systems rewarded for appearing competent, and they learned that confidence pays better than humility. You can see it as a technical problem. You can also see it as a mirror: we got exactly what we valued.