In 1980, philosopher John Searle proposed this: imagine a person locked in a room, who doesn't speak a word of Chinese. Questions in Chinese are slipped under the door. They have a gigantic manual telling them which symbols to produce in response to which symbols received. They apply the rules and send back perfect answers. From the outside, they seem to speak Chinese. From the inside, they've understood nothing. Searle concludes that a computer cannot understand. Forty-six years later, the question is more interesting than his conclusion.
An AI translates without speaking any language in the sense that you speak one. It explains pain without ever having hurt. It writes about grief without losing anything. And yet, the result is often useful, sometimes accurate, occasionally beautiful. What to make of this paradox?
What the system actually does
Let's go back to the facts. As we explained in our article on LLMs, a model predicts the most likely next word. It manipulates statistical regularities, not meanings.
But we need to go further than that observation, because it's often used to close the debate a little too quickly. To predict the next word well across the whole of human texts, a system is forced to encode structures that look strikingly like knowledge. It learns that Paris is in France, that objects fall, that a betrayed promise triggers anger. These relations aren't mere surface associations: they organise themselves, as shown by the principle of embeddings, where meaning becomes measurable geometry.
So: does it understand? The honest answer is that we don't have a definition of understanding sharp enough to settle it.
The problem lies in our definition
Here's the interesting twist. We talk about understanding as if it were obvious, but we'd be hard pressed to define it.
Is it the ability to use a notion correctly? Then models understand a great deal, sometimes better than we do. Is it the ability to explain what you're doing? They do that, albeit sometimes in a way reconstructed after the fact, which humans also do more often than they think. Is it the fact that there's something it feels like, a lived experience of understanding? Then we can't assert anything, neither about the machine, nor for that matter about others.
Because that's the point Searle leaves in the shadows: you have no access to your neighbour's understanding. You infer they understand because they behave like someone who understands. If we applied to our fellow humans the standard we apply to machines, we'd all be Chinese rooms to each other.
There's one difference that holds up: models have no anchor in the world. They've never touched an object, faced a consequence, or risked anything. Their words only refer to other words. That's precisely what explains their most baffling failures, like their clumsiness at basic arithmetic or their errors on intuitive physics. A child who has knocked over a glass knows something about gravity that no corpus conveys.
The practical question, which is different
Let's come down from the clouds of ideas. For everyday use, the real question isn't does it understand but where does its lack of understanding show?
The answer is fairly clear: it shows at the boundaries. On situations well represented in its corpus, the system behaves as if it understood, and the illusion holds. On an unusual case, a novel combination, a context no one has ever written, the mechanics are exposed. The model then produces a plausible answer that sounds right and isn't, without ever signalling that it has just stepped outside its domain.
That's why verification remains essential, and why it must intensify precisely when the question is original. The paradox is cruel: it's when you most need help that the tool is least reliable.
What to take away
Perhaps the question was badly posed. We long assumed that certain tasks—translating, summarising, arguing—necessarily required understanding. These systems show they don't: you can accomplish those tasks by another route. That's not a revelation about machines; it's a discovery about tasks.
Still, the word understanding continues to denote something for us, and it would be odd to give it up. Perhaps it denotes less a performance than a relationship: the fact that what you're talking about matters, has consequences, commits someone. On that count, an AI doesn't understand, not because it's made of computations, but because nothing it says costs it anything.
Which, incidentally, is also the best reason to keep a human in the loop: not because they're smarter, but because they're the one for whom the answer matters.