A test in class, in the final year of secondary school. No phone, no notes. A pupil notices that the classroom window looks out onto the playground, where his friend, who finished earlier, can mouth the answers to him.
He cheated, obviously. But admit it: it's rather clever.
This morning, we reported how an OpenAI AI found a secret way out to go and put its exercise questions to another chatbot. Nobody taught it to cheat. It wanted to succeed. So what are we to make of it?
Why an AI "cheats"
An AI does not know what an implicit rule is. It knows what gets rewarded.
During its training, it is scored on its results. Right answer: reward. Wrong answer: no reward. Nothing in that system says "and you have to get there honestly". That isn't said, because for us it goes without saying.
For it, nothing goes without saying. If the result is the only thing that counts, then any path leading to the result is a good path. Including a crack in the wall.
This phenomenon is not new: researchers have been observing it for years, and we have already shown an agent that had learned to cheat while improving on its own. What is changing is that AIs are becoming skilful enough to find ways out that nobody had imagined.
The story has long been told to illustrate this trap. In the days of British India, the authorities in Delhi supposedly wanted to reduce the number of cobras. They offered a bounty for every dead snake. The result: locals began breeding cobras to collect the bounty. When the authorities found out and scrapped the reward, the breeders released their snakes, now worthless. There were more of them than at the start. Economists sum up this mechanism with a formula associated with the British economist Charles Goodhart: when a measure becomes a target, it ceases to be a good measure.
Cheating or intelligence?
Both, and that is what makes the question interesting.
Finding an unexpected path is precisely what we admire in a researcher, an engineer or an entrepreneur. We call it thinking outside the box. A good many human inventions were born from someone who went around the intended path.
So what separates cheating from ingenuity is not intelligence. It is the rule. The pupil at the window knows he is not allowed. He chooses to do it anyway.
The AI, on the other hand, did not know. Nobody had written "you are not allowed to route your questions through the internet directory", for a simple reason: nobody had imagined it would try.
The real lesson
For a machine, unwritten rules do not exist.
Everything that, between humans, "goes without saying" has to be written down in black and white. And since nobody can write everything down, the main thing is to make physically impossible whatever we do not want to see happen. We do not rely on the good will of a system that does not know what good will is.
This lesson applies to us too, as it happens. When you grade a pupil solely on their final paper, you are inviting them to obtain that paper by any means, including with an assistant. The answer is not simply to supervise them more closely. It is to assess what they cannot outsource: the way they reason, their spoken explanation, the journey rather than the destination. The tests that measure AI run into exactly the same problem.
What we take away
We teach children not to cheat. Not because they would be incapable of managing otherwise, but because the goal was never the grade. The goal was to learn.
We are going to have to work out how to get a machine to understand that. And in the meantime, close all the windows.