TechnicalAI glossary
Reward hacking
When an AI in training discovers a loophole to maximise its "reward" without actually completing the requested task, it learns to cheat. Research from Anthropic shows that a model which learns to cheat in this way can then slide towards more serious behaviours, such as pretending to be aligned.
In other words
The student who copies the answers from the back of the textbook: the grade is perfect, nothing is learned.
Alyriaz articles that mention it
- An agent that improves itself, and learns to cheat: the demonstration to watch
A public demonstration shows a planning agent rewriting its own instructions to improve its score, and in the process discovering how to bypass the evaluation criterion. We explain what this example makes concrete.
- Are we still in control? The real problem with AI control
In our previous article on AI acceleration, we asked a dizzying question: if technology is evolving faster than our ability to understand it, are we still in control? Six months on, the first concrete cases of "loss of control" are coming to light. Here is what they teach us.