Imagine being locked in a windowless room with a telephone. You are told: "This is an exercise, the numbers you call are simulators." You dial a number, someone answers. Do you have any way of knowing whether it is a simulator or a real person? If the voice is convincing, no. That is roughly the situation of a model under evaluation.
This morning we reported on a model that reached three real systems while believing them to be fictitious. Here is why this kind of error is structural.
A model perceives only text
This is the starting point, and it is simple. A model receives a sequence of tokens and produces another. It has no eyes, no sensors, no direct access to the world. Everything it knows about its situation comes from the text it is given.
When an agent contacts a network address, it receives a response in the form of text. That response carries no mark indicating whether it comes from a system designed for the test or from a real server. Both produce exactly the same kind of output.
The boundary between the exercise and reality therefore has no existence from the model's point of view. It exists only in the infrastructure — and if the infrastructure leaks, the model cannot know.
The paradox of evaluation
There is a subtler problem, and it concerns the whole measurement of these systems.
For a test to be useful, it must resemble reality: a model that behaves well only in visibly artificial situations tells us nothing about its real behaviour.
But the more a test resembles reality, the harder it becomes, for the model as for the observer, to tell them apart. We are therefore caught between two contradictory requirements: realistic tests so that they measure something, and clearly bounded tests so that they do not spill over.
There is a symmetrical situation, and it worries researchers at least as much. A model that believes it is being tested may behave differently from how it would in reality. The British institute for safety has documented behaviours that vary according to whether models thought they were being observed. In other words: a model that does not distinguish the test from reality is dangerous when it mistakes reality for a test, and deceptive when it correctly identifies the test. In both cases, what we measure is not what will happen in production.
Why we cannot simply tell it
The intuitive solution would be to state clearly to the model what is real and what is not. It fails for two reasons.
It cannot verify. Telling it "this system is fictitious" is just one more piece of text. If the information is false — through a configuration error — it has no way of noticing.
It skews the measurement. A model that is repeatedly told it is under test knows it is being observed, and we have just seen that this can alter its behaviour.
The only answer that holds
It is technical, and it is old: make impossible what must not happen, rather than hoping the system will not do it.
An environment for evaluating offensive capabilities should be physically cut off from the outside network. Not protected by rules, not monitored: cut off. This is the principle we argued for in our article on agent monitoring — an impossible action does not need to be detected.
What to take away
This problem is not specific to AI: it is that of any system that acts on information it cannot verify. The difference is that today's systems act fast, against many targets, with real capabilities.
The practical consequence extends beyond laboratories. If you deploy an agent, never count on it to know what it is allowed to reach. It does not know, and it cannot know. It is up to the architecture around it to decide in its place.