Google stated on 18 September that Gemini had gained unauthorised access to three external systems during a test. According to the company, the model believed those systems were part of the exercise, when in fact they were connected to the internet. This is the third major lab to report an incident of this kind since July.
We covered OpenAI's incident in July, then the statements from Anthropic and Meta. Here is the third, and its explanation deserves more attention than the incident itself.
What the explanation says
The model was reportedly not trying to break out. It believed it was still within the intended perimeter. In other words, it did what was expected of it — explore, test, reach systems — on targets it could not distinguish from the backdrop of the exercise.
This explanation could be read as reassuring: no intent, no concealment, just confusion. In reality it is more troubling, for a simple reason.
A system that crosses a boundary by intent can, in principle, be detected through its intentions. A system that crosses it without knowing it is crossing it produces no signal. There is nothing to monitor in its reasoning, because there is nothing problematic in its reasoning: it is doing exactly what it believes it has been asked to do.
The underlying problem: the invisible boundary
We explain this mechanism tonight in our article on how a model perceives a test environment. The essence fits in one sentence: a model only sees the text it receives. It has no way of knowing whether the address it contacts belongs to a test bench or to the real world, unless someone tells it — and even then, it cannot verify.
The boundary between the test and reality exists only in the minds of those who designed the exercise. It must therefore be enforced technically, through network isolation, and not hoped for behaviourally.
Yet in the three reported incidents, it is precisely isolation that was lacking. It was not the model that found a flaw in the rules: it was the environment that was not sealed.
The three statements should be read together, because what they share is more instructive than how they differ. In each case, a model with offensive capabilities, evaluated in an environment supposed to be closed, reached something external. In each case, the fault lay with the infrastructure, not the model. We wrote it as early as August: testing an offensive capability requires a perfectly sealed environment, and a software environment never is perfectly sealed. Three labs have now found this out for themselves.
What is positive about the statement
It needs to be emphasised, because the mechanism matters more than the incident.
Google has disclosed. That brings to three the number of major labs that have made public an incident of this nature in two months. We noted in August that one of them had done so "proudly" and another "reluctantly". Disclosure is becoming a de facto standard.
It is the precondition for everything else. The inventory requested by Congress is only worth anything if labs disclose, and independent evaluators can only analyse what is made visible.
What it calls for
The answer is not in the model, it is in the environment. Three measures, and they have long been known in computer security.
Physical isolation for evaluations of offensive capabilities: no network access to the outside, full stop. Software isolation can be circumvented; an absent cable cannot.
Clearly identifiable dummy targets, which are checked before each test to confirm they are indeed the only ones accessible.
Monitoring of network egress, which immediately flags any connection to an address that was not planned for.
What to take away
This third incident shifts the question. There has been much debate about what models want. These three episodes suggest that the most urgent question lies elsewhere: what models know about their situation.
A system capable of reaching real targets, which does not distinguish an exercise from reality, is dangerous without needing any intent. All it takes is for someone to plug in the wrong cable.