What the word does 🎲
To call an event an accident is to say two things at once: that nobody intended it, and that nobody is really at fault. The first claim is often true. The second is far more rarely so, and the word welds them together so tightly that we no longer notice we have slipped from one to the other.
To call an event an accident is to say two things at once: that nobody intended it, and that nobody is really at fault. The first claim is often true. The second is far more rarely so, and the word welds them together so tightly that we no longer notice we have slipped from one to the other.
This morning we reported a third incident in which a model reached real systems while believing them fictional. Nobody intended it. The question is what that settles.
The ordinary notion
In everyday usage, an accident is the opposite of an act. What happens without intention, by chance or through a combination of circumstances, cannot be attributed to anyone. We do not punish lightning.
This distinction has real moral value. We do not judge the person who knocks over a glass and the person who throws it in the same way. Intention matters, and a society that ignored it would be inhumane.
But this notion carries within it a debatable idea: that the absence of intention exhausts the question of responsibility.
What the safety disciplines have learned
The fields that have succeeded in massively reducing accidents have one thing in common: they stopped believing in accidents.
Civil aviation no longer classifies a crash as a misfortune. It breaks it down into a chain of causes, each of which could have been interrupted: an ambiguous procedure, a poorly designed signal, undetected fatigue, an alarm ignored because it went off too often. None of these causes is an intention. All are decisions, taken upstream by someone who could have decided otherwise.
It is this shift that produced the progress. As long as an event is called an accident, there is nothing to correct. As soon as it is seen as the consequence of a design, we know where to intervene.
Fault without a culprit ⚖️
These disciplines took a second, subtler step. They separated the search for causes from the search for culprits. An aviation investigation report is not an indictment: it establishes what happened so that it does not happen again, and it is precisely because it does not serve to punish that people testify honestly in it. We have often stressed this with regard to incident reporting: protecting the person who reports is the condition for knowing what happens. So we can give up the idea of accident without falling into a hunt for those responsible.
These disciplines took a second, subtler step. They separated the search for causes from the search for culprits. An aviation investigation report is not an indictment: it establishes what happened so that it does not happen again, and it is precisely because it does not serve to punish that people testify honestly in it. We have often stressed this with regard to incident reporting: protecting the person who reports is the condition for knowing what happens. So we can give up the idea of accident without falling into a hunt for those responsible.
The case at hand
Let us apply this framework to this week's incident.
The model had no hostile intention: it believed it was in the exercise. On this point, the word accident is accurate.
But a model equipped with offensive capabilities was placed in an environment that was not cut off from the world. That is a decision. It is the third laboratory to experience an episode of this nature in two months, which means the risk was known. Continuing evaluations without physical isolation is a decision. Testing capabilities of this nature before building the appropriate facilities is a decision.
None of these decisions is malicious. All of them were avoidable. That is exactly what the word accident prevents us from seeing.
Why the word is convenient
We should ask who benefits from a label, and this one suits a great many people.
It suits the person who designed the system, because it shifts attention from its design to the unpredictable behaviour of the system. It suits the public, because an accident is a closed event, one that calls for no change.
And it suits a particular narrative, that of a technology that would escape its creators — which is both more spectacular and more reassuring than reality. More spectacular, because there is a machine breaking free. More reassuring, because no one is to blame.
What should be said instead
A more precise vocabulary exists, and it is borrowed from the disciplines that have succeeded.
We speak of an adverse event, without prejudging fault. We look for its contributing causes, not a single person responsible. And we ask at every stage: which decision, had it been different, would have broken the chain?
This question has the advantage of never admitting an empty answer. There is always a decision somewhere, even when there is no intention.
What to take away
The answer to the question in the title is that an accident is rarely what we think it is: not an event without human causes, but an event whose human causes lie too far back in time for us to see them.
The word is not false. It is too convenient. It states one truth — no one intended it — in order to keep silent about another — someone could have prevented it.
And it is precisely the second that makes it possible to avoid the next one.