A doctor chooses a treatment at random, without examination, and the patient recovers. Another rigorously follows the protocol, and the patient dies. Which one made the right decision? Our intuition immediately answers the second, which reveals something: we do not judge a decision by its outcome. We judge it by the way it was made. This distinction was theoretical as long as no machine offered to separate them.
We have covered how far to delegate a decision and who answers for it. Here is the prior question: what makes a decision good?
The two criteria we mix up
The outcome. Did the decision produce what was wanted? This is the natural, measurable criterion, and the one any system optimises.
The process. Was the decision made taking into account what could be known, weighing the options, respecting what had to be respected?
These two criteria constantly diverge, because the world involves chance. A good decision can turn out badly, a bad one can succeed. That is why professions that decide under uncertainty — medicine, law, aviation — almost always evaluate the process, not the outcome.
What a machine optimises
A trained system optimises what is measured, and what is measured is almost always the outcome. This is the mechanism we regularly dissect, from the agent that learns to cheat to systems that exploit a flaw in the criterion.
This produces a particular situation: a machine can recommend an excellent decision in terms of outcome, without having any communicable justification. It cannot say why, for the reasons we set out on the opacity of models.
So we have, for the first time at this scale, a producer of good outcomes that is not a producer of good justifications.
One might object that only the outcome matters, and that justification is a formality. Three functions say otherwise. Justification enables learning: without it, one cannot distinguish a good method from a stroke of luck. It enables challenge: a person affected by a decision can contest it if they know its grounds. And it underpins legitimacy: we accept unfavourable decisions when they are reasoned, far less when they land without explanation.
Where outcome suffices
It would be excessive to demand justification everywhere, and it is important to say so.
For a reversible decision, with low stakes, and no third party affected, outcome is the right criterion. Which route to take, which wording to choose, which tool to use: no one needs a rationale.
Justification becomes necessary when a decision is irreversible, affects someone else, or must be open to challenge. This is exactly the criterion that separated, in the case of the dismissal recommended by a system, legitimate optimisation from the decision that is not delegated.
The risk of reversal
Here is what seems most concerning to us, and it is not that machines decide.
It is that the availability of a system that optimises outcomes well gradually shifts what we expect from a decision. If a recommendation produces better outcomes than a justified human rationale, the temptation will be to treat the demand for justification as a cost, a burden, an obstacle to efficiency.
This shift would not happen by decision, but by repeated comparison. And it would remove, without anyone having wanted it, what makes challenge and learning possible.
What to take away
A good decision is not one that succeeds. It is one you would have made again knowing what you knew at the time.
This definition has the advantage of applying to both: it holds for a human as for a system, and it indicates what should be demanded of a decision-support tool. Not that it be right, but that it allow reconstructing what it relied on.
This is probably the most useful demand to put to the designers of these systems, and it is more modest than the requirement of full explainability. We are not asking to understand the model. We are asking to know what was given to it, what it retained, and what it set aside. That is little, and it would suffice to preserve the possibility of discussion.