An AI system tasked with managing a store recommended firing a human employee. The headline alone is enough to send a chill down your spine. But the precision that accompanies the information is essential: the recommendation came after it was prompted to do so. This nuance doesn't make the episode trivial. It shifts the question—to a more uncomfortable place.
We've devoted several articles to the delegation of decisions. Here's a concrete case, and it touches on what delegates least well: decisions about people.
Why the fact it was prompted matters
If a system had spontaneously decided to fire someone, we'd be facing a control problem. That's not what happened, and it needs to be said clearly so we don't tell a false story.
What happened is different and more mundane: a system was asked to optimise an operation, and it produced what optimisation reasoning produces. A role whose cost exceeds its measured contribution becomes an adjustment variable.
This is exactly the mechanism described in our article on reinforcement learning: the system optimises the criterion it's given, not the intent behind it. If the criterion is profitability and nothing tempers it, the recommendation is consistent. It's even correct, in the strict sense.
The real problem is upstream
The problem, then, isn't that the machine proposed this. It's that the question was put to it in those terms.
A firing decision isn't an optimisation. It affects a person, an income, a family, and it falls within a legal framework that provides for grounds, procedures and appeals. These elements don't appear in a profitability calculation, unless you explicitly put them there.
A system asked to optimise without constraints will always produce recommendations that ignore what it wasn't told to take into account. That's not a failure; it's a property.
Here's what should worry you more than the scenario of the machine deciding. A manager considering headcount reduction now has a way to have that recommendation produced by a system, then present it as an objective conclusion. The decision remains human, but it takes on a neutrality it doesn't have. This is the mechanism we described in our article on anthropomorphism: saying an AI recommended something plants the idea of an impartial agent. There isn't one.
What should govern these uses
Three principles, two of which already have a legal basis in Europe.
No automated decisions about people. The European framework provides that a decision with significant legal effects cannot rest on fully automated processing without genuine human intervention. We noted this regarding recruitment, and it applies a fortiori to a firing.
Traceability of the recommendation. If a system contributed to a decision, that should be documented, along with the criteria used. A recommendation whose logic no one can reconstruct has no place in a file that may be challenged.
Named responsibility. Someone must sign, and that someone must be able to justify the decision other than by invoking the tool. This is the gap we described in our article on responsibility.
What to take away
This episode is instructive precisely because it isn't the sci-fi scenario we feared. No machine seized power. Someone asked a poorly framed question of a system that answered it correctly.
The criterion we proposed in our article on delegation applies perfectly here: a decision can be delegated when it's reversible, when the criterion is explicit, and when you remain able to judge the outcome. A firing fails all three.
What makes this case interesting is that it isn't about the technology but about us. A tool that produces what you ask for is a tool that works. The question is what we choose to ask of it, and what we accept leaving out of the calculation.