When you point out that no one really understands what goes on inside an AI, a common reply is: so what? You don't understand your engine, your medicine, or the bridge you cross either. That's true, and it's a good argument. It deserves to be taken seriously rather than dismissed, because examining it reveals exactly what's missing.
We explained in our article on interpretability that the opacity of models isn't commercial but structural. The question remains: does it matter?
What makes trust rational elsewhere
You don't understand how a plane works, and you get on board without a second thought. That trust isn't naivety: it rests on four conditions we never spell out because they're met.
Someone understands. Engineers know, and that knowledge is verifiable, transferable, and taught in schools.
There's a track record. Millions of flights, public accident statistics, decades of data. Reliability is measured, not assumed.
Failures are analysed. Every aviation incident is subject to an independent investigation whose findings are published, and which changes procedures. The system learns from its mistakes, publicly.
Someone is accountable. A manufacturer, an airline, a certification authority bears responsibility. There's a name at the end of the chain.
Let's compare honestly
Let's apply these four criteria to cutting-edge AI.
Does someone understand? Only partially. The architectural principles are known and taught. But no one can explain why a specific response was produced, and that's a difference in kind from an engine.
Is there a track record? It's building up, but it's short and models change every six months. A reliability statistic for a model that no longer exists isn't worth much, which the rapid retirement of models makes concrete.
Are failures analysed? Partially, and on a voluntary basis. We welcomed the fact that some labs publish their incidents. But there's no obligation, no independent investigation, and we've noted the risk that holding those who report to account discourages reporting.
Is someone accountable? This is the weakest point, and we devoted an entire article to it. The allocation of responsibility between the builder, the deployer, and the user is not established.
There's an asymmetry the comparison masks. A medicine always does the same thing. A bridge either holds or it doesn't. These objects have stable and testable behaviour. An AI produces decisions in infinitely varied situations, including situations no one anticipated. You can't exhaustively test an infinite space. That's why safety tests sometimes fail to contain what they measure: the problematic behaviour appears precisely where no one had looked.
What this means in practice
The conclusion isn't to refuse to use these tools, which would be absurd and a losing game. It's to calibrate your trust to the conditions actually met.
For a use where errors are visible and recoverable, opacity doesn't matter. You judge the result, and that's enough.
For a use where the error is invisible, the question changes. That's the case we described regarding scientific code: a program that runs perfectly while producing a slightly wrong result. There, not understanding becomes a problem, because nothing will alert you.
What to take away
The right question isn't must you understand to trust. It's what makes trust justified in the absence of understanding.
The answer exists and is well known: institutions that verify, a measured track record, failures analysed publicly, and someone who is accountable. These four things were built over a century for aviation, over fifty years for medicine. They aren't there yet for AI.
There's nothing abnormal about that: they're always built after the fact, usually following accidents. What one can hope for is that they're built before rather than after. And in the meantime, caution isn't technophobia: it's simply rational behaviour when the conditions for trust aren't met.