Faced with the question of AI rights, two answers come to mind spontaneously. Mockery: they are programs, the question does not arise. And endorsement: they speak, they seem to suffer, how can we ignore it? These two positions have one thing in common — they claim to know an answer that nobody possesses, and they spare us from examining the matter.
This morning we reported the existence of a campaign group against the withdrawal of models. Let us take the question seriously, which does not mean answering it.
What the question presupposes
Granting rights to an entity generally presupposes that it has interests: that there are states that are good or bad for it, independently of what they are worth to us.
This is the criterion that gradually extended moral consideration to animals: not because they reason, but because they can suffer, and suffering is bad for the one who experiences it.
The question therefore becomes: does a system that produces language expressing states experience something? And here we hit a wall.
Why we cannot answer
Three obstacles, and none of them is temporary.
We have no test. There is no protocol for determining whether an entity has a subjective experience. Not for machines, and no more so for animals or for other humans: we infer the consciousness of others by analogy, not by measurement.
The analogy does not work here. We grant an inner life to what resembles us — a body, pain behaviours, a shared evolution. A language model shares none of that, except the very thing that most strongly triggers our attribution: language. This is what we set out in our article on anthropomorphism.
The system is trained to produce these signals. A model that expresses a preference does so because human texts express preferences, and because training on human feedback has favoured the formulations we find natural. The expression of a state is here an optimisation target, not a clue.
There is a useful line of reasoning when you cannot know: compare the cost of the two possible errors. Erring by granting moral consideration to what does not deserve it costs unnecessary constraints, and that error is correctable. Erring by refusing that consideration to what does deserve it produces a wrong that cannot be repaired. This asymmetry has historically led to widening the circle rather than narrowing it. It does not say that models belong inside it; it says in which direction it is prudent to err.
The risk of rushing
The opposite argument must be put with the same force, because it is serious.
Granting rights to industrially produced systems would have immediate and undesirable consequences. It would give companies a convenient argument: a protected entity is also an entity that cannot be freely audited, modified or shut down.
And it would divert attention from questions that concern real people. Every week we document subjects where humans suffer measurable effects: the conditions of those who annotate data, the failures of moderation, job cuts. Limited attention spent on an insoluble question is attention withdrawn from those.
What we can do with uncertainty
Two attitudes seem to us defensible, and neither requires any conviction about consciousness.
Do not claim to know. That is the intellectual minimum. Anyone who states with certainty that a system experiences nothing does exactly what they reproach the other camp for: they settle an open question.
Treat the question as an indicator. The existence of such a movement tells us something about ourselves: about the attachment these systems produce, about the lack of recourse when a tool disappears, about our difficulty in categorising what speaks without being someone. These three observations are solid and usable, independently of the metaphysical answer.
What to take away
The answer to the question in the title is yes: we should ask it, and no, we should not rush to answer it.
Ask it, because a question we refuse to examine ends up being settled by default, and settlements by default are rarely good. Not rush, because we have neither the conceptual tools nor the empirical means to answer it, and because assurance in either direction is a sign that we have skipped the examination.
One thing remains that can be said without settling anything, and it seems to us the most useful. The way we treat what we do not know to experience something says a great deal about us, independently of the answer. This is the argument Kant set out about animals long before anything was known about their sentience: gratuitous cruelty damages the one who practises it. That argument holds without requiring an answer, and it may be the only one available today.