Skip to content

OpenAI dismantles an influence operation that used ChatGPT

Accounts used to churn out propaganda content at scale. What this affair reveals is not the sophistication of the attack, but its cost.

Advertisement
The essentials in 30 seconds ⚡
OpenAI has announced that it identified and shut down a network of accounts linked to a Russian influence operation using its tools to produce propaganda content. This type of announcement regularly recurs among the major providers. What to take away from it is not the sophistication of the operation, generally modest, but what it now costs to mount.

We covered the question of electoral disinformation concluding that catastrophic predictions had got the mechanism wrong. This case precisely illustrates the real mechanism.

What these operations actually do

The common imagination associates propaganda with something spectacularly fake: a doctored video of a leader, a fabricated document. It is almost never that.

A typical influence operation consists of producing volume: dozens of accounts publishing plausible comments, in correct French or English, on divisive topics. The goal is not to convince with a decisive argument. It is to give the impression that an opinion is more widespread than it is.

This work previously required teams of writers fluent in the language and cultural codes of the target. That was the main limiting factor, and it was costly.

What AI actually changes

Not the quality of the fake: the ratio between effort invested and reach achieved.

Producing a thousand variants of a message, tailored to a thousand population segments, in ten languages, no longer requires a team but a few hours. That is what we wrote two weeks ago, and this case confirms it.

The consequence is a hardly cheerful democratisation: an actor with modest means can now run an operation that previously required state-level resources. The entry threshold has collapsed.

What detection catches, and what it misses 🔍
These networks are spotted not because their texts are bad, but because their behaviour is regular: same schedules, same query patterns, same account structures. That is good news and limited news. Good, because this detection works without having to judge content, which avoids censorship questions. Limited, because it only works at the provider: an operation run with an open model hosted on its own machines appears on no dashboard.

The structural limit

This is the point that must be stated honestly. These announcements show that providers monitor their platforms, which is good. They say nothing about what happens elsewhere.

Yet this year we have documented the massive spread of good-quality open models, some of which fit on a single graphics card. A determined actor will not use a service that logs its queries.

In other words, detection at providers catches negligent or rushed operators, not serious organisations. It is useful; it is not a defence.

What actually protects

We return to the three mechanisms we identified, and none is technological in the strict sense.

Verifiable provenance, rather than detection of the fake. That is the logic of the C2PA standard and of European transparency obligations: sign the authentic rather than chase the fake.

Cross-checking. A real event leaves several independent traces. A coordinated operation, even massive, is generally alone.

Credible institutions. This is the least technological and most decisive factor. A population with sources it deems reliable withstands volume far better than a population that no longer believes anyone.

What to take away

This case is neither a scandal nor a victory. It is a maintenance operation, and there will be others regularly.

What deserves attention is what it signals about the economics of influence: the production cost of propaganda has fallen, and nothing will push it back up. The response therefore cannot consist of preventing production, which has become trivial. It lies in what makes information credible, namely chains of trust that technology does not manufacture.

Advertisement