A model you can query is a model whose know-how you can extract. This is not a security flaw: it is a property of what these systems are. A model is built to answer, and its answers contain its knowledge. The only inextractable model is one nobody can access.
This morning we reported accusations of extraction on an industrial scale. Here is how it works, and why it is so hard to prevent.
The mechanism
You query a model with a very large number of questions, collect its answers, and train a second model to reproduce them. This is exactly the distillation we explained, applied to a model you do not own.
What makes the operation formidable is that it transfers far more than answers. The second model learns the first one's way of going about things: how it structures an explanation, what trade-offs it makes, how it handles ambiguity.
The economics are striking. Training a frontier model costs hundreds of millions. Collecting a few million answers costs a few tens of thousands of euros. You do not reproduce the model, you recover a substantial fraction of it for a tiny fraction of the price.
Why defences fail
Rate limiting. You can cap the number of requests per account. The attacker spreads across hundreds of accounts, goes through resellers or routing platforms. It slows things down, it does not prevent them.
Detecting suspicious patterns. An extraction produces a particular distribution of questions: very diverse, very broad, unrelated to any business use. It is detectable, and it can be evaded by mixing in legitimate traffic.
Watermarking outputs. You could introduce a signal that later reveals a model was trained on those answers. We have shown what text watermarks are worth: the signal degrades as soon as the text is reworded, and training itself dilutes it.
Hiding the reasoning. Some providers encrypt the internal scratchpad so as not to expose the method. We have documented how that encryption was circumvented.
Any serious defence degrades the service for legitimate users. Rate limiting inconveniences the big customers, who are the most profitable. Filtering unusual questions blocks research uses. Refusing diverse requests penalises exactly those who explore the most. A provider therefore has to arbitrate between protecting its model and serving its best customers. It is a commercial trade-off, and it almost always tips the same way.
What this changes in labs' strategy
If you accept that capabilities leak anyway, several choices become legible.
Publishing deliberately. This is the calculation we described regarding labs that release their weights under a permissive licence. If the knowledge escapes regardless, better to capture influence and the ecosystem than to try to hold on to it.
Differentiating elsewhere. A model can be extracted; an infrastructure, a proprietary corpus, a customer relationship cannot. This is exactly the shift we document, from the publisher building its model on its own corpus to the players investing in chips and platforms.
Accelerating. If a model can be extracted in a few months, the only lasting protection is to bring out a better one in the meantime. This partly explains the current pace of releases.
What to take away
Model extraction is a problem with no technical solution, and that is what makes it interesting.
It illustrates a general property of these systems that we encounter in several forms: what makes them useful is exactly what makes them vulnerable. A model that answers well exposes its know-how. An agent that reads your files can be hijacked by their contents. An assistant that remembers widens its attack surface over time.
There is no powerful version without a surface. So the question is never about eliminating the risk, but about knowing which one you accept and what you put up against it.