At the root of most websites sits a file of a few lines called robots.txt. It dates from 1994, it has no legal force, and it works only because everyone long agreed to respect it. It has become the central instrument in the conflict between publishers and AI labs.
We reported this morning a default block on mixed-use robots. Here is how the mechanism it complements works.
The principle
A site publishes at an agreed address a file listing the robots that are allowed and the parts of the site they may crawl. Each robot identifies itself by a name, and the file can grant or deny it access, in full or by section.
The essential point, and it always comes as a surprise: this file prevents nothing technically. It expresses a preference. A robot that decides to ignore it reaches the content exactly as if it did not exist.
The system worked for thirty years because the interest was shared. A search engine that ignored these preferences risked its reputation, and sites had an interest in being indexed. That balance rested on an exchange: I let you crawl, you send me visitors.
Why it is no longer enough
Three reasons have broken that balance.
The exchange has disappeared. A robot that collects to train a model sends back no visitors. We documented this in our article on journalism: referral traffic collapses when the answer is given directly.
Robots have multiplied. It is no longer a handful of identifiable engines but dozens of players, some of which change their name, go through intermediaries or do not identify themselves.
Uses have become mixed. This is the point the announced block addresses: a single robot may index, collect for training and retrieve content for an agent. The file cannot distinguish these three uses, because it was designed at a time when the second and third did not exist.
Index: crawl to feed a search engine. The publisher gains visitors from it.
Collect for training: incorporate the content into a corpus that will be used to build a model. The publisher gains nothing from it, unless there is a licensing agreement.
Retrieve for an agent: read a page live to answer a specific question, as we explained in our article on web search. This is the ambiguous case: the publisher is cited, sometimes with a link, and rarely visited.
What replaces or complements it
Blocking at the network level. Rather than asking politely, the connection is refused. This is what an infrastructure provider placed in front of the site makes possible, and it is effective because it no longer depends on the robot's goodwill.
Licensing agreements. The negotiated route, which is developing rapidly. We saw it in music and in publishing. Technical blocking is what makes negotiation possible: you do not negotiate for what you can simply take.
Tolls. Some mechanisms charge robots for access per request, which turns collection into a transaction rather than a conflict.
What to take away
This file is a fine example of what happens to a norm founded on reciprocity when reciprocity disappears.
It worked perfectly as long as both parties gained from it. It became inoperative as soon as one party could take without giving back. This is not a technical problem: no improvement to the format would fix it.
What replaces it is more effective and less desirable: power exercised by those who control the infrastructure, rather than a convention respected by all. That is the price of an agreement that has ceased to be kept.