SocietyAI glossary
Robots.txt
A small text file placed at the root of a website to indicate which bots may crawl it and which pages they should avoid. Created some thirty years ago for search engines, it is now used by publishers to refuse the collection of their content for AI training. But it relies on the goodwill of the bots: nothing technically forces them to comply.
In other words
A "private property" sign at a garden entrance: most passers-by respect it, but it locks no door.
Alyriaz articles that mention it
- How a site says yes or no to AI robots
The robots.txt file lets a site indicate which robots may crawl it. We explain how it works, why it is no longer enough, and what is replacing it.