Who the new web lets read it
Every site we profile hands us its robots.txt before we fetch anything, because we have to read it to know whether we may. That same file says which AI crawlers a site turns away. This page is what it adds up to, measured on sites at the moment they launch rather than on an established sample.
Measured on 11,717 live sites so far. Every site gets measured on its next crawl, so this number grows for a while yet.
New sites against established ones
Sites whose domain was registered in the last 30 days, against everything else we have measured, which is mostly the established web. The same check, run on both.
| Blocks an AI crawler | Has an llms.txt | Sites measured | |
|---|---|---|---|
| Registered in the last 30 days | 1.6% | 14.8% | 128 |
| Everything else | 5.9% | 12.7% | 11,589 |
Share of new sites blocking AI crawlers, by day
Each point is the share of the sites first seen that day that disallow at least one AI crawler from their root, and the share that publish an llms.txt.
Which crawlers get turned away
Counted per site: one robots.txt naming five bots counts once for each of them. A site is only counted when it disallows the bot from the site root, not merely from a directory.
llms.txt
llms.txt is the other side of the same choice: a plain-text file at the root telling a language model what the site is and which pages are worth reading. It is a young convention, so the interesting number is not how large it is today but how fast it moves.
A caveat worth knowing before quoting the figure: a good share of these are not a decision anyone made. Several site builders now write an llms.txt by default, and it shows up as byte-identical files on unrelated domains. The count here is of files served, not of deliberate choices.
The most recent sites we have seen with one:
- boxing-master.com BOXING MASTER – ボクシングマスターが最新の世界ボクシングニュースをお届け。
- guardsjobs.com Client Challenge
- venithy.com VENITHY - Marketplace Premium Bien-être & Performance
- camminatametabolica.it Camminata Metabolica® | Camminata Metabolica
- bellafootwears.com Bella Footwears | Where Fashion Meets Comfort!
- atelierultau.ro Atelierul Tau.ro - Magazin online de Scule Electrice
- vkf-renzel.de VKF Renzel | Online-Shop für Verkaufsförderung | seit 40 Jahren
- evergreenconsultants.com.au Evergreen Consultants | Managed Accounts Specialists | Asset Consultants
- stack-auth.com Hexclave Deploy — You build it. We run it.
- axaipay.my Axaipay – Online Payment Gateway for Malaysia, Australia & ASEAN
- fastaccounting.jp ファーストアカウンティング – 経理シンギュラリティで 経理パーソンの力を解き放つ
- annuaire-cotisations.com ALIBARBAR Vape - Australia Online Vape Store
How this is measured, and what it misses
- The crawlers counted are the ones that collect text for AI training, retrieval or answers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai and Diffbot. Ordinary search crawlers are not in the list: blocking Googlebot means something else entirely.
- Group matching follows the robots.txt rules a real crawler would apply, so a wildcard
Disallow: /with anAllowfor a named bot counts as allowing that bot. - robots.txt is a request, not a fence. This measures what sites ask for, not what crawlers do.
- A site with no robots.txt is counted as blocking nobody, which is what it means, but the "have a robots.txt at all" figure above is there so that share is visible rather than hidden in the denominator.
- Sites are measured once per crawl, so a site that changed its rules yesterday shows its old answer until its next visit. See how this works for crawl frequency.