Updated daily · November 6, 2026
Stats
How sites write their robots.txt: the scores they earn, the mistakes they make, the crawlers they turn away and the rules they share. Drawn from the 1,189 current files on record.
- robots.txt files
- 1,189
- Average score
- 72
- Declare a sitemap
- 79%
- Have something to fix
- 67%
Scores
How the files grade
The median file scores 71 out of 100. How scoring works.
File size
- Smallest
- 13 B
- Median
- 198 B
- Average
- 900 B
- Largest
- 41.9 KB
Crawlers stop reading at 500 KB. Files that declare a sitemap declare 3.1 on average.
Mistakes
What files get wrong
67% of files have at least one thing to fix, and 34% have a warning or worse.
12% of files carry rules that never take effect
920 rules in all, each always overridden by another rule in the same group. They change nothing for crawlers and only add to the file.
- NoticeEvery URL open to every crawler32%
- WarningNo sitemap declared21%
- NoticeRules that can never take effect12%
- NoticeNo crawl rules declared6%
- CriticalDeclared sitemap could not be fetched5%
- NoticeUses Crawl-delay, which Google ignores5%
- WarningLines crawlers will ignore4%
- WarningNames a location nobody would guess3%
- NoticeRules declared more than once2%
- WarningDeclared sitemap does not return XML1%
- WarningNames a secret or data file1%
- NoticeBlocks everything by default1%
- WarningBlocks a user-triggered fetch while the same operator still crawls0%
- WarningSitemap reference is not an absolute URL0%
- CriticalWhole site closed to all crawlers0%
Crawlers turned away
Who gets blocked
Every file is checked against the same list of well-known crawlers, so these shares compare: a crawler counts as blocked when it may not fetch the site root, whether the file names it or blocks every crawler.
| Crawlers | Block some | Block all |
|---|---|---|
| AI training crawlers | 8% | 3% |
| AI answer engines | 5% | 1% |
| Search engines | 4% | 0% |
| User-triggered AI fetches | 2% | 1% |
| Social link previews | 1% | 0% |
The crawlers blocked most often
- BytespiderByteDance6%
- CCBotCommon Crawl6%
- GPTBotOpenAI6%
- ClaudeBotAnthropic5%
- meta-externalagentMeta5%
- Google-ExtendedGoogle5%
- AmazonbotAmazon4%
- BaiduspiderBaidu4%
- PerplexityBotPerplexity2%
- ChatGPT-UserOpenAI1%
- OAI-SearchBotOpenAI1%
- Claude-SearchBotAnthropic1%
- Perplexity-UserPerplexity1%
- Claude-UserAnthropic1%
- YandexBotYandex1%
Crawlers named by hand
Who files single out
The user agents files name in a group of their own, and whether that group turns them away or lets them in.
Named and blocked
- Bytespider6%
- CCBot5%
- GPTBot5%
- ClaudeBot5%
- Applebot-Extended4%
- meta-externalagent4%
- Google-Extended4%
- PetalBot4%
- Diffbot4%
- Amazonbot4%
- cohere-ai4%
- anthropic-ai4%
- omgili4%
- Claude-Web3%
- Baiduspider3%
Named and explicitly allowed
- PerplexityBot10%
- GPTBot10%
- ChatGPT-User9%
- Google-Extended9%
- ClaudeBot9%
- OAI-SearchBot9%
- Applebot-Extended6%
- Claude-User6%
- CCBot6%
- Perplexity-User6%
- Claude-SearchBot5%
- Googlebot5%
- anthropic-ai5%
- Bingbot4%
- Amazonbot4%
By category
Share of files that name a crawler of the category and block it, or let it in.
| Category | Blocked | Allowed |
|---|---|---|
| AI Data Scraper | 8% | 11% |
| Uncategorized | 6% | 2% |
| AI Search Crawler | 5% | 12% |
| Intelligence Gatherer | 5% | 4% |
| Search Engine Crawler | 5% | 7% |
| AI Assistant | 4% | 10% |
| Undocumented AI Agent | 4% | 6% |
| Developer Helper | 3% | 0% |
| SEO Crawler | 3% | 2% |
| Scraper | 2% | 0% |
| Archiver | 1% | 0% |
| Fetcher | 1% | 4% |
| Security Scanner | 1% | 1% |
| AI Agent | 0% | 1% |
| AI Coding Agent | 0% | 0% |
| Automated Agent | 0% | 0% |
Rules
The most common rules
The Allow and Disallow lines most files declare, in any group.
- Allow: /46%
- Disallow:15%
- Disallow: /13%
- Disallow: /api/10%
- Disallow: /wp-admin/7%
- Allow: /wp-admin/admin-ajax.php6%
- Disallow: /admin/4%
- Disallow: /login4%
- Disallow: /search/3%
- Disallow: /dashboard3%
- Disallow: /Admin3%
- Disallow: /cdn-cgi/3%
- Disallow: /search3%
- Disallow: /account2%
- Disallow: /account/2%
Method
How this is worked out
- Each site counts once, by the newest fetch of any of its URLs, since every page of a site shares one robots.txt.
- Reports older than 7 days have expired and are left out, so the figures describe files as they were last seen.
- Files pasted or uploaded on this site are private and never counted.
- Of the sites analysed in the week before these figures, 7 sites blocked our bot in robots.txt, 129 sites blocked it with bot protection, 9 sites asked it to slow down, 57 sites could not be reached, 113 sites had no robots.txt. Only the files of sites that let the bot read them are part of these figures.
- Rules and crawler names are listed only when at least two files share them, so no list points at a single site.
- The figures are worked out again every day. The files come from sites people analyse here and from the sites our bot visits.