RobotsTxtParser

What the bot does

For each site it analyses, the bot makes at most three kinds of request, and follows no links:

• /robots.txt at the root of the site, following up to 5 redirects.
• The one page being analysed, only if your robots.txt allows it. The bot reads the response headers and the <head>, for X-Robots-Tag and robots meta tags. It stops reading at </head> or after 1 MB, and never runs scripts.
• The sitemaps your robots.txt declares, to check they exist: the first 10, one at a time, and none after 20 seconds. Only the first 2 KB of each are read, enough to recognise a sitemap.

A site gets analysed in one of two ways: someone enters its address on this site, or it is on a list of sites the bot checks on a schedule. Once a site has been analysed, it is checked again on that schedule each time its result expires. Scheduled checks are slow on purpose. One site is fetched at a time, with 15 seconds between sites. A URL is fetched at most once every 7 days. A site that could not be reached or asked for fewer requests is left alone for 1 hour, then twice as long after each failure in a row, up to 7 days. A Retry-After header is honoured, up to the same limit. A site that refuses the bot or shows it a challenge is left alone from a day, up to 30 days. A Crawl-delay addressed to the bot spaces its requests to your site within one analysis; it waits at most 10 seconds in all, and skips the requests that would take longer.

What it collects

For each analysis, the bot stores:

• The robots.txt file itself, up to 500 KB, with its status code, size, final address and any redirects.
• The X-Robots-Tag headers sent with robots.txt and with the analysed page.
• The robots meta tags of the analysed page, and the status code it answered with.
• For each declared sitemap: whether it answered, its status code and content type, and whether it looks like a sitemap.
• The audit findings and score worked out from all of the above.
• The site's icon, shrunk to 64 pixels and shown beside its name. The bot looks for it again once it is 30 days old, or once per 7 days while it has none: first among the icons the home page declares, then at /favicon.ico, skipping any path robots.txt forbids it.

It does not store page content, cookies, form data or anything about the people who visit your site.

How long it is kept

Each report is kept in full for 6 months. After that, the reports of each domain are folded into one summary per month: how many reports there were, how many URLs they covered, and the best, average and lowest score. The reports themselves are then deleted.

The one exception is the newest report of each URL, which is kept until a newer one replaces it. It is the site's current state, so the ranking and the domain pages still have something to show.

A robots.txt pasted or uploaded on this site is never published. It is deleted after the same 6 months, with no summary kept.

Why

To show the state of the web when it comes to robots.txt: how sites use it, which crawlers they name, where files have mistakes, and how that changes over time. Each file is audited against the rules and scored out of 100, so sites can be compared. How scoring works.

Where the data is published

• The ranking on the home page shows six domains of each grade, picked at random every 30 minutes, each by its best current score, with totals for every site analysed.
• A page for each domain, at /score/{domain}, lists every report of the domain, newest first, followed by the monthly summaries of earlier ones.
• A page for each report, at /score/{domain}/{report}, shows one analysis in full: the rules for each crawler, sitemaps, headers, meta tags, findings and score. Once a report has been folded into its monthly summary, its link leads to the domain page.

How it honours RFC 9309

The bot follows RFC 9309, the Robots Exclusion Protocol, before it requests anything other than robots.txt itself:

• It matches its product token against your User-agent lines, ignoring case, and falls back to the * group only when no group names it (section 2.2.1).
• The longest matching Allow or Disallow rule decides. On a tie, Allow wins (section 2.2.2).
• A robots.txt that answers 4xx means there are no rules (section 2.3.1.3). One that answers 5xx is unreachable, and the bot treats every path as disallowed (section 2.3.1.4). It also treats 429 Too Many Requests that way.
• It follows up to 5 redirects to find robots.txt (section 2.3.1.2) and reads up to 500 KB of it (section 2.5).
• Your robots.txt is fetched again for every analysis, just before the page, so the bot never acts on old rules (section 2.4).

Headers and meta tags too

RFC 9309 is about crawling. A page can also ask not to be indexed, with noindex or none in an X-Robots-Tag header or a robots meta tag. We honour that for the reports about the page, and treat a page your robots.txt keeps our bot or every crawler (User-agent: *) off the same way:

• A URL whose page asks every crawler not to index it is left out of the ranking and out of our sitemap.xml, along with its older reports.
• Its report pages are sent with a noindex header and meta tag, so search engines do not list them either. A request made only to named crawlers is passed on to those crawlers in the same way.
• A domain page is kept out of search engines when none of the domain's URLs may be indexed.

We do not rank such a page or offer it to search engines, and we give them no reason to index it.

Blocking the bot

To keep the bot away from your pages, add this to your robots.txt:

User-agent: RobotsTxtParser
Disallow: /

The bot will still read robots.txt itself, because that is how it learns the rule, and it will report on the file. It will not request any page or sitemap your rules forbid. A sitemap on another host, including www against the bare domain, follows that host's own robots.txt, which the bot reads first.

Removing your data

Blocking the bot stops new requests, but it does not remove reports already published. To have the reports and monthly summaries of your domain removed, write to robots@leopoletto.dev from an address at that domain, or name a way we can confirm you run it.

Anyone can still analyse a public robots.txt on this site, which creates a new report. Say in your message if you want the domain kept off the ranking as well.