Validator

Is your robots.txt any good?

Check any site's robots.txt for mistakes, get a score out of 100 with every finding explained, and see how it ranks against other sites.

Enter any URL — the robots.txt will be automatically fetched from the root domain.

The ranking

Scores by grade

Sites scanned
1,177
Average score
72
Grade B or above
43%
  • Managed configurationCloudflare or another identifiable provider
  • Automated policyGenerated or maintained automatically
  • Explicit restrictionsCrawler-specific access policies
  • Policy exceptionsNamed agents override wildcard restrictions
  • Redundant rulesDirectives with no additional effect
  • WarningsConfiguration defects or compatibility issues
  • Unknown directivesNonstandard or unsupported directives
  • Full restrictionAll paths disallowed for selected crawlers

How scoring works

Every robots.txt is reduced to a single number from 0 to 100 so results can be compared. Two things go into it: quality, how clean the file is, and complexity, how much it actually does. Without complexity the emptiest files would score highest, since a two-line file has little to get wrong.

score = quality × (60% + 40% × complexity ÷ 100)

1. Quality

Starts at 100 and loses points for every audit finding that asks for a change, down to a minimum of 0.

  • Critical−30
  • Warning−12
  • Notice−4
  • InfoNot checkedPass0

Info findings describe a policy, such as blocking every AI crawler. That is a choice, not a defect, so it costs nothing. Neither does a check the bot could not make, such as on a page that showed it a bot challenge.

A site with no robots.txt gets a report, but no grade: there is no file to score, so it is left out of the ranking.

2. Complexity

A measure from 0 to 100 of how much the file does. Each signal earns its points in proportion to how close it comes to its target, so repeating yourself past the target gains nothing.

  • Rules30 pts

    Full points at 20 Allow or Disallow rules

  • Named crawlers30 pts

    Full points at 5 distinct user-agents other than *

  • Sitemaps25 pts

    Full points at 1 Sitemap line

  • Rule variety15 pts

    Full points with both Allow and Disallow in use

A file with no complexity keeps 60% of its quality; a file at full complexity keeps all of it.

3. Grade

The final score is rounded and mapped to a letter. Only a file that is both correct and substantial reaches the top grade.

  • A90–100
  • B75–89
  • C60–74
  • D40–59
  • F0–39

Example

A file with one warning has a quality of 88. At a complexity of 50, it keeps 80% of that, for a score of 70 C.

Who makes the ranking

  • Each site is listed once, under its domain, by its best-scoring page whose report has not expired, or its best-scoring page when every report has. Only the newest report of each page counts. Every report of every page is shown on the domain's page.
  • A page that asks not to be indexed, or that the site's robots.txt keeps our bot or every crawler off, is not ranked.
  • A site the bot can no longer fetch keeps its place for 30 days from its report, shown with that report's date.
  • Only scores worked out by the current rules (version 2) are compared; older ones return once they are rescored.
  • A URL whose page asks every crawler not to index it is left out.