robots.txt report

a11y-collective.com

https://a11y-collective.com

C

62/100

Complexity 54/100

How scoring works
User Agents
1
Rules
9
Sitemaps
1
Warning
2
File Size
485 B
Cached result · Expires in 21 hours 5 minutes

2 things to look at

2 warning · 8 pass

1 crawlers named · 1 unrecognised

Who this file lets in

  • Every other crawler all 1 allowed

Path Access Checker

Detailed Analysis

Warning

The requested page is set to noindex

A meta robots tag or X-Robots-Tag header instructs crawlers not to index this URL.

Consequence. The page will be dropped from search results. That is the correct mechanism for removing a page — worth confirming it is intended for this URL.

If that is not intended. If the page should rank, remove the noindex directive. Keep the URL crawlable either way, since a crawler must fetch the page to see the change.

  • <meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1">Page HTML
  • X-Robots-Tag: noindex, followRobots.txt response
Warning

1 rule names a location nobody would have guessed

Rules name a sensitive location made specific by a qualifier or by where it sits, or a spreadsheet by its full path.

Consequence. A bare "/backup" is in every scanner wordlist; "/backup-2023/" is in none. Listing it advertises the location to the readers least likely to honour the rule.

If that is not intended. Put these locations behind authentication or remove them; once they are protected, the line can stay or go. Responses behind authentication are not indexed, so they need no "noindex". Content that is public by design and only has to stay out of search wants a "noindex" without the Disallow: a crawler only sees a noindex on a page it may fetch.

  • Disallow: /wp-content/uploads/wc-logs/L2"logs" below the first level
Pass

Search engines can reach the site

All 6 checked search engines are allowed at the site root.

Pass

AI answer engines can reach the site

All 4 checked AI answer engines are allowed at the site root.

Pass

User-triggered AI fetches can reach the site

All 3 checked user-triggered AI fetches are allowed at the site root.

Pass

Social link previews can reach the site

All 5 checked social link previews are allowed at the site root.

Pass

AI training crawlers can reach the site

All 6 checked AI training crawlers are allowed at the site root.

Pass

1 sitemap declared

Every declared sitemap is an absolute URL and is crawlable.

  • https://www.a11y-collective.com/sitemap_index.xmlL16
Pass

No syntax problems

Every line parses as a directive crawlers understand.

Pass

File size is 485 bytes

Comfortably within the 500 KB that crawlers will read.