robots.txt report

betalist.com

https://betalist.com

D

50/100

Complexity 0/100

How scoring works
User Agents
0
Rules
0
Sitemaps
0
Warning
2
File Size
99 B
Cached result · Expires in 21 hours 34 minutes

2 things to look at

1 warning · 1 notice · 8 pass

0 crawlers named

Path Access Checker

Detailed Analysis

Warning

No sitemap is declared

The file contains no "Sitemap:" line.

Consequence. Crawlers have to discover every URL by following links. Pages that are deep, newly published, or poorly linked internally take longer to be found, and some may never be. A sitemap is the most direct way to hand a crawler the full list.

If that is not intended. Publish an XML sitemap and declare it with an absolute URL, e.g. "Sitemap: https://example.com/sitemap.xml". The line is global — it belongs outside any User-agent group and applies to every crawler.

Notice

No crawl rules are declared

The file declares no User-agent groups, so every crawler may fetch every URL.

Consequence. Nothing is restricted. That is a valid and common configuration, but it also means no crawl budget is being steered away from low-value pages.

If that is not intended. If some sections should not be crawled — faceted search, internal endpoints, duplicate parameter URLs — declare them with Disallow rules.

Pass

Search engines can reach the site

All 6 checked search engines are allowed at the site root.

Pass

AI answer engines can reach the site

All 4 checked AI answer engines are allowed at the site root.

Pass

User-triggered AI fetches can reach the site

All 3 checked user-triggered AI fetches are allowed at the site root.

Pass

Social link previews can reach the site

All 5 checked social link previews are allowed at the site root.

Pass

AI training crawlers can reach the site

All 6 checked AI training crawlers are allowed at the site root.

Pass

No indexing directives on the requested page

The page sends no robots meta tag and no X-Robots-Tag header.

Pass

No syntax problems

Every line parses as a directive crawlers understand.

Pass

File size is 99 bytes

Comfortably within the 500 KB that crawlers will read.