Consequence. Crawlers have to discover every URL by following links. Pages that are deep, newly published, or poorly linked internally take longer to be found, and some may never be. A sitemap is the most direct way to hand a crawler the full list.
If that is not intended. Publish an XML sitemap and declare it with an absolute URL, e.g. "Sitemap: https://example.com/sitemap.xml". The line is global — it belongs outside any User-agent group and applies to every crawler.
Warning
1 line will be ignored by crawlers
Lines that do not parse as a known directive: unknown_directive.
Consequence. A crawler skips what it cannot parse, without reporting anything. A misspelled "Dissalow" leaves a path fully crawlable while the file appears to restrict it.
If that is not intended. Correct the spelling of each field name, and make sure every line follows "Field: value". Rules must also appear after a User-agent line to belong to a group.
Unknown directive '"user-agent' — crawlers will ignore this lineL1unknown_directive
Notice
No crawl rules are declared
The file declares no User-agent groups, so every crawler may fetch every URL.
Consequence. Nothing is restricted. That is a valid and common configuration, but it also means no crawl budget is being steered away from low-value pages.
If that is not intended. If some sections should not be crawled — faceted search, internal endpoints, duplicate parameter URLs — declare them with Disallow rules.
Pass
Search engines can reach the site
All 6 checked search engines are allowed at the site root.
Pass
AI answer engines can reach the site
All 4 checked AI answer engines are allowed at the site root.
Pass
User-triggered AI fetches can reach the site
All 3 checked user-triggered AI fetches are allowed at the site root.
Pass
Social link previews can reach the site
All 5 checked social link previews are allowed at the site root.
Pass
AI training crawlers can reach the site
All 6 checked AI training crawlers are allowed at the site root.
Pass
Page indexing directives are consistent
3 meta tags and 0 X-Robots-Tag headers, with no contradictions.
<meta name="robots" content="index, follow">Page HTML
<meta name="googlebot" content="index, follow">Page HTML
<meta name="bingbot" content="index, follow">Page HTML
Pass
File size is 28 bytes
Comfortably within the 500 KB that crawlers will read.
