rragie.ai

A meta robots tag or X-Robots-Tag header instructs crawlers not to index this URL.

Consequence. The page will be dropped from search results. That is the correct mechanism for removing a page — worth confirming it is intended for this URL.

If that is not intended. If the page should rank, remove the noindex directive. Keep the URL crawlable either way, since a crawler must fetch the page to see the change.

• X-Robots-Tag: noindexRobots.txt response

The file declares no User-agent groups, so every crawler may fetch every URL.

Consequence. Nothing is restricted. That is a valid and common configuration, but it also means no crawl budget is being steered away from low-value pages.

If that is not intended. If some sections should not be crawled — faceted search, internal endpoints, duplicate parameter URLs — declare them with Disallow rules.

All 6 checked search engines are allowed at the site root.

All 4 checked AI answer engines are allowed at the site root.

All 3 checked user-triggered AI fetches are allowed at the site root.

All 5 checked social link previews are allowed at the site root.

All 6 checked AI training crawlers are allowed at the site root.

Every declared sitemap is an absolute URL and is crawlable.

• https://gd629.github.io/ragie-sitemap/sitemap_4575713.xmlL1

Every line parses as a directive crawlers understand.

Comfortably within the 500 KB that crawlers will read.