qqrfy.com

A meta robots tag or X-Robots-Tag header instructs crawlers not to index this URL.

Consequence. The page will be dropped from search results. That is the correct mechanism for removing a page — worth confirming it is intended for this URL.

If that is not intended. If the page should rank, remove the noindex directive. Keep the URL crawlable either way, since a crawler must fetch the page to see the change.

• X-Robots-Tag: noindex, nofollowRobots.txt response

The file contains no "Sitemap:" line.

Consequence. Crawlers have to discover every URL by following links. Pages that are deep, newly published, or poorly linked internally take longer to be found, and some may never be. A sitemap is the most direct way to hand a crawler the full list.

If that is not intended. Publish an XML sitemap and declare it with an absolute URL, e.g. "Sitemap: https://example.com/sitemap.xml". The line is global — it belongs outside any User-agent group and applies to every crawler.

All 6 checked search engines are allowed at the site root.

All 4 checked AI answer engines are allowed at the site root.

All 3 checked user-triggered AI fetches are allowed at the site root.

All 5 checked social link previews are allowed at the site root.

All 6 checked AI training crawlers are allowed at the site root.

Every line parses as a directive crawlers understand.

Comfortably within the 500 KB that crawlers will read.