This report is outdated. It was fetched on , and a newer version from is available.
View the latest versionrobots.txt report
https://ourworldindata.org/
1 notice · 9 pass
0 crawlers named
The file declares no User-agent groups, so every crawler may fetch every URL.
Consequence. Nothing is restricted. That is a valid and common configuration, but it also means no crawl budget is being steered away from low-value pages.
If that is not intended. If some sections should not be crawled — faceted search, internal endpoints, duplicate parameter URLs — declare them with Disallow rules.
All 6 checked search engines are allowed at the site root.
All 4 checked AI answer engines are allowed at the site root.
All 3 checked user-triggered AI fetches are allowed at the site root.
All 5 checked social link previews are allowed at the site root.
All 6 checked AI training crawlers are allowed at the site root.
The page sends no robots meta tag and no X-Robots-Tag header.
Every declared sitemap is an absolute URL and is crawlable.
https://ourworldindata.org/sitemap.xmlL1Every line parses as a directive crawlers understand.
Comfortably within the 500 KB that crawlers will read.