mmoz.com

Crawl-delay was never part of the standard. Google has never supported it; Bing and Yandex do.

Consequence. Any rate limit intended for Googlebot is not being applied. If the goal was to reduce load from Google specifically, it is not working.

If that is not intended. Set the crawl rate for Google in Search Console instead. Keep the directive if the limit is meant for Bing or Yandex, which do honour it.

• Crawl-delay: 10L6
• Crawl-delay: 10L9

The same group declares Allow and Disallow for an identical path.

Consequence. Nothing is blocked that the author wanted blocked, and nothing breaks — but the file does not do what reading it suggests. Order does not decide this: equal-length patterns resolve to the least restrictive rule, so the Allow wins and the Disallow has no effect at all.

If that is not intended. Delete whichever rule is not wanted. If the intent was to disallow a directory but permit one page inside it, make the Allow more specific than the Disallow rather than identical to it.

• Allow: /researchtools/ose/ and Disallow: /researchtools/ose/L34In the group for *; the Allow on line 30 wins

A broader rule of the same type already covers these paths.

Consequence. These lines change nothing. They are harmless, but they add length to a file that has a size limit, and they make the real policy harder to read.

If that is not intended. Remove the redundant rules, or narrow the broader rule if it is catching more than intended.

• allow: /researchtools/ose/$L20Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/dotbot$L21Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/links$L22Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/just-discovered$L23Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/pages$L24Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/domains$L25Already covered by "allow: /researchtools/ose/" on line 30
• allow: /researchtools/ose/anchors$L26Already covered by "allow: /researchtools/ose/" on line 30

The same directive and path appear repeatedly within one group.

Consequence. Duplicates have no effect on crawling, but they inflate the file toward the 500 KB limit and usually signal that the file is generated without dedup.

If that is not intended. Remove the repeats.

• allow: /researchtools/ose/dotbot$L31Already declared on line 21 for *

All 6 checked search engines are allowed at the site root.

All 4 checked AI answer engines are allowed at the site root.

All 3 checked user-triggered AI fetches are allowed at the site root.

All 5 checked social link previews are allowed at the site root.

All 6 checked AI training crawlers are allowed at the site root.

1 meta tag and 1 X-Robots-Tag header, with no contradictions.

• <meta name="robots" content="all">Page HTML
• X-Robots-Tag: allPage response

Every declared sitemap is an absolute URL and is crawlable.

• https://moz.com/sitemaps-1-sitemap.xmlL2
• https://moz.com/community/q/sitemap.xmlL3

Every line parses as a directive crawlers understand.

Comfortably within the 500 KB that crawlers will read.