cchatgpt.com

The URL is declared in robots.txt but does not return a successful response.

Consequence. Crawlers find nothing at the address they were pointed to, so no URLs are submitted. This is worse than declaring no sitemap, because the declaration suggests discovery is covered when it is not.

If that is not intended. Publish the sitemap at the declared URL, or correct the URL to where it lives.

• https://chatgpt.com/sitemap.xmlL229Responded HTTP 403
• https://chatgpt.com/marketing-sitemap.xmlL230Responded HTTP 403
• https://chatgpt.com/writing-tools-sitemap.xmlL231Responded HTTP 403
• https://chatgpt.com/acquisition-landing-pages-sitemap.xmlL232Responded HTTP 403

"Disallow: /" applies to "User-agent: *", but 13 groups name crawlers that are exempt from it and 164 Allow rules open specific paths.

Consequence. Any crawler this file does not name is refused every URL. A crawler that is named ignores the wildcard group entirely and follows only its own rules, so the default block does not apply to it.

If that is not intended. To let a new crawler in, give it its own group — adding an Allow to the "User-agent: *" group will not reach it, because a named group replaces the wildcard rather than extending it.

• Disallow: /L220Applies to every crawler not named elsewhere
• Allow: /unauth-mweb/assets/L48Opens this path to crawlers not named elsewhere
• Allow: /$L52Opens this path to crawlers not named elsewhere
• Allow: /?*L54Opens this path to crawlers not named elsewhere
• Allow: /api/share/og/L55Opens this path to crawlers not named elsewhere
• Allow: /g/L56Opens this path to crawlers not named elsewhere

A broader rule of the same type already covers these paths.

Consequence. These lines change nothing. They are harmless, but they add length to a file that has a size limit, and they make the real policy harder to read.

If that is not intended. Remove the redundant rules, or narrow the broader rule if it is catching more than intended.

• allow: /cdn/assets/faviconL72Already covered by "allow: /cdn/assets/" on line 73
• allow: /writing-tools-sitemap.xmlL83Already covered by "allow: /writing" on line 135
• allow: /students/2026L119Already covered by "allow: /students" on line 118
• allow: /100chats-projectL124Already covered by "allow: /100chats" on line 85
• disallow: /auth/logoutL223Already covered by "disallow: /" on line 220
• disallow: /auth/login?*L224Already covered by "disallow: /" on line 220
• disallow: /backend-anon/sentinel/*L225Already covered by "disallow: /" on line 220
• disallow: /backend-anon/conversation$L226Already covered by "disallow: /" on line 220
• disallow: /account-link/*L227Already covered by "disallow: /" on line 220

Blocked: PerplexityBot, Google-Extended.

• PerplexityBotPerplexityL40

  Perplexity answers

  "Disallow: /" blocks PerplexityBot.

• Google-ExtendedGoogleL10

  Google AI Overviews and Gemini grounding

  "Disallow: /" blocks Google-Extended.

If that is the intention. If your intention is to keep the site out of AI-generated answers, the current configuration achieves that for these crawlers — the rest of the group is still allowed.

Consequence. The site cannot be cited as a source in AI-generated answers. Unlike training crawlers these send referral traffic, so blocking them forfeits those visits.

If that is not intended. Remove the matching Disallow rule, or allow these agents on the sections you want cited.

Blocked: Bytespider, CCBot.

• BytespiderByteDanceL34

  ByteDance model training

  "Disallow: /" blocks Bytespider.

• CCBotCommon CrawlL4

  Common Crawl, widely redistributed as training data

  "Disallow: /" blocks CCBot.

If that is the intention. If your intention is to prevent this content being used as model training data, the current configuration achieves that for these crawlers — the rest of the group is still allowed.

Consequence. These crawlers cannot retrieve content from the site, so it will not be collected as training data. They send no traffic back, and blocking them does not affect search rankings.

If that is not intended. Remove the site-wide block, or replace it with narrower path restrictions.

All 6 checked search engines are allowed at the site root.

All 3 checked user-triggered AI fetches are allowed at the site root.

All 5 checked social link previews are allowed at the site root.

The page answered HTTP 403 with a bot challenge instead of its content, so its meta robots tags and X-Robots-Tag headers were not read.

Consequence. Nothing is known about whether this page asks to be kept out of search results. Its absence of directives here proves nothing either way.

If that is not intended. Analyse the page again once it answers normally to our bot.

Every line parses as a directive crawlers understand.

Comfortably within the 500 KB that crawlers will read.