Can AI crawlers read your site?

Retrieval user-agents (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) fetch pages while answering a question — block them and your site can never be cited in an answer. Training user-agents (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for future models; blocking them is a legitimate policy choice, but it removes you from what the next model learns.

robots.txt is a polite request. A CDN, WAF or bot-management rule can still answer a crawler with a 403, a 503 or a “Just a moment…” challenge page while a browser sees the real site. We send your homepage one request per bot with that bot's genuine user-agent string and compare it with a normal browser request. One honest caveat: our request does not come from the vendor's IP ranges, so a CDN that allow-lists verified bots will challenge us and still let the real crawler in. That is why a challenge is reported as “check your bot rules”, in amber — while a block in robots.txt is your own policy and counts as a real block.

The same semantics the engines use: the most specific matching user-agent group wins, otherwise the * group; within a group the longest matching path rule wins, and Allow wins a tie. Wildcards * and end anchors $ are supported. We evaluate the homepage path /; a rule that only blocks a sub-folder does not count as a block here.

Rate limits, per-page rules beyond the homepage, JavaScript-only content and how the engines describe you once they are in. The free Reflexa check covers the rest: it asks ChatGPT, Claude and Perplexity your buyers' real questions and shows whether they mention, recommend and cite you.