robots.txt allows AI crawlers. Crawlers that respect robots.txt stop at a Disallow.
The check reads your robots.txt and evaluates it for each AI user-agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended — the way the crawlers themselves do: the most specific agent group wins, the longest matching rule wins, allow wins a tie.
Crawlers that respect robots.txt stop at a Disallow. A page they cannot fetch cannot be read, quoted or recommended — the rule is the first door, and every other check is behind it. Many sites added a blanket block in 2023 to keep training crawlers out and never noticed it also keeps the retrieval crawlers out, the ones that fetch a page to answer a live question.
Open /robots.txt on your domain and look for a group naming an AI user-agent, or a User-agent: * group with a Disallow that covers your pages. The crawler check tool does this per bot and also fetches your homepage with each bot's real user-agent string.
AI Crawler Access Check → — free, no signup.
User-agent: OAI-SearchBot then Allow: / — or remove the Disallow that catches it.What the agent does: the agent appends the AI-crawler groups to robots.txt from inside the site — one click, reversible, nothing else in the file is touched.
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
The ones from OpenAI, Anthropic, Google and Perplexity document that their crawlers honour it; user-initiated fetchers such as ChatGPT-User and Claude-User fetch a page a person asked for and are governed by their own rules. Bots that ignore robots.txt are a different problem, handled at the CDN.
Yes. Allow OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot; disallow GPTBot, ClaudeBot, Google-Extended and CCBot. The check evaluates each group separately and tells you which side each crawler lands on.
A User-agent: * group applies to every crawler not named elsewhere. A Disallow in it covers the AI crawlers too.
The page check reads your public pages from outside and runs all 22 — free, no signup, no AI engine involved.