Six user-agents decide whether ChatGPT, Claude and Perplexity can see your site at all: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot and their live-fetch counterparts. Each one is sent by a different company, for a different purpose — some build the model's long-term knowledge, others fetch a page in real time to answer one question. This guide names each crawler, who sends it, what it feeds, and the exact robots.txt rule that lets it in.
Every major AI engine actually runs two different kinds of crawler: one that trains or indexes over time, and one that fetches a page live while answering a single question. Blocking the wrong one produces confusing results — a site can be fully absent from a model's trained knowledge while still getting fetched and quoted correctly in a live search, or the reverse. Treating "AI crawlers" as one bot is the most common mistake in a robots.txt file.
Bottom line: check both categories, per engine, not just the one everyone's heard of.
OpenAI sends two separate bots: GPTBot crawls pages to help train and improve its models, and OAI-SearchBot powers live web search inside ChatGPT. A site can block GPTBot and still show up in a ChatGPT search answer through OAI-SearchBot, or the other way around — the two rules are independent in robots.txt. ChatGPT-User is a third, separate user-agent OpenAI uses when a user's action inside ChatGPT triggers a direct page fetch.
Proof: OpenAI documents both bots and their user-agent strings — OpenAI — GPTBot & OAI-SearchBot.
Do this: the full five-step playbook for winning ChatGPT's shortlist — once GPTBot and OAI-SearchBot can reach you — lives on the get recommended by ChatGPT guide.
Anthropic runs three bots — ClaudeBot for training, Claude-User for fetches Claude makes on a user's behalf, and Claude-SearchBot for search-quality crawling — and states that all three honor robots.txt. That's the one place Claude differs from the other two engines: there's no separate "search bot exception" to plan around, because Anthropic describes the same rule as covering live fetches too.
Proof: Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them.
Good news: one correct robots.txt entry per bot name is, by Anthropic's own description, a complete answer for Claude — see the get recommended by Claude guide for the rest of what controls Claude's citations.
Google-Extended is a separate opt-out that controls whether your content can train Gemini and feed AI Overviews — it does not touch standard Google Search indexing, which is governed by the regular Googlebot rules. A site can block Google-Extended and keep ranking normally in Google Search; the two crawlers are deliberately decoupled so a business can opt out of one without losing the other.
Proof: Google — overview of Google crawlers, including Google-Extended.
Check it now: if you want Gemini and AI Overviews to be able to use your content, confirm Google-Extended carries no Disallow: / — it's easy to miss because it's not the same rule as blocking Google Search itself.
PerplexityBot is the indexing crawler that Perplexity's own documentation says respects robots.txt, while Perplexity-User is a real-time agent Perplexity has described as acting on a user's explicit request rather than as a standard bot. That framing is the one genuine point of disagreement among the major engines about whether a live, user-triggered fetch should be treated as crawling at all.
Proof: Perplexity — crawler documentation (PerplexityBot & Perplexity-User).
Rule of thumb: block PerplexityBot and you likely lose indexing; the get cited by Perplexity guide covers the Perplexity-User nuance in full.
The safest robots.txt for AI visibility gives every major AI user-agent its own explicit Allow: / line, rather than relying on the catch-all. A broad Disallow: / under User-agent: * silently applies to every AI crawler that doesn't have its own more specific rule — the most common way sites block AI by accident.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: /
Check it now: robots.txt is only step one — a CDN or WAF (Cloudflare, Akamai, Vercel) can still challenge or block these same bots at the network level without touching the file. The Reflexa audit tests both layers together and names the exact rule that's blocking each engine.
The free check tests every AI crawler against your site — robots and CDN-level — in 3 minutes, with the evidence.