For a business that wants AI to recommend it, blocking any major AI crawler is almost always a net loss. Training bots and live-answer bots are not the same thing, and only one narrow split, GPTBot versus OAI-SearchBot, lets you opt out of training without losing today's answers. In Reflexa's study, 18% of 40 well-known sites blocked at least one AI crawler outright (Reflexa AI Crawler Study, August 2026).
This is the access layer the what is GEO pillar puts first in the chain: access, then identity, then content, then trust. Blocking a crawler is an access decision, and it is worth making on purpose rather than by CDN default, because it can cancel every other GEO effort before it starts.
A training bot shapes what a future version of the model knows about you, while a live-answer bot fetches your page right now to answer one specific question. OpenAI runs both under separate rules: GPTBot crawls to train future models, and OAI-SearchBot powers live web search inside ChatGPT, with ChatGPT-User as a third bot for fetches triggered directly by a user's action. Blocking GPTBot does not block OAI-SearchBot, and the reverse is also true, since the two rules are independent in robots.txt.
| Bot | Sent by | Purpose | Effect of blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Trains future models | Opts you out of future training; today's ChatGPT answers unaffected |
| OAI-SearchBot / ChatGPT-User | OpenAI | Live web search & user-triggered fetch | Removes you from live ChatGPT answers now |
| ClaudeBot / Claude-User / Claude-SearchBot | Anthropic | Training, user fetches & search-quality crawling | Removes you from Claude's training and live answers together |
| Google-Extended | Trains Gemini & feeds AI Overviews | Opts out of Gemini/AI Overviews only; Google Search ranking unaffected | |
| PerplexityBot / Perplexity-User | Perplexity | Indexing vs on-demand fetch | Blocking PerplexityBot likely loses indexing entirely |
Proof: OpenAI — GPTBot & OAI-SearchBot documentation; full crawler roles in the Reflexa AI crawler guide.
Do this: if you want to opt out of training without losing today's answers, disallow GPTBot only and keep OAI-SearchBot and ChatGPT-User allowed.
Yes, because Anthropic collapses training and live use into one rule, so there is no equivalent split for Claude. ClaudeBot handles training, Claude-User handles fetches Claude makes on a user's behalf, and Claude-SearchBot handles search-quality crawling, but Anthropic's own documentation states that all three honor robots.txt the same way. Blocking ClaudeBot does not just opt you out of training, it removes you from Claude's live answers at the same time.
Proof: Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them.
Bottom line: a company that wants Claude's training exclusion but not its live-answer exclusion has no way to have both, unlike with OpenAI's two bots.
Neither works exactly like OpenAI's split, and each has its own wrinkle. Google-Extended is a deliberate opt-out that only controls Gemini training and AI Overviews inclusion, fully decoupled from standard Googlebot, so blocking it does not touch your regular Google Search ranking at all. PerplexityBot is described in Perplexity's own documentation as the indexing crawler that respects robots.txt, while Perplexity-User is a real-time agent Perplexity frames as acting on a user's explicit request rather than as ordinary crawling, the one point where the major engines genuinely disagree about what counts as a "crawl."
Proof: Google — overview of Google crawlers, including Google-Extended; Perplexity — crawler documentation.
Rule of thumb: block PerplexityBot and you likely lose indexing outright; see the get cited by Perplexity guide for the Perplexity-User nuance in full.
Blocking is concentrated in one sector, media, while most businesses leave the door open. Reflexa's robots.txt study of 40 well-known companies found media/news sites blocking at 57% (4 of 7), against retail at 14% (1 of 7), SaaS at 7% (1 of 15), and finance and travel at 0%. The New York Times, BBC, Bloomberg and Reuters each block all six major AI crawlers outright, a deliberate stance to protect paid journalism (Reflexa AI Crawler Study, August 2026).
| Sector | Blocks ≥1 AI crawler |
|---|---|
| Media / news | 57% (4 of 7) |
| Retail | 14% (1 of 7) |
| SaaS | 7% (1 of 15) |
| Finance | 0% (0 of 5) |
| Travel | 0% (0 of 4) |
Proof: Reflexa AI Crawler Access Study (robots.txt of 40 well-known companies checked August 22, 2026).
The nuance: for a publisher protecting paid content, blocking can be a considered strategy. For a business that wants to be recommended, the same rule is usually an own goal set by a CDN default nobody chose, exactly the pattern covered in why ChatGPT recommends my competitor and not me.
Mainly control over whether your specific content trains a model, and, for a publisher, leverage over paid content licensing. That is a real and legitimate reason for some businesses. What it does not protect is your visibility in AI answers today: blocking a live-answer bot removes you from being named the moment someone asks, and Access is the first of five layers an engine has to clear before it can recommend you, the one that cancels every other layer when it fails.
Do this: decide per bot, not per engine. Training exclusion and live-answer exclusion are two different choices, and only OpenAI currently lets you make them separately.
None of these checks require buying anything, and each takes a few minutes:
Bottom line: the decision to block is fine when it is made on purpose. It usually is not.
Yes, that split actually works: OpenAI treats GPTBot and OAI-SearchBot as independent rules in robots.txt, so you can keep your content out of model training while OAI-SearchBot and ChatGPT-User still fetch and cite you in live ChatGPT answers.
No. Anthropic states that ClaudeBot, Claude-User and Claude-SearchBot all honor the same robots.txt rule, so blocking ClaudeBot removes you from Claude's training and its live answers together, unlike the OpenAI split.
It can be: Reflexa's study found 57% of media sites (4 of 7) block at least one AI crawler outright, a deliberate stance to protect paid journalism. That same block is usually a self-inflicted loss for a business that wants AI to recommend it.
No. Google-Extended only controls whether your content can train Gemini and feed AI Overviews; standard Google Search indexing runs on the separate Googlebot rules and is unaffected.
Run the free AI Crawler Access Check at /tools/ai-crawler-check. It reads your robots.txt rule for every major AI bot and fetches your homepage with each bot's real user-agent, since CDN-level blocks often will not show up in the file at all.
The free check tests every major AI crawler against your site, robots and CDN-level, in 3 minutes, with the evidence.