We checked the robots.txt of 40 well-known companies to see who lets the major AI crawlers in — and who locks them out before content, structured data or authority ever matter. Nearly one in five blocks at least one. Among news publishers, it's the majority. Here's the full method and the raw numbers.
7 of 40 major sites (18%) block at least one major AI crawler in robots.txt. When a site blocks, it usually blocks broadly — the same companies tend to disallow several engines at once, not just one. The block rate is close across engines:
| AI crawler | Sent by | Sites blocking it |
|---|---|---|
| GPTBot | OpenAI | 7 of 40 · 18% |
| ClaudeBot | Anthropic | 7 of 40 · 18% |
| ChatGPT-User | OpenAI | 6 of 40 · 15% |
| Google-Extended | 6 of 40 · 15% | |
| OAI-SearchBot | OpenAI | 5 of 40 · 12% |
| PerplexityBot | Perplexity | 5 of 40 · 12% |
Bottom line: for these sites, no amount of great content or authority matters. The engine never gets in to read it.
The blocking is concentrated in one sector — media — while the businesses that actually want AI recommendations mostly leave the door open. Split by sector:
| Sector | Blocks ≥1 AI crawler |
|---|---|
| Media / news | 4 of 7 · 57% |
| Retail | 1 of 7 · 14% |
| SaaS | 1 of 15 · 7% |
| Finance | 0 of 5 · 0% |
| Travel | 0 of 4 · 0% |
The New York Times, BBC, Bloomberg and Reuters each block all six major AI crawlers outright — a deliberate stance to protect paid journalism. Amazon does the same. A handful of others block selectively: Figma disallows GPTBot, ClaudeBot and Google-Extended; WebMD blocks the OpenAI and Anthropic bots. Every one of these is public — open any of these domains' /robots.txt and you'll see the same rules we did.
The nuance: for a publisher, blocking can be a strategy. For a business that wants to be recommended, the same rule is an own goal — and it's often set by a CDN default nobody chose.
Not blocking is step zero, not a finish line — and “open” is easier to lose than you'd think. Two traps catch companies that assume they're fine: a security or CDN layer (Cloudflare, Akamai) can challenge AI bots by default without touching your robots.txt, and a broad Disallow: / under User-agent: * silently applies to every AI crawler too. Access is only the first of five layers an engine has to clear before it can recommend you — but it's the one that cancels all the others when it fails.
Do this: confirm access first, then earn the recommendation. In that order.
Shortcut: the free Reflexa check tests all of this for you — every AI crawler, robots and CDN-level, with the exact rule that blocks each one.
This is a first read, and we'll say exactly how we got it. On August 22, 2026 we fetched the robots.txt of 44 well-known companies across six sectors and successfully parsed 40 (four returned 403 or a redirect with no readable file and were excluded). We counted a crawler as “blocked” when its own user-agent group — or the catch-all User-agent: * — carried a Disallow: / without an overriding Allow: /. This is a small, curated sample of recognizable brands, not a random probability sample, and robots.txt is only one access layer — CDN- and WAF-level blocking is invisible to this method. It's a snapshot, not a census; we'd rather show the method than dress it up.
The free check tests every AI crawler against your site — robots and CDN-level — in 3 minutes, with the evidence.