No CDN or bot filter blocks AI crawlers. robots.txt can say "welcome" while the bot management layer in front of the site says "prove you are human".
The check fetches your homepage with each AI crawler's real user-agent and looks at what comes back: a 403, a 503, a Cloudflare "just a moment" challenge page or a managed-challenge header means the crawler is being served a wall instead of the site.
robots.txt can say "welcome" while the bot management layer in front of the site says "prove you are human". AI crawlers cannot solve a challenge. To them the site is an error page, and retrieval moves on to a competitor that answers. This is invisible from a browser, which is why it goes unnoticed for months.
Fetch your homepage with curl -A "<the bot's user-agent>" and compare the status and body to a normal request. The crawler check tool runs this for eight AI user-agents and flags challenge pages by their markers.
AI Crawler Access Check → — free, no signup.
What the agent does: the bot filter sits in front of the site, outside anything on the site can change; the agent names the rule and the vendor, and you flip it in the CDN.
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
Yes, and it is the common case. robots.txt is read by the crawler; the challenge is served by the CDN before the crawler sees anything. The two are independent.
Check the source IP against the published ranges (openai.com/gptbot.json, searchbot.json, chatgpt-user.json). The bot verifier tool does this and adds reverse DNS with forward confirmation.
No. The allow list is for crawlers that verify by IP range. A scraper spoofing the user-agent fails the IP check and is handled as before.
The page check reads your public pages from outside and runs all 22 — free, no signup, no AI engine involved.