An AI crawler is the bot an AI company sends to fetch web pages — to train a model (GPTBot, ClaudeBot), to build a search index (OAI-SearchBot, PerplexityBot) or to read a page a user just asked about (ChatGPT-User, Claude-User).
Every AI answer about your site starts with one of these fetching a page. Each has its own user-agent and its own rules in robots.txt; blocking one does not block the others.
Check /robots.txt for each token and your server logs for the user-agents.
Checked by Reflexa: Does your robots.txt let AI crawlers in? — part of the 23 checks on every page check.
An AI crawler is the bot an AI company sends to fetch web pages — to train a model (GPTBot, ClaudeBot), to build a search index (OAI-SearchBot, PerplexityBot) or to read a page a user just asked about (ChatGPT-User, Claude-User).
Every AI answer about your site starts with one of these fetching a page. Each has its own user-agent and its own rules in robots.txt; blocking one does not block the others.
Check /robots.txt for each token and your server logs for the user-agents.
The free page check runs the 23 checks on your public pages — the terms on this page, observed on your pages.