A training crawler collects pages to train a model; a retrieval crawler fetches pages to answer a question now. Blocking the first keeps you out of future models; blocking the second keeps you out of today's answers.
Most 2023-era robots.txt blocks aimed at training and hit retrieval too. The distinction is the single most useful thing to know about AI crawlers.
OpenAI: GPTBot trains, OAI-SearchBot indexes, ChatGPT-User fetches live. Anthropic: ClaudeBot trains, Claude-User fetches live.
Checked by Reflexa: Does your robots.txt let AI crawlers in? — part of the 23 checks on every page check.
A training crawler collects pages to train a model; a retrieval crawler fetches pages to answer a question now. Blocking the first keeps you out of future models; blocking the second keeps you out of today's answers.
Most 2023-era robots.txt blocks aimed at training and hit retrieval too. The distinction is the single most useful thing to know about AI crawlers.
OpenAI: GPTBot trains, OAI-SearchBot indexes, ChatGPT-User fetches live. Anthropic: ClaudeBot trains, Claude-User fetches live.
The free page check runs the 23 checks on your public pages — the terms on this page, observed on your pages.