ClaudeBot is Anthropic's crawler for collecting web content that may be used to train Claude. Collects pages to train models; blocking it does not affect whether AI can answer about you today.
ClaudeBot is Anthropic's crawler for collecting web content that may be used to train Claude. Older tokens — anthropic-ai and Claude-Web — are legacy names you may still see in robots.txt files.
Training: collects pages to train models; blocking it does not affect whether AI can answer about you today.
User-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)robots.txt token: ClaudeBot. Documented: honours robots.txt.
Anthropic documents its outbound range — 160.79.104.0/21 and 2607:6bc0::/48 — at https://platform.claude.com/docs/en/api/ip-addresses. A request from outside it is not ClaudeBot.
AI bot verifier → checks an IP and user-agent against the published ranges and reverse DNS, free.
Blocking ClaudeBot keeps your pages out of Anthropic's training crawl. It does not stop Claude from fetching a page a user asks about — that is Claude-User.
Check your site: the AI Crawler Access Check reads your robots.txt for ClaudeBot and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking ClaudeBot keeps your pages out of Anthropic's training crawl. It does not stop Claude from fetching a page a user asks about — that is Claude-User.
Add a group for it: User-agent: ClaudeBot on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for ClaudeBot.
User-agent: ClaudeBot then Disallow: /. Anthropic documents that ClaudeBot honours robots.txt.
Anthropic documents its outbound range — 160.79.104.0/21 and 2607:6bc0::/48 — at https://platform.claude.com/docs/en/api/ip-addresses. A request from outside it is not ClaudeBot.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.