Every AI answer about your site starts with one of these fetching a page. They are not one bot: training crawlers collect pages for future models, search crawlers build the index an AI search product answers from, and user-initiated fetchers read a page because someone just asked about it. Each has its own name, its own robots.txt token and its own rules.
Collects pages to train models; blocking it does not affect whether AI can answer about you today.
GPTBot is OpenAI's crawler for collecting web content that may be used to train its models.
AnthropicClaudeBot is Anthropic's crawler for collecting web content that may be used to train Claude.
GoogleGoogle-Extended is not a bot you will see in your logs.
Common CrawlCCBot is Common Crawl's crawler.
ByteDanceBytespider is ByteDance's crawler, associated with TikTok and ByteDance's language models.
AmazonAmazonbot crawls the web for Amazon's services, including Alexa's answers and Amazon's models..
AppleApplebot-Extended is Apple's control token, like Google-Extended: a Disallow keeps your content out of Apple Intelligence training without affecting Siri and Spotlight search, which use Applebot..
MetaMeta-ExternalAgent crawls the web for Meta's AI models and products.
Builds the index an AI search product answers from; blocking it removes you from those answers.
OAI-SearchBot builds the index behind ChatGPT search — the results and links ChatGPT shows when it searches the web for an answer.
PerplexityPerplexityBot builds Perplexity's search index — the pages it can retrieve and cite in its answers.
GoogleGooglebot is the crawler behind Google Search — and therefore behind AI Overviews and AI Mode, which answer from the Search index.
Fetches a page because a person asked the assistant about it right now; blocking it makes you unreadable at the moment it matters.
ChatGPT-User fetches a page when a person in ChatGPT (or a GPT, or an action) asks about it — "summarise this page", "what does this company do".
AnthropicClaude-User fetches a page when a person using Claude asks about it or Claude searches the web to answer them.
PerplexityPerplexity-User fetches a page because a person asked Perplexity about it or clicked through.
DuckDuckGoDuckAssistBot fetches pages to generate DuckDuckGo's AI-assisted answers (DuckAssist) and summaries in response to a search..
Free, from outside, in minutes — with the exact finding behind every line.