Claude answers from fixed training data unless web search is switched on, so a company absent from the open web it learned from is simply unknown to it. Getting recognized means being in that web: allowed for ClaudeBot and Claude-User, documented on Wikidata or Wikipedia, and named on pages Claude's training already crossed.
This sits inside the wider challenge covered in how to get recommended by ChatGPT: being accessible, readable and verifiable to every engine, not just one. Claude's version of that challenge has its own shape, because its default answer doesn't start with a live search the way some engines' does.
Because Claude's baseline answer comes from a frozen training snapshot, and a company that never appeared anywhere in the public web it learned from, and that Claude hasn't fetched live since, simply has nothing stored to recall. Anthropic's own documentation describes Claude's web search as a tool the model calls explicitly during a conversation, not something that runs behind every reply by default. Unless that tool fires, or unless your company already existed somewhere in the training data, Claude has no record of you at all, not a low ranking, an absence.
Proof: Anthropic — Claude web search tool documentation.
Bottom line: the fix means existing in both places: on the public web a live search tool can find today, and on pages widely linked enough that a future training run is likely to pick you up too.
Three separate bots, each with its own job, and each governed by its own robots.txt rule. ClaudeBot crawls the public web to build training data. Claude-User fetches a specific page the moment a person asks Claude a question about it, live, during a conversation. Claude-SearchBot crawls separately to improve the quality of Claude's search results. Anthropic states that all three honor robots.txt, including the real-time fetcher, which is unusual among the major engines.
| Bot | What it does | When it runs |
|---|---|---|
| ClaudeBot | Crawls public pages to build training data | Continuously, independent of any one conversation |
| Claude-User | Fetches a specific page on a user's behalf | Live, the moment someone asks Claude about it |
| Claude-SearchBot | Crawls to improve search-result quality | Continuously, feeding Claude's search index |
Proof: Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them.
Do this: the full playbook for all three bots, plus the rest of what controls Claude's citations, lives on the get recommended by Claude guide.
No. Disallowing ClaudeBot only opts a site out of future model training, it does nothing to Claude-User or Claude-SearchBot, which each need their own rule. A site can block ClaudeBot and still get fetched and cited live the moment a customer asks Claude about it, through Claude-User, which a lot of broad "block AI bots" rules never separate out. Reflexa's own crawler study found this kind of partial, accidental blocking is common: 7 of 40 major sites checked (18%) block ClaudeBot in robots.txt, usually as one line in a wider block nobody reviewed bot by bot.
Proof: Reflexa — study: 18% of major sites block AI crawlers at the front door.
Check it now: open yoursite.com/robots.txt and look for ClaudeBot, Claude-User and Claude-SearchBot individually, a rule naming one says nothing about the other two.
It helps because Wikidata and Wikipedia are widely reused, sourced answers to "who is this" that systems trained on public web and encyclopedic text lean on more than an average page. Neither is a lever you can pull on demand; Wikipedia requires independent notability and Wikidata expects a sourced, verifiable structure before an item is trusted. The mechanism, and what's realistically within reach for a company that isn't already a household name, is covered in full on why Wikipedia and Wikidata quietly decide what AI knows about you.
Do this: a Wikidata item and a plain About page that states the facts outright don't guarantee a mention, but they give any model, including a future Claude, a structured, sourced answer instead of nothing at all.
Related, but the starting point differs. ChatGPT and Perplexity lean more heavily on live search by default when they answer a buying question, so a page they can fetch right now carries more weight. Claude's baseline answer reaches for training data first and only searches live when the tool is triggered, so a company that's invisible to Claude can still be named by the other two, and the reverse is just as possible. The full comparison, engine by engine, lives on ChatGPT vs Claude vs Perplexity: how each picks who to recommend.
Bottom line: treat each engine as its own check, not one pass that covers all three.
None of this requires buying anything, and each step takes a few minutes:
Bottom line: most of this is access and identity, not content you haven't written yet.
Claude answers from a frozen training snapshot unless its web search tool is switched on, so a company that never appears in the public web it learned from, and that Claude never fetches live, has nothing for the model to recall.
It lets Claude fetch and cite your pages the moment someone asks, but only if ClaudeBot, Claude-User and Claude-SearchBot can all reach your site. A robots.txt block on any one of them keeps that door shut regardless of web search being switched on.
Not strictly, but it helps. Wikidata and Wikipedia are widely reused as a structured, sourced answer to who a company is, and a company with neither relies entirely on its own site and scattered mentions to establish identity.
Run the free AI Crawler Access Check, which reads your robots.txt and CDN rules for ClaudeBot, Claude-User and Claude-SearchBot by name, since a rule that blocks one says nothing about the other two.
Related but not identical. ChatGPT and Perplexity lean more on live search by default, while Claude's baseline answer comes from training data first, so an entity invisible to Claude can still be visible to the other two.
The free check reads your crawler access and the rest of the 35+ checks — evidence included, nothing installed.
Let the engines in, with one click: the robots.txt switch for AI crawlers · install Reflexa Agent · på svenska