Claude is Anthropic's assistant, and when it recommends a business it is usually doing so with a live, cited web search behind it — not a fixed ranking. Getting named comes down to being reachable, readable and checkable: Anthropic runs three separate bots for training, live answers and search quality, and — unusually among the major engines — all three say they honor robots.txt. Blocking the wrong one, or none at all, decides whether Claude can even see you before content quality ever gets a vote. Here's what actually controls it.
Claude's web search runs as a live tool call, not a lookup against frozen training data — it searches, reads results and attaches an inline citation to the exact passage it drew a claim from. That's the same basic shape as getting recommended by ChatGPT: no paid placement, no fixed leaderboard, just a model deciding whether your page is worth reading and worth citing. The difference is in the mechanics — Anthropic's documentation shows every citation carries the source URL, title and the exact cited text, and it happens automatically whenever Claude searches.
Proof: Anthropic — Claude web search tool documentation (citations).
Bottom line: if Claude can't fetch your page in the moment someone asks, it can't cite you — no amount of past authority substitutes for being reachable right now.
Anthropic runs ClaudeBot, Claude-User and Claude-SearchBot as separate crawlers with separate robots.txt user-agents, and each does a different job. ClaudeBot collects public content for model training. Claude-User fetches a specific page the moment a person asks Claude a question about it. Claude-SearchBot crawls to improve the quality of Claude's search results. Anthropic states that all three respect robots.txt — including Claude-User, the real-time fetcher — which puts Claude in a different position than engines whose user-triggered fetchers (like ChatGPT-User or Perplexity-User) aren't held to the same rule.
Proof: Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them.
Good news: a correct robots.txt is a complete, reliable answer for Claude specifically — there's no separate "real-time exception" to plan around the way there is elsewhere.
Disallowing ClaudeBot only opts a site out of future model training — it does nothing to Claude-User or Claude-SearchBot, which need their own rules. A site that blocks ClaudeBot but leaves Claude-User open can still be fetched and cited the moment a customer asks about it live; a site that blocks Claude-User instead goes dark for exactly those live answers, even if its content shaped Claude's training long ago. Reflexa's crawler study found this kind of partial, accidental blocking is common: 7 of 40 major sites (18%) block ClaudeBot in robots.txt, often as one line in a broader AI-bot block nobody separately reviewed per bot.
Check it now: open yoursite.com/robots.txt and look for ClaudeBot, Claude-User and Claude-SearchBot individually — a rule naming one says nothing about the other two.
Content that only appears after JavaScript runs is invisible to a crawler that fetches raw HTML. This isn't unique to Claude — it's the same failure mode across every AI crawler Reflexa tests, and it's often the real reason a page never gets fetched usefully even when robots.txt is wide open. Pricing, specs and the plain statement of what a business does and for whom need to exist in the HTML a crawler actually receives.
Check it now: view your page's source (Cmd+U / Ctrl+U) and search for your core claim as plain text — if it's only in the rendered DOM, Claude never reads it.
Claude cites sources, and a source is only useful if the model is confident about who it's citing. Inconsistent naming across a site, LinkedIn and directories makes it harder for any engine to attach a citation to the right entity with confidence — the same problem covered in full on the entity clarity guide.
Do this: use one legal name and one canonical URL consistently across your site, About page and every external profile you control.
A citation-based engine quotes the passage that answers a question most directly, not the page that's biggest or best-known. The practice of front-loading the answer — covered on the answer-first writing guide — matters as much for Claude's live search as it does for any other engine that reads and cites in real time.
Rule of thumb: your page's first 100 words should state the answer to the buying question, not build up to it.
The direct test is asking Claude the exact question a buyer would ask, in Claude's own web search, and reading the citations it returns. Reflexa runs a wider, repeatable version of that same check across ChatGPT, Claude and Perplexity — the same buying questions, tracked over time — described in the tools overview.
The free check reads your crawler access and shows real citation data — 3 minutes, evidence included.