CCBot is Common Crawl's crawler. Collects pages to train models; blocking it does not affect whether AI can answer about you today.
CCBot is Common Crawl's crawler. Common Crawl is a non-profit that publishes a free copy of the web, and that copy has been the training corpus for many models, including early GPTs.
Training: collects pages to train models; blocking it does not affect whether AI can answer about you today.
User-agent string:
CCBot/2.0 (https://commoncrawl.org/faq/)robots.txt token: CCBot. Documented: honours robots.txt.
Common Crawl publishes no IP-range feed for CCBot. Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
Blocking CCBot keeps your pages out of the public crawl that many models train on. It has no effect on any live AI answer.
Check your site: the AI Crawler Access Check reads your robots.txt for CCBot and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking CCBot keeps your pages out of the public crawl that many models train on. It has no effect on any live AI answer.
Add a group for it: User-agent: CCBot on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for CCBot.
User-agent: CCBot then Disallow: /. Common Crawl documents that CCBot honours robots.txt.
Common Crawl publishes no IP-range feed for CCBot. Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.