Googlebot is the crawler behind Google Search — and therefore behind AI Overviews and AI Mode, which answer from the Search index. Builds the index an AI search product answers from; blocking it removes you from those answers.
Googlebot is the crawler behind Google Search — and therefore behind AI Overviews and AI Mode, which answer from the Search index. It renders JavaScript, with a delay.
Search index: builds the index an AI search product answers from; blocking it removes you from those answers.
User-agent string:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)robots.txt token: Googlebot. Documented: honours robots.txt.
Google publishes its IP ranges as a feed at https://developers.google.com/static/search/apis/ipranges/googlebot.json. A request claiming to be Googlebot from an address outside those ranges is an impostor.
AI bot verifier → checks an IP and user-agent against the published ranges and reverse DNS, free.
Blocking Googlebot removes you from Google, AI Overviews and AI Mode alike. Nobody does this on purpose; it happens by accident with a User-agent: * Disallow.
https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
Check your site: the AI Crawler Access Check reads your robots.txt for Googlebot and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking Googlebot removes you from Google, AI Overviews and AI Mode alike. Nobody does this on purpose; it happens by accident with a User-agent: * Disallow.
Add a group for it: User-agent: Googlebot on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for Googlebot.
User-agent: Googlebot then Disallow: /. Google documents that Googlebot honours robots.txt.
Google publishes its IP ranges as a feed at https://developers.google.com/static/search/apis/ipranges/googlebot.json. A request claiming to be Googlebot from an address outside those ranges is an impostor.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.