GPTBot is OpenAI's crawler for collecting web content that may be used to train its models. Collects pages to train models; blocking it does not affect whether AI can answer about you today.
GPTBot is OpenAI's crawler for collecting web content that may be used to train its models. It is the one most sites meant when they added an AI block to robots.txt in 2023.
Training: collects pages to train models; blocking it does not affect whether AI can answer about you today.
User-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)robots.txt token: GPTBot. Documented: honours robots.txt.
OpenAI publishes its IP ranges as a feed at https://openai.com/gptbot.json. A request claiming to be GPTBot from an address outside those ranges is an impostor.
AI bot verifier → checks an IP and user-agent against the published ranges and reverse DNS, free.
Blocking GPTBot keeps your pages out of future training data. It does not stop ChatGPT from reading your site when a user asks about it — that is ChatGPT-User — and it does not affect ChatGPT search, which is OAI-SearchBot. Many sites block GPTBot and allow the other two.
https://platform.openai.com/docs/bots
Check your site: the AI Crawler Access Check reads your robots.txt for GPTBot and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking GPTBot keeps your pages out of future training data. It does not stop ChatGPT from reading your site when a user asks about it — that is ChatGPT-User — and it does not affect ChatGPT search, which is OAI-SearchBot. Many sites block GPTBot and allow the other two.
Add a group for it: User-agent: GPTBot on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for GPTBot.
User-agent: GPTBot then Disallow: /. OpenAI documents that GPTBot honours robots.txt.
OpenAI publishes its IP ranges as a feed at https://openai.com/gptbot.json. A request claiming to be GPTBot from an address outside those ranges is an impostor.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.