Amazonbot crawls the web for Amazon's services, including Alexa's answers and Amazon's models.. Collects pages to train models; blocking it does not affect whether AI can answer about you today.
Amazonbot crawls the web for Amazon's services, including Alexa's answers and Amazon's models.
Training: collects pages to train models; blocking it does not affect whether AI can answer about you today.
User-agent string:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)robots.txt token: Amazonbot. Documented: honours robots.txt.
Amazon publishes no IP-range feed for Amazonbot; Amazon documents reverse-DNS verification (hosts ending in crawl.amazonbot.amazon). Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
Blocking Amazonbot keeps your pages out of Amazon's crawl. Verify a claimed Amazonbot by reverse DNS, as Amazon documents.
https://developer.amazon.com/amazonbot
Check your site: the AI Crawler Access Check reads your robots.txt for Amazonbot and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking Amazonbot keeps your pages out of Amazon's crawl. Verify a claimed Amazonbot by reverse DNS, as Amazon documents.
Add a group for it: User-agent: Amazonbot on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for Amazonbot.
User-agent: Amazonbot then Disallow: /. Amazon documents that Amazonbot honours robots.txt.
Amazon publishes no IP-range feed for Amazonbot; Amazon documents reverse-DNS verification (hosts ending in crawl.amazonbot.amazon). Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.