Bytespider is ByteDance's crawler, associated with TikTok and ByteDance's language models. Collects pages to train models; blocking it does not affect whether AI can answer about you today.
Bytespider is ByteDance's crawler, associated with TikTok and ByteDance's language models. It is among the heaviest AI crawlers by request volume on many sites.
Training: collects pages to train models; blocking it does not affect whether AI can answer about you today.
User-agent string:
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)robots.txt token: Bytespider. The vendor states it honours robots.txt; site owners have reported otherwise.
ByteDance publishes no IP-range feed for Bytespider. Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
Blocking Bytespider in robots.txt is the documented route. If the requests continue, block at the CDN by user-agent — the vendor publishes no IP ranges to verify against.
Check your site: the AI Crawler Access Check reads your robots.txt for Bytespider and the other AI crawlers and fetches your homepage with each one's real user-agent — the CDN challenge that robots.txt cannot show.
Blocking Bytespider in robots.txt is the documented route. If the requests continue, block at the CDN by user-agent — the vendor publishes no IP ranges to verify against.
Add a group for it: User-agent: Bytespider on one line and Allow: / on the next. If a User-agent: * group has a Disallow, the named group takes precedence for Bytespider.
User-agent: Bytespider then Disallow: /. The vendor states it honours robots.txt; site owners have reported otherwise.
ByteDance publishes no IP-range feed for Bytespider. Treat the user-agent as a claim, and rate-limit or block by behaviour if it misbehaves.
The free crawler check reads your robots.txt per bot and fetches your homepage with each one's real user-agent.