A user-agent is the string a client sends to identify itself — a browser version, or a crawler's name such as GPTBot/1.2 — and the token robots.txt groups are addressed to.
robots.txt rules match on it, CDNs filter on it, and impostors spoof it. Identity by user-agent is a claim; verification is by IP range.
Your server log's last quoted field is the user-agent.
Checked by Reflexa: Is your CDN or bot filter challenging AI crawlers? — part of the 23 checks on every page check.
A user-agent is the string a client sends to identify itself — a browser version, or a crawler's name such as GPTBot/1.2 — and the token robots.txt groups are addressed to.
robots.txt rules match on it, CDNs filter on it, and impostors spoof it. Identity by user-agent is a claim; verification is by IP range.
Your server log's last quoted field is the user-agent. The free tool does it for you →
The free page check runs the 23 checks on your public pages — the terms on this page, observed on your pages.