robots.txt is the plain-text file at the root of a site that tells crawlers, by user-agent, which paths they may fetch.
It is the first door. A Disallow that covers a page stops a compliant crawler from ever reading it — and a User-agent: * rule applies to every AI crawler not named separately.
Open https://yourdomain/robots.txt and read the groups.
Checked by Reflexa: Does your robots.txt let AI crawlers in? — part of the 23 checks on every page check.
robots.txt is the plain-text file at the root of a site that tells crawlers, by user-agent, which paths they may fetch.
It is the first door. A Disallow that covers a page stops a compliant crawler from ever reading it — and a User-agent: * rule applies to every AI crawler not named separately.
Open https://yourdomain/robots.txt and read the groups. The free tool does it for you →
The free page check runs the 23 checks on your public pages — the terms on this page, observed on your pages.