All pages return HTTP 200. A crawler that is rate-limited or served an error does not retry patiently.
The check requests each priority page and records the final status after redirects: 200 is a page, 404 is a missing one, 429 is rate limiting, 5xx is a server error. Anything but 200 is a page a crawler did not get.
A crawler that is rate-limited or served an error does not retry patiently. It records the failure and moves on; the page is absent from the index until the next visit, if there is one. Rate limiting (429) is the common one on sites that defend against scrapers.
Fetch each page with curl -I and read the status line. Check the pages in your sitemap, not only the homepage.
What the agent does: the agent lists the failing pages and their status; the cause is in your server or host.
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
No — the check follows redirects and judges the final response. A chain of several redirects is slow but not failing.
The check reads the homepage and the handful of pages it fetches each run. The agent, once on the site, sees the status of every request.
The page check reads your public pages from outside and runs all 22 — free, no signup, no AI engine involved.