sitemap.xml is present and valid. Crawlers discover pages by following links and by reading the sitemap.
The check looks for the sitemap where crawlers look — /sitemap.xml and the Sitemap: line in robots.txt — fetches it, and checks that it parses as a sitemap or a sitemap index with absolute URLs.
Crawlers discover pages by following links and by reading the sitemap. Without a valid one, new or deep pages are found late or never; with one, every page you want found is listed in one machine-readable place, with the date it last changed.
Open /sitemap.xml. It should be XML with
Sitemap: https://yourdomain/sitemap.xml.What the agent does: the agent serves a sitemap built from the pages the site publishes — one click, and it stays current.
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
Yes, and it is cheap: 20 pages is 20 lines. The sitemap is also how engines learn a page changed.
The retrieval engines crawl the same way search engines do, and AI search (Perplexity, ChatGPT search, Google AI Mode) draws on search indexes that read sitemaps.
The page check reads your public pages from outside and runs all 22 — free, no signup, no AI engine involved.