In our scans, the single most common critical failure is also the cheapest to fix: a robots.txt that blocks AI crawlers wholesale. GPTBot, ClaudeBot, PerplexityBot, Google-Extended — blocked, usually via a template someone copied during the 2023 "block the scrapers" wave.
Know which crawler does what
This is where the nuance lives. Some bots fetch pages to train future models; others fetch them to answer a live user question, right now, with citations. Blocking the second kind means a user asks "best invoicing tool for freelancers" and the engine literally cannot read your page to include you.
The fix
Open yourdomain.com/robots.txt and read it. If you find blanket Disallow: / rules for AI user agents and you want to be discovered by AI, remove them — or scope them to the paths you actually need private. Check your CDN too: some providers ship an "AI bot blocking" toggle that overrides whatever your file says.
The analyzer's crawler-access category checks each major bot individually and tells you exactly which ones can't get in. It's the first category in the report for a reason: nothing else matters if the crawler never sees the page.