Skip to content
GEOoptimised

Free tool · no signup · live fetch

Does your site block AI crawlers?

Enter a domain. We fetch its robots.txt and llms.txt live, the same way we fetched them for every firm in our sector indices, and check the same 12 AI crawler tokens against the same benchmark: whether the file holds any of them to stricter rules than it gives Googlebot. Nothing is stored.

Plain domain only, for example example.com. No scheme, path or port.

What this checks, and what it cannot tell you

This tool reads exactly two files, live, at the moment you press the button: /robots.txt and /llms.txt. It parses robots.txt with standard user-agent grouping (RFC 9309) and derives one judgement, AI-specific stance, by comparing each of the 12 AI tokens' effective rules against the same file's effective rules for Googlebot. A token is only marked as an AI-specific restriction when it is fully disallowed while Googlebot is not equally disallowed in the same file. That is the whole methodology, and it is the same code (src/lib/robots-parse.mjs) that produced our four sector indices, so a domain checked here and a domain in an index can never disagree.

  • robots.txt is a request, not enforcement. A crawler operator may honour it or ignore it. Nothing here measures whether a crawler actually complied, only what the file asked for.
  • Naming a token is not restricting it. Most sites that name an AI crawler give it Allow: /, explicit permission, not a limit. Read the per-token status, not just whether a row exists.
  • Per-token precision matters. GPTBot (training) and OAI-SearchBot (ChatGPT search) are different decisions in the same file; blocking one says nothing about the other. The same split applies to ClaudeBot vs Claude-Web/anthropic-ai, and PerplexityBot vs Perplexity-User.
  • A block does not erase existing knowledge. Models retain what they were trained on before a rule existed. This tool cannot tell you whether a domain is "in" any model, only what its files currently request.
  • Bot protection can look like a block but is not one. A WAF challenge page (Cloudflare, Vercel, Azure) served instead of the real file is reported as "unreadable, stance unknown", never parsed as if it were a policy.
  • This is a single snapshot. Files change. Refetch any time; nothing is cached beyond a short edge cache to keep the tool responsive under repeat load.

For the full write-up of every rule in the table, including what GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Bingbot, CCBot and Applebot-Extended each actually control, see "what a block actually prevents" on the law firms index.

The same methodology, run at scale

This tool checks one domain at a time. We have already run the identical fetch-and-parse across four UK sectors: law firms, fintech companies, private healthcare providers and accountancy firms, with every raw response retained.