AI Crawler Access Checker
Paste your robots.txt to see which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more — may fetch your site, and which you've locked out.
Most sites block or allow AI crawlers by accident: a Disallow written for one bot, a wildcard group that catches everything, or a copied-in list from a blog post that was out of date on day one. This tool parses your robots.txt exactly the way a crawler does — including * wildcards, $ anchors and the most-specific-rule-wins precedence — and tests fifteen well-known AI user-agents from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, ByteDance, Meta and Common Crawl against / and any path you give it. It also tells you what each token actually controls, because Google-Extended and Applebot-Extended are training opt-outs, not search crawlers, and blocking them changes nothing about your rankings. Everything runs in your browser; nothing is sent to a server.
Three steps
Paste your robots.txt
Copy the contents of yoursite.com/robots.txt into the box, or start from the sample to see the logic.
Add a path (optional)
Enter a path such as /blog/my-post to test a specific section in addition to the site root.
Read the table
Each AI crawler is marked Allowed or Blocked, with a one-line note on what that bot is used for — training, live fetches, or search-style indexing.
Common questions
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended is a separate token that only controls whether Google may use your content for Gemini training and grounding. Google Search crawling, indexing and AI Overviews follow Googlebot, which this token does not touch.
What happens if I block GPTBot, ClaudeBot or PerplexityBot?
Those assistants can no longer read your pages, so they cannot cite or link to you in their answers. If you want to appear in ChatGPT, Claude or Perplexity results, keep them allowed. If you specifically want to opt out of training, note that OpenAI and Anthropic publish separate tokens for training (GPTBot, ClaudeBot) and for user-triggered fetches (ChatGPT-User, Claude-User).
Which AI crawlers does the tool check?
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Bingbot, CCBot, Applebot-Extended, Amazonbot, Bytespider and meta-externalagent. Well-behaved bots honour these tokens; a few crawlers ignore robots.txt entirely, and no robots.txt rule can stop those.
Does this fetch my live robots.txt?
No — it analyses the text you paste, entirely in your browser, so it works on staging and private sites too.
Keep going
Audit your whole site, not just one check
CrawlX runs 150+ technical checks in the cloud and ships the fix as a pull request. Free for 500 URLs.