Robots.txt Tester
Paste your robots.txt and a URL to see instantly whether Googlebot (or any user-agent) is allowed to crawl it — with the exact rule that decides.
A single stray Disallow can hide a whole section of your site from search engines. This tester parses your robots.txt exactly the way a crawler does — honouring wildcards, $ anchors, and the most-specific-rule-wins logic — and tells you whether a given URL is allowed for a given user-agent. Everything runs in your browser; nothing is sent to a server.
Three steps
Paste your robots.txt
Copy the contents of yoursite.com/robots.txt into the box.
Enter a URL to test
Type the path you want to check, e.g. /admin/dashboard, and optionally a user-agent.
Read the verdict
See Allowed or Blocked, the rule that matched, and any sitemaps declared.
Common questions
Does this fetch my live robots.txt?
No — it analyses the text you paste, entirely in your browser. Nothing is sent to a server, so it works on staging and private sites too.
Does it support wildcards and $?
Yes. It honours * wildcards, the $ end-of-path anchor, and the standard 'most specific rule wins, Allow breaks ties' precedence that Googlebot uses.
Which user-agents can I test?
Any — enter Googlebot, Bingbot, or leave it as * to test the default group.
Keep going
Audit your whole site, not just one check
CrawlX runs 65+ technical checks in the cloud and ships the fix as a pull request. Free for 500 URLs.