Robots.txt Tester: Simulate Rules Before You Deploy
This tester works on robots.txt text you paste in, not the file already live on your domain. Drop in a draft or a short rules snippet, list the paths you care about, and choose a user-agent such as * or Googlebot. The simulation reports Allowed or Disallowed for each path, shows the line that matched, and explains which user-agent group was applied. Use it before a release, during a migration, or when a developer hands you a staging file that still says Disallow: /. Once the file is published, confirm the production copy with our robots.txt checker, because a simulation of a draft cannot see caching, fetch errors, or server responses.
How it works
-
Paste robots.txt
Use your live file or a draft you are about to deploy.
-
Choose a crawler
Test as Googlebot, Bingbot, an AI bot, or the wildcard.
-
Test URLs
Enter paths or full URLs and read the verdict and the matching line.
What you get
-
User-agent group selection
Finds which group applies to the crawler you pick.
-
Longest-match precedence
Compares every Allow and Disallow pattern in the chosen group against the path.
-
Allow versus Disallow ties
When an Allow and a Disallow pattern are exactly the same length, the tool lets Allow win.
-
Wildcard and end anchor
Supports * to match any run of characters and $ to anchor the end of a URL.
Who it's for
- Technical SEOs
- Web developers and DevOps engineers
- Site owners preparing a launch or migration
- Agencies auditing client sites
Good to know
Test your robots.txt before deploying, because a single misplaced slash can block an entire site from search.
Get a Free AuditHow the tester decides between Allow and Disallow
Google documents a longest-match rule for paths: inside the group that applies, the rule with the most specific (longest) matching pattern decides the outcome. If an Allow and a Disallow rule match with patterns of equal length, Google treats the URL as allowed. This is different from older parsers that stop at the first matching line, which is why a file that looks fine in one tool can behave differently for Googlebot. For example, with Disallow: /shop/ and Allow: /shop/sale/, the path /shop/sale/boots is allowed because the Allow pattern is longer. The tester shows the winning line next to each path so you can see why, not just what.
Wildcards, end anchors, and case-sensitive paths
Two special characters matter when you test robots.txt rules. The asterisk matches any sequence of characters, so Disallow: /*?sort= blocks any URL containing that parameter. The dollar sign anchors a pattern to the end of the URL path, so Disallow: /*.pdf$ blocks files ending in .pdf without touching /pdf-guides/. Path matching is case-sensitive: a rule for /Admin/ does not cover /admin/. When a simulated result surprises you, check capitalization, trailing slashes, and whether a wildcard is catching more than you intended.
Which user-agent group gets applied
A crawler follows only one group in the file. Google picks the group with the most specific user-agent token that matches it and falls back to the * group only when nothing more specific exists. If you write a Googlebot group, Googlebot ignores the * group entirely, so rules you expected to inherit will not apply. Specialized tokens such as Googlebot-Image use their own group when one is present. Also note that Googlebot ignores Crawl-delay. Some other crawlers still read it, but it is not a way to slow Google, and the tester does not treat it as one.
Crawl blocking is not deindexing
Robots.txt tells cooperating crawlers what they may fetch. It does not remove pages from search. A disallowed URL can still be indexed from links and other signals, often shown without a snippet. To keep a page out of the index, leave it crawlable and use noindex, put it behind authentication, or use a removal process. One more caution: avoid blocking CSS and JavaScript by default, because Google needs those files to render pages properly.
Questions
Questions we get asked
How is this tester different from the robots.txt checker?
Does a passing simulation guarantee how Googlebot will crawl my site?
Can I still use the old Search Console robots.txt tester?
What should I test before pushing a staging robots.txt to production?
Why does my Allow rule not override a Disallow rule?
Free audit
Want a second pair of eyes on your crawl rules?
Send us your draft robots.txt and the URLs that matter most. We will review group logic, blocked resources, and indexing controls before launch.
Get a Free Audit