rule Technical SEO

Robots.txt Tester: Simulate Rules Before You Deploy

This tester works on robots.txt text you paste in, not the file already live on your domain. Drop in a draft or a short rules snippet, list the paths you care about, and choose a user-agent such as * or Googlebot. The simulation reports Allowed or Disallowed for each path, shows the line that matched, and explains which user-agent group was applied. Use it before a release, during a migration, or when a developer hands you a staging file that still says Disallow: /. Once the file is published, confirm the production copy with our robots.txt checker, because a simulation of a draft cannot see caching, fetch errors, or server responses.

build_circle Try the free tool

How it works

  1. Paste robots.txt

    Use your live file or a draft you are about to deploy.

  2. Choose a crawler

    Test as Googlebot, Bingbot, an AI bot, or the wildcard.

  3. Test URLs

    Enter paths or full URLs and read the verdict and the matching line.

What you get

  • User-agent group selection

    Finds which group applies to the crawler you pick.

  • Longest-match precedence

    Compares every Allow and Disallow pattern in the chosen group against the path.

  • Allow versus Disallow ties

    When an Allow and a Disallow pattern are exactly the same length, the tool lets Allow win.

  • Wildcard and end anchor

    Supports * to match any run of characters and $ to anchor the end of a URL.

Who it's for

  • Technical SEOs
  • Web developers and DevOps engineers
  • Site owners preparing a launch or migration
  • Agencies auditing client sites

Good to know

Test your robots.txt before deploying, because a single misplaced slash can block an entire site from search.

Get a Free Audit

How the tester decides between Allow and Disallow

Google documents a longest-match rule for paths: inside the group that applies, the rule with the most specific (longest) matching pattern decides the outcome. If an Allow and a Disallow rule match with patterns of equal length, Google treats the URL as allowed. This is different from older parsers that stop at the first matching line, which is why a file that looks fine in one tool can behave differently for Googlebot. For example, with Disallow: /shop/ and Allow: /shop/sale/, the path /shop/sale/boots is allowed because the Allow pattern is longer. The tester shows the winning line next to each path so you can see why, not just what.

Wildcards, end anchors, and case-sensitive paths

Two special characters matter when you test robots.txt rules. The asterisk matches any sequence of characters, so Disallow: /*?sort= blocks any URL containing that parameter. The dollar sign anchors a pattern to the end of the URL path, so Disallow: /*.pdf$ blocks files ending in .pdf without touching /pdf-guides/. Path matching is case-sensitive: a rule for /Admin/ does not cover /admin/. When a simulated result surprises you, check capitalization, trailing slashes, and whether a wildcard is catching more than you intended.

Which user-agent group gets applied

A crawler follows only one group in the file. Google picks the group with the most specific user-agent token that matches it and falls back to the * group only when nothing more specific exists. If you write a Googlebot group, Googlebot ignores the * group entirely, so rules you expected to inherit will not apply. Specialized tokens such as Googlebot-Image use their own group when one is present. Also note that Googlebot ignores Crawl-delay. Some other crawlers still read it, but it is not a way to slow Google, and the tester does not treat it as one.

Crawl blocking is not deindexing

Robots.txt tells cooperating crawlers what they may fetch. It does not remove pages from search. A disallowed URL can still be indexed from links and other signals, often shown without a snippet. To keep a page out of the index, leave it crawlable and use noindex, put it behind authentication, or use a removal process. One more caution: avoid blocking CSS and JavaScript by default, because Google needs those files to render pages properly.

Questions

Questions we get asked

How is this tester different from the robots.txt checker?

The checker fetches the robots.txt file that is live on a domain. This tester simulates rules you paste in, so you can try a draft, compare versions, or check a snippet before anything ships. Use both: test the draft here, then confirm the deployed file with the checker.

Does a passing simulation guarantee how Googlebot will crawl my site?

No. The simulation applies Google's published matching logic to the text you provide. Real crawling can differ because Google caches robots.txt, server errors change how the file is treated, and product-specific token groups may apply. Treat the result as a pre-deploy check, not a promise.

Can I still use the old Search Console robots.txt tester?

Google retired that tool. The URL Inspection tool shows what Googlebot did with a live URL, but it does not let you paste a draft file and simulate paths, which is the gap this tester covers.

What should I test before pushing a staging robots.txt to production?

Look for Disallow: / under the * or Googlebot group, since that blocks the whole site for matching bots. Then test your homepage, key category and service pages, and the asset paths for CSS, JavaScript, and images that you want crawlers to reach.

Why does my Allow rule not override a Disallow rule?

Under Google's logic the longer matching pattern wins. If your Disallow pattern is longer or more specific than the Allow pattern for that path, the path stays blocked. Lengthen the Allow pattern or narrow the Disallow and run the simulation again.

Free audit

Want a second pair of eyes on your crawl rules?

Send us your draft robots.txt and the URLs that matter most. We will review group logic, blocked resources, and indexing controls before launch.

Get a Free Audit