smart_toy Technical SEO

Free Robots.txt Checker

Type in a domain and the checker requests the robots.txt file that is actually published at the root of that host, exactly as a crawler would. It reports the HTTP status, groups the rules by user-agent, lists Allow, Disallow and Sitemap lines, and warns about patterns that tend to cause trouble, such as a blanket block on the whole site. Because it reads the production file, it is the quickest way to confirm what search engines see right now after a launch, migration or CMS update. If you want to try out a draft that is not live yet, use the robots.txt tester instead.

build_circle Try the free tool

How it works

  1. Enter site root

    We fetch /robots.txt from your domain automatically.

  2. Scan rules

    See allow and disallow patterns per user-agent.

  3. Adjust carefully

    Unblock money paths before opening low-value folders.

What you get

  • Parser error reporting

    Lists line numbers and tokens that break standard parsers.

  • Critical path allow tests

    Probes whether money paths are reachable per major bots.

  • Asset path rules

    Checks JS, CSS, and image directories needed for rendering.

  • Unsupported directives flag

    Notes crawl-delay and nonstandard lines major bots may ignore.

Who it's for

  • DevOps and SEO teams sharing ownership of edge and origin configuration.
  • WordPress and Shopify owners verifying plugin-generated robots rules.
  • Enterprises balancing AI visibility with IP protection on select paths.
  • Agencies running crawl access reviews during technical onboarding.

Good to know

robots.txt is the first gate crawlers read; one typo in Disallow can block entire path trees.

Get a Free Audit

What checking a robots.txt file actually means

A check answers three questions. Does the file exist and return a normal response, or a 404 or server error? Is the syntax readable, with each rule sitting under a User-agent line? And do the rules say what you intended for the crawlers you care about?

A few details trip people up. Paths are case-sensitive, so /Blog/ and /blog/ are different rules. For Google, the most specific matching rule wins, and the * and $ wildcards follow Google's published robots.txt specification. Googlebot ignores Crawl-delay. Robots.txt also controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, use a noindex robots meta tag or HTTP header, and leave that page crawlable so the tag can be seen.

The Disallow: / mistake that hides whole sites

Staging and preview environments often carry a two-line file, User-agent: * followed by Disallow: /, to stop them being crawled. The trouble starts when that file is deployed to production along with the rest of the build. Every compliant crawler is then told to stay away from every URL, and new or updated pages stop being picked up.

The checker flags a root-level Disallow for any user-agent group, including one aimed only at Googlebot. Run it straight after each release, domain move or hosting change, and on every subdomain you publish, since each host serves its own robots.txt.

When to use the robots.txt tester instead

This page reads what is live. The robots.txt tester works on text you paste in, so you can edit rules before publishing them and test specific paths to see whether they would be allowed or blocked for a chosen user-agent. Use the tester when you are writing new rules, debugging a single URL, or reviewing a change a developer has proposed. Use this checker to confirm what production is serving after the change goes out. Neither tool can promise that a URL will be indexed or ranked; they only show what the rules permit.

Questions

Questions we get asked

Where does the checker look for the file?

At the root of the host you enter, for example example.com/robots.txt. A file in a subfolder is ignored by crawlers, and www and non-www versions or other subdomains each need their own file.

What happens if my site has no robots.txt?

If the file returns a 404, crawlers generally treat the site as fully crawlable. That is fine for many sites. A 5xx error is different, as Google may pause crawling until the file is reachable again.

Can robots.txt remove a page from Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed from links elsewhere. Use a noindex tag on a crawlable page, or a removal request in Search Console for urgent cases.

Does Google still have its own robots.txt tester?

The old standalone tester in Search Console has been retired. Search Console now offers a robots.txt report, and URL Inspection shows how Google fetched a specific known URL.

Should the Sitemap line be in robots.txt?

It is optional but helpful. A Sitemap line with the full URL tells crawlers where to find your XML sitemap. The checker lists any it finds so you can confirm they point to the right location.

Free audit

Robots.txt looks fine but pages still missing?

Request a free audit and we will check crawlability, indexing signals and technical issues across your whole site.

Get a Free Audit