Free Robots.txt Checker
Type in a domain and the checker requests the robots.txt file that is actually published at the root of that host, exactly as a crawler would. It reports the HTTP status, groups the rules by user-agent, lists Allow, Disallow and Sitemap lines, and warns about patterns that tend to cause trouble, such as a blanket block on the whole site. Because it reads the production file, it is the quickest way to confirm what search engines see right now after a launch, migration or CMS update. If you want to try out a draft that is not live yet, use the robots.txt tester instead.
How it works
-
Enter site root
We fetch /robots.txt from your domain automatically.
-
Scan rules
See allow and disallow patterns per user-agent.
-
Adjust carefully
Unblock money paths before opening low-value folders.
What you get
-
Parser error reporting
Lists line numbers and tokens that break standard parsers.
-
Critical path allow tests
Probes whether money paths are reachable per major bots.
-
Asset path rules
Checks JS, CSS, and image directories needed for rendering.
-
Unsupported directives flag
Notes crawl-delay and nonstandard lines major bots may ignore.
Who it's for
- DevOps and SEO teams sharing ownership of edge and origin configuration.
- WordPress and Shopify owners verifying plugin-generated robots rules.
- Enterprises balancing AI visibility with IP protection on select paths.
- Agencies running crawl access reviews during technical onboarding.
Good to know
robots.txt is the first gate crawlers read; one typo in Disallow can block entire path trees.
Get a Free AuditWhat checking a robots.txt file actually means
A check answers three questions. Does the file exist and return a normal response, or a 404 or server error? Is the syntax readable, with each rule sitting under a User-agent line? And do the rules say what you intended for the crawlers you care about?
A few details trip people up. Paths are case-sensitive, so /Blog/ and /blog/ are different rules. For Google, the most specific matching rule wins, and the * and $ wildcards follow Google's published robots.txt specification. Googlebot ignores Crawl-delay. Robots.txt also controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, use a noindex robots meta tag or HTTP header, and leave that page crawlable so the tag can be seen.
The Disallow: / mistake that hides whole sites
Staging and preview environments often carry a two-line file, User-agent: * followed by Disallow: /, to stop them being crawled. The trouble starts when that file is deployed to production along with the rest of the build. Every compliant crawler is then told to stay away from every URL, and new or updated pages stop being picked up.
The checker flags a root-level Disallow for any user-agent group, including one aimed only at Googlebot. Run it straight after each release, domain move or hosting change, and on every subdomain you publish, since each host serves its own robots.txt.
When to use the robots.txt tester instead
This page reads what is live. The robots.txt tester works on text you paste in, so you can edit rules before publishing them and test specific paths to see whether they would be allowed or blocked for a chosen user-agent. Use the tester when you are writing new rules, debugging a single URL, or reviewing a change a developer has proposed. Use this checker to confirm what production is serving after the change goes out. Neither tool can promise that a URL will be indexed or ranked; they only show what the rules permit.
Questions
Questions we get asked
Where does the checker look for the file?
What happens if my site has no robots.txt?
Can robots.txt remove a page from Google?
Does Google still have its own robots.txt tester?
Should the Sitemap line be in robots.txt?
Free audit
Robots.txt looks fine but pages still missing?
Request a free audit and we will check crawlability, indexing signals and technical issues across your whole site.
Get a Free Audit