robots.txt Tester
Paste your robots.txt, enter a URL, and see instantly whether it's allowed or blocked for a given crawler — using the same longest-match rules Google applies. Runs entirely in your browser.
An Allow rule explicitly permits this path.
Matched rule → Allow: /admin/public/
Follows Google's matching rules: the most specific (longest) path wins, * is a wildcard, a trailing $ anchors the end, and an Allow beats a Disallow of equal length. Remember: robots.txt controls crawling, not indexing — to keep a page out of results use a noindex tag instead.
Catch the accidental block before Google does
The single most damaging robots.txt mistake is a broad Disallow: / that ships from staging to production and quietly blocks the whole site. The second is a rule that's more aggressive than intended — a Disallow: /*? that catches URLs you actually want crawled. Testing specific paths is how you catch both.
Once the file is right, generate a clean version with the robots.txt generator, and remember the golden rule: to remove a page from search results, don't block it here — let it be crawled and add a noindex tag, which you can build with the robots meta generator.
Frequently asked questions
How does robots.txt matching work?
Crawlers pick the group whose User-agent best matches them (a specific name beats the * wildcard), then apply the most specific rule in that group — the one with the longest matching path. A * is a wildcard, a trailing $ anchors the end of the URL, and when an Allow and a Disallow of equal length both match, the Allow wins.
Does robots.txt stop a page from being indexed?
No — this is the most common misunderstanding. robots.txt controls crawling, not indexing. A blocked URL can still appear in Google (usually without a description) if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag, or use HTTP auth.
Why did Google remove its robots.txt tester?
Google retired the standalone robots.txt Tester tool from Search Console. You can still see the fetched file and test individual URLs via the URL Inspection tool, but a quick side-by-side tester like this one is often faster for checking rules as you edit them.
What path should I enter?
Either a full URL or just the path. The tester uses the path and query string (e.g. /admin/page?sort=asc), which is what crawlers match against — the domain itself is not part of robots.txt rule matching.
Related tools
Security Header Checker
Grade any URL A+ to F on HSTS, CSP & more, with copy-paste nginx fixes.
HTTP Header Checker
See status, redirect chain & every response header a crawler receives.
Keyword Density Checker
Word count, reading time & 1/2/3-word keyword density — catch stuffing.
Readability Checker
Flesch Reading Ease & grade level with live sentence and word stats.
This tool solves one thing.
Scan a live URL against 100 on-page checks — with the code to fix each issue.
Analyze any URL free