robots.txt Generator
Build a valid robots.txt to control how search engines crawl your site. Add disallow rules, your sitemap, and an optional crawl-delay.
User-agent: * Disallow: /admin/ Disallow: /cgi-bin/
Save as robots.txt in your site root: https://yoursite.com/robots.txt
Check your live robots.txt & crawlability
Analyze any URL freeControl crawling without breaking your SEO
A robots.txt file tells search engine crawlers which parts of your site they may request. It is powerful — and easy to get catastrophically wrong. One misplaced Disallow: / can wipe your site out of the index.
This generator builds a valid file with sensible defaults, your sitemap, and per-path rules. Copy it to /robots.txt at your domain root, then confirm it in Search Console before relying on it.
Frequently asked questions
Where does robots.txt go?
It must live at the root of your domain: https://example.com/robots.txt. Search engines only look there — a robots.txt in a subfolder is ignored.
Does Disallow hide a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results (without a snippet) if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Should I add my sitemap here?
Yes. A Sitemap: line pointing to your sitemap.xml helps search engines discover all your URLs. You can list multiple sitemaps.
What is crawl-delay?
It asks crawlers to wait N seconds between requests to reduce server load. Google ignores it (control Googlebot in Search Console), but Bing and others honor it.
Can a bad robots.txt hurt my SEO?
Absolutely. A stray “Disallow: /” blocks your entire site from crawling. Always double-check the output and test it in Google Search Console’s robots.txt tester.
Related tools
Meta Tag Generator
Title, description, Open Graph & Twitter Card tags with live length checks.
Open Graph Image Generator
Design & download a 1200×630 social share image in your browser.
UTM Campaign URL Builder
Tag links for GA4, Meta, Google Ads & email with presets and one-click copy.
URL Slug Generator
Turn any title into a clean, hyphenated, SEO-friendly URL slug.
This tool solves one thing.
Scan a live URL against 100 on-page checks — with the code to fix each issue.
Analyze any URL free