robots.txt Generator

Build a valid robots.txt to control how search engines crawl your site. Add disallow rules, your sitemap, and an optional crawl-delay.

Crawling

Disallow paths (one per line)

Sitemap URL

Crawl-delay (seconds, optional)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/

Save as robots.txt in your site root: https://yoursite.com/robots.txt

Check your live robots.txt & crawlability