Page blocked by robots.txt

What this check looks for

Checks whether the page's URL is disallowed for search crawlers by a rule in the site's robots.txt file.

Why it matters

A Disallow rule that matches the page keeps crawlers from fetching it, so it usually won't be indexed and can't rank, and any page it links to may go undiscovered. This is a common accidental cause of pages vanishing from search after a staging config ships to production. Note that robots.txt blocks crawling, not indexing, so use a noindex meta tag rather than a Disallow when you want to keep a URL out of results.

How to fix it

# robots.txt: allow crawling of the page (remove or narrow the Disallow)
User-agent: *
Allow: /

# Keep only the paths you truly want blocked, e.g.
Disallow: /admin/
Disallow: /cart/

Sitemap: https://seosnapshot.dev/sitemap.xml

Go deeper

• robots.txt guide →
• robots.txt generator →