noindex, nofollow, and Disallow sound similar but do completely different things. Confusing them is one of the most common — and most damaging — technical SEO mistakes.
Quick Comparison
| Directive | Where | What it does |
|---|---|---|
Disallow |
robots.txt | Asks crawlers not to crawl a URL |
noindex |
meta tag / header | Tells Google not to index (show) a page |
nofollow |
meta tag / link attr | Tells Google not to follow / pass authority through links |
Disallow (robots.txt)
Controls crawling, not indexing. It stops well-behaved bots from fetching a URL:
User-agent: *
Disallow: /admin/
The trap: a Disallowed page can still be indexed (without content) if other sites link to it — because Google never crawls it, it never sees a noindex. To truly hide a page, use noindex and let Google crawl it.
noindex (meta tag or header)
Keeps a page out of search results. The page can still be crawled:
<meta name="robots" content="noindex">
Or via HTTP header:
X-Robots-Tag: noindex
Use it for thank-you pages, internal search results, staging pages, and thin filter/pagination URLs.
nofollow (link attribute)
Tells Google not to pass ranking authority through a specific link:
<a href="https://example.com" rel="nofollow">Sponsored link</a>
Use it for paid links, user-generated content, and untrusted destinations. It does not stop indexing of your own page.
The Golden Rules
- To hide a page from search: use
noindex— and do not also Disallow it (Google must crawl the page to see the noindex). - To save crawl budget on worthless URLs: use
Disallow. - To manage link equity: use
nofollow/sponsored/ugc.
Common Mistakes
- Disallowing a page you also
noindexed → Google never sees the noindex → page stays indexed. - Using robots.txt to hide sensitive data (it is public and just lists what to avoid).
nofollow-ing your own internal links and starving pages of authority.
Audit Your Directives
Run your URL through SEO Snapshot — it parses your robots.txt and meta robots tags and flags conflicting or risky indexing directives.