Robots.txt

Robots.txt controls crawling, not indexing — a disallowed URL can still show up in results as a bare link if other pages point to it, so it's the wrong tool for hiding a page (use noindex for that). Its real jobs are conserving crawl budget on large sites and keeping bots out of admin, cart, or infinite-parameter areas. One misplaced "Disallow: /" can deindex an entire site, so test changes in Search Console's robots tester before pushing them live.