What robots.txt controls — and what it doesn't
robots.txt is the first file a well-behaved crawler requests from your domain. It controls crawling: which paths bots may fetch. That protects your crawl budget and stops search engines wasting attention on carts, admin panels and internal search results.
What it does not control is indexing. This distinction causes more SEO damage than any other robots.txt misunderstanding. If another site links to a page you've disallowed, Google can still list that URL in search results — it simply won't have crawled the content, so it shows a bare URL with no description. To keep a page out of the index you must let it be crawled and serve a noindex meta tag. Blocking it in robots.txt actively prevents Google from seeing that tag.
The mistake that deindexes entire sites
A single line — Disallow: / — blocks your whole site. It's the correct setting for a staging environment, and catastrophic on production. The classic disaster is a staging robots.txt being deployed to live, and it happens often enough that checking robots.txt should be a standard post-deploy step.
Two subtler traps: robots.txt must live at the domain root (example.com/robots.txt) — anywhere else and it's ignored entirely. And it doesn't apply across subdomains, so blog.example.com needs its own file.
Never use it to hide sensitive paths
robots.txt is a public file that anyone can read, and listing a path there advertises its existence. Writing Disallow: /admin-panel-v2/ tells every curious visitor exactly where to look. It's an instruction to polite crawlers, not an access control, and malicious bots ignore it completely.
Anything genuinely sensitive needs authentication, not a crawler directive. Our security essentials covers the difference between obscurity and actual protection.
Always declare your sitemap
Adding a Sitemap: line is the cheapest indexing help available. It works for every major search engine, requires no account, and gives crawlers a direct route to every URL you care about. Use the absolute URL, and note that this directive is independent of user-agent groups — it applies globally wherever you place it.
After changing robots.txt, test it. Google Search Console's robots.txt report shows exactly how Googlebot interprets your file, which is far more reliable than reading the rules yourself and hoping.