Robots.txt generator

How to use this tool

robots.txt is the first file a well-behaved crawler requests from your domain. It controls crawling: which paths bots may fetch. That protects your crawl budget and stops search engines wasting attention on carts, admin panels and internal search results.

What it does not control is indexing. This distinction causes more SEO damage than any other robots.txt misunderstanding. If another site links to a page you've disallowed, Google can still list that URL in search results — it simply won't have crawled the content, so it shows a bare URL with no description. To keep a page out of the index you must let it be crawled and serve a noindex meta tag. Blocking it in robots.txt actively prevents Google from seeing that tag.

The mistake that deindexes entire sites

A single line — Disallow: / — blocks your whole site. It's the correct setting for a staging environment, and catastrophic on production. The classic disaster is a staging robots.txt being deployed to live, and it happens often enough that checking robots.txt should be a standard post-deploy step.

Two subtler traps: robots.txt must live at the domain root (example.com/robots.txt) — anywhere else and it's ignored entirely. And it doesn't apply across subdomains, so blog.example.com needs its own file.

Never use it to hide sensitive paths

robots.txt is a public file that anyone can read, and listing a path there advertises its existence. Writing Disallow: /admin-panel-v2/ tells every curious visitor exactly where to look. It's an instruction to polite crawlers, not an access control, and malicious bots ignore it completely.

Anything genuinely sensitive needs authentication, not a crawler directive. Our security essentials covers the difference between obscurity and actual protection.

Always declare your sitemap

Adding a Sitemap: line is the cheapest indexing help available. It works for every major search engine, requires no account, and gives crawlers a direct route to every URL you care about. Use the absolute URL, and note that this directive is independent of user-agent groups — it applies globally wherever you place it.

After changing robots.txt, test it. Google Search Console's robots.txt report shows exactly how Googlebot interprets your file, which is far more reliable than reading the rules yourself and hoping.

Keeping staging out of Google: robots.txt alone won't do it

The common recipe is to serve Disallow: / on staging and a normal file on production. That stops well-behaved crawlers fetching staging pages, but it doesn't keep them out of the index. If a staging URL is linked from anywhere public — a ticket, a preview link in a pull request, a stray absolute URL on the live site — Google can index the bare URL without ever crawling it. And because the page is never fetched, a noindex tag on it is never seen.

The reliable fix is HTTP authentication on every non-production host. Where that isn't practical, leave staging crawlable and send X-Robots-Tag: noindex with every response, which works for HTML, PDFs and images alike. Production gets a normal robots.txt with your sitemap. Google doesn't penalise duplicate content as such; the real risk is test data and unfinished pages turning up in results under your name.

import { createServer } from 'node:http';

const isProduction = process.env.APP_ENV === 'production';

const ROBOTS = isProduction
  ? 'User-agent: *\nDisallow: /cart\nDisallow: /search\n\nSitemap: https://example.com/sitemap.xml\n'
  : 'User-agent: *\nAllow: /\n'; // stays crawlable, so bots can see the header

createServer((req, res) => {
  // Every response from a non-production host is marked noindex
  if (!isProduction) res.setHeader('X-Robots-Tag', 'noindex');

  if (req.url === '/robots.txt') {
    res.writeHead(200, { 'Content-Type': 'text/plain; charset=utf-8' });
    return res.end(ROBOTS);
  }
  res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
  res.end('<!doctype html><title>Home</title><h1>Home</h1>');
}).listen(3000, () => console.log('production:', isProduction));

// $ curl -i http://localhost:3000/robots.txt   (staging)
// X-Robots-Tag: noindex
// User-agent: *
// Allow: /

Note the failure mode this creates: if APP_ENV is missing on production, the live site goes out with noindex. Add a post-deploy check — curl -sI https://example.com/ | grep -i x-robots-tag should print nothing — alongside confirming that production's robots.txt hasn't picked up a blanket Disallow: /. Google documents both mechanisms in its introduction to robots.txt and its guide to blocking indexing with noindex.