Robots.txt generator

How to use this tool

robots.txt is the first file a well-behaved crawler requests from your domain. It controls crawling: which paths bots may fetch. That protects your crawl budget and stops search engines wasting attention on carts, admin panels and internal search results.

What it does not control is indexing. This distinction causes more SEO damage than any other robots.txt misunderstanding. If another site links to a page you've disallowed, Google can still list that URL in search results — it simply won't have crawled the content, so it shows a bare URL with no description. To keep a page out of the index you must let it be crawled and serve a noindex meta tag. Blocking it in robots.txt actively prevents Google from seeing that tag.

The mistake that deindexes entire sites

A single line — Disallow: / — blocks your whole site. It's the correct setting for a staging environment, and catastrophic on production. The classic disaster is a staging robots.txt being deployed to live, and it happens often enough that checking robots.txt should be a standard post-deploy step.

Two subtler traps: robots.txt must live at the domain root (example.com/robots.txt) — anywhere else and it's ignored entirely. And it doesn't apply across subdomains, so blog.example.com needs its own file.

Never use it to hide sensitive paths

robots.txt is a public file that anyone can read, and listing a path there advertises its existence. Writing Disallow: /admin-panel-v2/ tells every curious visitor exactly where to look. It's an instruction to polite crawlers, not an access control, and malicious bots ignore it completely.

Anything genuinely sensitive needs authentication, not a crawler directive. Our security essentials covers the difference between obscurity and actual protection.

Always declare your sitemap

Adding a Sitemap: line is the cheapest indexing help available. It works for every major search engine, requires no account, and gives crawlers a direct route to every URL you care about. Use the absolute URL, and note that this directive is independent of user-agent groups — it applies globally wherever you place it.

After changing robots.txt, test it. Google Search Console's robots.txt report shows exactly how Googlebot interprets your file, which is far more reliable than reading the rules yourself and hoping.

Developer Implementation Guide: Serving robots.txt Dynamically

While static sites can just drop a robots.txt file in their public folder, dynamic applications often need to change their robots policy based on the environment (e.g., blocking crawlers entirely on staging environments).

Implementation in Node.js / Express

Here is how to serve a dynamic robots.txt file that automatically blocks crawlers unless the application is running in the production environment.

const express = require('express');
const app = express();

app.get('/robots.txt', (req, res) => {
  res.type('text/plain');
  
  if (process.env.NODE_ENV === 'production') {
    // Production: Allow everything, declare sitemap
    res.send(User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml);
  } else {
    // Staging/Dev: Block everything
    res.send(User-agent: *
Disallow: /);
  }
});
        

This simple middleware guarantees that your staging server will never accidentally be indexed by Google, which is a common cause of duplicate content penalties.