---
title: "Robots.txt Generator: Crawler Rules & Sitemap"
description: "Free robots.txt generator. Create valid crawler rules with allow/disallow paths and sitemap URLs, and avoid common mistakes that deindex websites."
url: https://frontendhelpers.com/robots-txt-generator
updated: 2026-10-12
---

# Robots.txt generator

Create a valid robots.txt that tells search engines what to crawl and what to skip — including your sitemap location.

## How to use this tool

robots.txt is the first file a well-behaved crawler requests from your domain. It controls **crawling**: which paths bots may fetch. That protects your crawl budget and stops search engines wasting attention on carts, admin panels and internal search results.

What it does _not_ control is **indexing**. This distinction causes more SEO damage than any other robots.txt misunderstanding. If another site links to a page you've disallowed, Google can still list that URL in search results — it simply won't have crawled the content, so it shows a bare URL with no description. To keep a page out of the index you must let it be crawled and serve a `noindex` meta tag. Blocking it in robots.txt actively prevents Google from seeing that tag.

## The mistake that deindexes entire sites

A single line — `Disallow: /` — blocks your whole site. It's the correct setting for a staging environment, and catastrophic on production. The classic disaster is a staging robots.txt being deployed to live, and it happens often enough that checking robots.txt should be a standard post-deploy step.

Two subtler traps: robots.txt must live at the domain root (`example.com/robots.txt`) — anywhere else and it's ignored entirely. And it doesn't apply across subdomains, so `blog.example.com` needs its own file.

## Never use it to hide sensitive paths

robots.txt is a public file that anyone can read, and listing a path there advertises its existence. Writing `Disallow: /admin-panel-v2/` tells every curious visitor exactly where to look. It's an instruction to polite crawlers, not an access control, and malicious bots ignore it completely.

Anything genuinely sensitive needs authentication, not a crawler directive. Our [security essentials](https://frontendhelpers.com/essentials#ess-sec) covers the difference between obscurity and actual protection.

## Always declare your sitemap

Adding a `Sitemap:` line is the cheapest indexing help available. It works for every major search engine, requires no account, and gives crawlers a direct route to every URL you care about. Use the absolute URL, and note that this directive is independent of user-agent groups — it applies globally wherever you place it.

After changing robots.txt, test it. Google Search Console's robots.txt report shows exactly how Googlebot interprets your file, which is far more reliable than reading the rules yourself and hoping.

## Keeping staging out of Google: robots.txt alone won't do it

The common recipe is to serve `Disallow: /` on staging and a normal file on production. That stops well-behaved crawlers fetching staging pages, but it doesn't keep them out of the index. If a staging URL is linked from anywhere public — a ticket, a preview link in a pull request, a stray absolute URL on the live site — Google can index the bare URL without ever crawling it. And because the page is never fetched, a `noindex` tag on it is never seen.

The reliable fix is HTTP authentication on every non-production host. Where that isn't practical, leave staging crawlable and send `X-Robots-Tag: noindex` with every response, which works for HTML, PDFs and images alike. Production gets a normal robots.txt with your sitemap. Google doesn't penalise duplicate content as such; the real risk is test data and unfinished pages turning up in results under your name.

```javascript
import { createServer } from 'node:http';

const isProduction = process.env.APP_ENV === 'production';

const ROBOTS = isProduction
  ? 'User-agent: *\nDisallow: /cart\nDisallow: /search\n\nSitemap: https://example.com/sitemap.xml\n'
  : 'User-agent: *\nAllow: /\n'; // stays crawlable, so bots can see the header

createServer((req, res) => {
  // Every response from a non-production host is marked noindex
  if (!isProduction) res.setHeader('X-Robots-Tag', 'noindex');

  if (req.url === '/robots.txt') {
    res.writeHead(200, { 'Content-Type': 'text/plain; charset=utf-8' });
    return res.end(ROBOTS);
  }
  res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
  res.end('<!doctype html><title>Home</title><h1>Home</h1>');
}).listen(3000, () => console.log('production:', isProduction));

// $ curl -i http://localhost:3000/robots.txt   (staging)
// X-Robots-Tag: noindex
// User-agent: *
// Allow: /
```

Note the failure mode this creates: if `APP_ENV` is missing on production, the live site goes out with `noindex`. Add a post-deploy check — `curl -sI https://example.com/ | grep -i x-robots-tag` should print nothing — alongside confirming that production's robots.txt hasn't picked up a blanket `Disallow: /`. Google documents both mechanisms in its [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) and its guide to [blocking indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing).

## Frequently asked questions

### Does robots.txt stop a page appearing in Google?

No. It stops crawling, not indexing. A disallowed page can still be listed if other sites link to it — Google just shows the bare URL with no description. To keep a page out of the index, allow crawling and use a noindex meta tag instead.

### Where does robots.txt need to be located?

At the root of the domain, for example example.com/robots.txt. Placed anywhere else it is ignored completely. Each subdomain needs its own file — blog.example.com will not use the main domain’s robots.txt.

### Can I use robots.txt to hide private pages?

No. It is a public file, so listing a path there advertises its existence to anyone who looks, and malicious bots ignore it entirely. Anything genuinely private needs authentication or server-level access control.

### Should I include my sitemap in robots.txt?

Yes. A Sitemap: line with the absolute URL is the cheapest indexing hint available, works across all major search engines, and applies globally regardless of which user-agent group it sits in.

## Related tools

- [Meta Tag Generator](https://frontendhelpers.com/meta-tag-generator): Generate the meta tags — including noindex — that control indexing.
 
- [Schema Markup Generator](https://frontendhelpers.com/schema-markup-generator): Structured data that helps search engines understand your pages.
 
- [robots.txt vs noindex vs canonical](https://frontendhelpers.com/robots-txt-vs-noindex-vs-canonical): Which one to use for staging sites, PDFs, parameters, moved pages and AI crawlers.

---

This page is a free interactive tool. Use it in your browser at https://frontendhelpers.com/robots-txt-generator