Three questions, not one
Developers reach for these three interchangeably, but each controls a different stage of how a search engine handles a URL:
- robots.txt controls crawling: may the bot fetch this URL at all?
- noindex controls indexing: having fetched it, may the engine store and show it?
- rel="canonical" controls consolidation: of several URLs showing the same content, which one should represent the group?
Most mistakes come from using a crawling tool to solve an indexing problem, or an indexing tool to solve a duplication problem.
robots.txt blocks crawling, not indexing
A Disallow rule asks compliant crawlers not to fetch a path. It does not remove anything from search. Google's robots.txt introduction is explicit that a disallowed URL can still be indexed if other pages link to it; it then appears as a bare result with no description, because Google never read the content.
The second trap follows from the first. If you disallow a page and add noindex to it, Google cannot fetch the page, so it never sees the noindex. The block defeats the very rule you added. Use robots.txt for what it is good at: keeping crawlers out of large, low-value URL spaces such as internal search results or endless filter combinations. Our robots.txt generator builds a valid file.
noindex: meta tag or HTTP header
For HTML pages, put the rule in the <head>:
<meta name="robots" content="noindex">PDFs, images and other non-HTML files have no head, so send the same rule as an X-Robots-Tag response header. Google's noindex documentation covers both forms. On Cloudflare, a _headers file does it in two lines:
/downloads/*
X-Robots-Tag: noindexEither way, the URL must stay crawlable. Google does not support noindex inside robots.txt, and a page blocked there will never have its tag read. The meta tag generator outputs the robots tag alongside the rest of your head.
rel=canonical is a hint, not a command
A canonical link tells Google which URL you prefer when several show the same content:
<link rel="canonical" href="https://example.com/guides/print-styles">Google treats this as one signal among several and makes its own choice. Its guide to consolidating duplicate URLs ranks the inputs: redirects and rel="canonical" are strong signals, inclusion in a sitemap is a weak one, and internal links should point at the canonical version. HTTPS over HTTP also counts. When those signals disagree, for example the tag says one URL, your navigation links to another and your sitemap lists a third, Google may pick a canonical you didn't intend.
Do not pair noindex with a canonical that points to a different URL. One says "this page's signals belong to that URL"; the other says "drop this page entirely". Google's guidance is to use rel="canonical", not noindex, for duplicates within a site.
Redirects: 301 versus 302 and 307
When a URL has genuinely moved, a redirect beats a canonical tag because visitors move too. The status code matters. Google's redirects documentation says permanent redirects (301 and 308) are used as a signal that the target should be canonical, and the target is shown in results. Temporary redirects (302, 303 and 307) are followed, but are not used as that signal, and the source URL tends to stay in results. Use a temporary redirect only when the move really is temporary.
The decision table
Real scenarios, the tool that fits, and the mistake to avoid:
| Scenario | Use | Why | Don't |
|---|---|---|---|
| Staging site | HTTP authentication or an access gate | Blocks every visitor and bot outright; nothing to crawl or index | Rely on Disallow: /, which leaked links can bypass, or ship staging robots.txt to production |
| Thank-you page | noindex meta tag | No search value, but it must stay crawlable for the tag to be read | Also disallow it in robots.txt |
| Printer-friendly duplicate | rel="canonical" to the main page | Consolidates signals while the print version stays usable | Add noindex on top of the canonical |
| Old URL moved | 301 (or 308) redirect | Permanent redirects are a canonical signal and move users too | Use 302 or 307, or block the old URL so the redirect can't be seen |
| Filter and sort parameters | robots.txt Disallow on parameter patterns you don't need indexed | Stops crawlers spending time on near-infinite combinations | Expect canonical or noindex alone to cut crawling quickly |
| PDF you don't want in results | X-Robots-Tag: noindex header | Non-HTML files can't carry a meta tag | Disallow it in robots.txt; the URL can still appear without a snippet |
| Out of AI training, still in search | robots.txt groups for training tokens such as GPTBot, ClaudeBot and Google-Extended | Training and search crawlers use separate user-agent tokens | Block Googlebot, OAI-SearchBot or Claude-SearchBot, which removes you from those search products |
Parameters and faceted navigation
Shop filters and sort options multiply URLs fast: five filters with a handful of values each produce thousands of crawlable addresses showing near-identical products. Google's guidance on faceted navigation is to prevent crawling of these URLs with robots.txt if you don't need them in search, and to treat canonicals as a slower, weaker way to reduce that crawling. Google supports * wildcards in paths:
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*colour=If some filtered pages deserve to rank, such as a category plus a popular brand, give them clean URLs, keep the parameter order consistent, and return a 404 when a combination has no results.
Staging sites: use authentication
A staging site protected only by Disallow: / is protected from polite crawlers and nobody else. One link in a ticket, a chat preview or a public repository is enough for the URL to be found and potentially indexed without content. Google's own advice for keeping a page out of results is to password-protect it or use noindex. For staging, password protection wins: it also keeps out people. HTTP basic auth or your host's access controls take minutes to set up.
AI crawlers: search bots versus training bots
The major AI companies split their crawlers by purpose, and robots.txt rules apply per user-agent token:
- Google-Extended is not a separate crawler but a control token. Per Google's crawler documentation, it governs use of your content for training future Gemini models and for grounding, and does not affect inclusion or ranking in Google Search.
- GPTBot collects training data; OAI-SearchBot powers ChatGPT search. OpenAI's bot documentation confirms you can allow the second while disallowing the first.
- ClaudeBot collects training data; Claude-SearchBot indexes content for search results in Claude. Anthropic documents both, plus
Claude-Userfor fetches a person requests.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /The Content-Signal line comes from Cloudflare's Content Signals proposal, documented at contentsignals.org. It states your preference for search indexing, AI input (grounding and retrieval) and AI training. It is a declaration, not an access control: crawlers that ignore it are not blocked, which is why the explicit Disallow groups still matter.
A real example: this site's clean-URL migration
FrontendHelpers originally served pages at .html addresses. Our host strips the extension automatically, but with a 307, a temporary redirect. Google doesn't treat temporary redirects as a canonical signal, so the old .html URLs could have stayed the indexed versions indefinitely. We changed three things together:
- Explicit 301s from every
.htmlpath to its clean URL, so the move is declared permanent. - Self-referencing canonicals on every page, pointing at its clean URL, so each page names itself as the preferred version.
- An updated sitemap and internal links that list only clean URLs, so the weaker signals agree with the strong ones.
Just as important was what we didn't do: block the .html paths in robots.txt. Google has to crawl an old URL to see its redirect.
Checking your work in Search Console
URL Inspection answers the per-page questions: whether crawling is allowed, whether a noindex was detected, and both the canonical you declared and the one Google selected. Use "Test live URL" after a change.
The Page indexing report shows the patterns. Statuses worth knowing:
- Discovered – currently not indexed: Google found the URL but hasn't crawled it yet.
- Crawled – currently not indexed: crawled, but Google chose not to index it.
- Indexed, though blocked by robots.txt: the crawling-versus-indexing trap above, in practice.
- Duplicate, Google chose different canonical than user: your canonical hint lost to other signals.
- Alternate page with proper canonical tag: working as intended, no action needed.
If a page you expected to rank is missing altogether, work through why your website isn't on Google for the full checklist.