---
title: "robots.txt vs noindex vs canonical: Which to Use When"
description: "robots.txt controls crawling, noindex controls indexing, canonical picks one URL. A decision table for staging, PDFs, parameters, moved pages and AI bots."
url: https://frontendhelpers.com/robots-txt-vs-noindex-vs-canonical
updated: 2026-10-12
---
# robots.txt, noindex or canonical: which to use when
Three tools that answer three different questions. Pick the wrong one and the page you meant to hide stays in Google, or the one you meant to keep disappears.
Guide · By [AurumCodeWorks](https://frontendhelpers.com/about) · 7 min read · Updated 12 October 2026
## Three questions, not one
Developers reach for these three interchangeably, but each controls a different stage of how a search engine handles a URL:
- **robots.txt** controls _crawling_: may the bot fetch this URL at all?
- **noindex** controls _indexing_: having fetched it, may the engine store and show it?
- **rel="canonical"** controls _consolidation_: of several URLs showing the same content, which one should represent the group?
Most mistakes come from using a crawling tool to solve an indexing problem, or an indexing tool to solve a duplication problem.
## robots.txt blocks crawling, not indexing
A `Disallow` rule asks compliant crawlers not to fetch a path. It does not remove anything from search. Google's [robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) is explicit that a disallowed URL can still be indexed if other pages link to it; it then appears as a bare result with no description, because Google never read the content.
The second trap follows from the first. If you disallow a page _and_ add `noindex` to it, Google cannot fetch the page, so it never sees the `noindex`. The block defeats the very rule you added. Use robots.txt for what it is good at: keeping crawlers out of large, low-value URL spaces such as internal search results or endless filter combinations. Our [robots.txt generator](https://frontendhelpers.com/robots-txt-generator) builds a valid file.
## noindex: meta tag or HTTP header
For HTML pages, put the rule in the ``:
```html
```
PDFs, images and other non-HTML files have no head, so send the same rule as an `X-Robots-Tag` response header. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) covers both forms. On Cloudflare, a `_headers` file does it in two lines:
```
/downloads/*
X-Robots-Tag: noindex
```
Either way, the URL must stay crawlable. Google does not support `noindex` inside robots.txt, and a page blocked there will never have its tag read. The [meta tag generator](https://frontendhelpers.com/meta-tag-generator) outputs the robots tag alongside the rest of your head.
## rel=canonical is a hint, not a command
A canonical link tells Google which URL you prefer when several show the same content:
```html
```
Google treats this as one signal among several and makes its own choice. Its guide to [consolidating duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) ranks the inputs: redirects and `rel="canonical"` are strong signals, inclusion in a sitemap is a weak one, and internal links should point at the canonical version. HTTPS over HTTP also counts. When those signals disagree, for example the tag says one URL, your navigation links to another and your sitemap lists a third, Google may pick a canonical you didn't intend.
Do not pair `noindex` with a canonical that points to a different URL. One says "this page's signals belong to that URL"; the other says "drop this page entirely". Google's guidance is to use `rel="canonical"`, not `noindex`, for duplicates within a site.
## Redirects: 301 versus 302 and 307
When a URL has genuinely moved, a redirect beats a canonical tag because visitors move too. The status code matters. Google's [redirects documentation](https://developers.google.com/search/docs/crawling-indexing/301-redirects) says permanent redirects (301 and 308) are used as a signal that the target should be canonical, and the target is shown in results. Temporary redirects (302, 303 and 307) are followed, but are not used as that signal, and the source URL tends to stay in results. Use a temporary redirect only when the move really is temporary.
## The decision table
Real scenarios, the tool that fits, and the mistake to avoid:
| Scenario | Use | Why | Don't |
| --- | --- | --- | --- |
| **Staging site** | HTTP authentication or an access gate | Blocks every visitor and bot outright; nothing to crawl or index | Rely on `Disallow: /`, which leaked links can bypass, or ship staging robots.txt to production |
| **Thank-you page** | `noindex` meta tag | No search value, but it must stay crawlable for the tag to be read | Also disallow it in robots.txt |
| **Printer-friendly duplicate** | `rel="canonical"` to the main page | Consolidates signals while the print version stays usable | Add `noindex` on top of the canonical |
| **Old URL moved** | 301 (or 308) redirect | Permanent redirects are a canonical signal and move users too | Use 302 or 307, or block the old URL so the redirect can't be seen |
| **Filter and sort parameters** | robots.txt `Disallow` on parameter patterns you don't need indexed | Stops crawlers spending time on near-infinite combinations | Expect canonical or `noindex` alone to cut crawling quickly |
| **PDF you don't want in results** | `X-Robots-Tag: noindex` header | Non-HTML files can't carry a meta tag | Disallow it in robots.txt; the URL can still appear without a snippet |
| **Out of AI training, still in search** | robots.txt groups for training tokens such as `GPTBot`, `ClaudeBot` and `Google-Extended` | Training and search crawlers use separate user-agent tokens | Block `Googlebot`, `OAI-SearchBot` or `Claude-SearchBot`, which removes you from those search products |
## Parameters and faceted navigation
Shop filters and sort options multiply URLs fast: five filters with a handful of values each produce thousands of crawlable addresses showing near-identical products. Google's guidance on faceted navigation is to prevent crawling of these URLs with robots.txt if you don't need them in search, and to treat canonicals as a slower, weaker way to reduce that crawling. Google supports `*` wildcards in paths:
```
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*colour=
```
If some filtered pages deserve to rank, such as a category plus a popular brand, give them clean URLs, keep the parameter order consistent, and return a 404 when a combination has no results.
## Staging sites: use authentication
A staging site protected only by `Disallow: /` is protected from polite crawlers and nobody else. One link in a ticket, a chat preview or a public repository is enough for the URL to be found and potentially indexed without content. Google's own advice for keeping a page out of results is to password-protect it or use `noindex`. For staging, password protection wins: it also keeps out people. HTTP basic auth or your host's access controls take minutes to set up.
## AI crawlers: search bots versus training bots
The major AI companies split their crawlers by purpose, and robots.txt rules apply per user-agent token:
- **Google-Extended** is not a separate crawler but a control token. Per [Google's crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), it governs use of your content for training future Gemini models and for grounding, and does not affect inclusion or ranking in Google Search.
- **GPTBot** collects training data; **OAI-SearchBot** powers ChatGPT search. [OpenAI's bot documentation](https://developers.openai.com/api/docs/bots) confirms you can allow the second while disallowing the first.
- **ClaudeBot** collects training data; **Claude-SearchBot** indexes content for search results in Claude. Anthropic documents both, plus `Claude-User` for fetches a person requests.
```
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```
The `Content-Signal` line comes from Cloudflare's Content Signals proposal, documented at contentsignals.org. It states your preference for search indexing, AI input (grounding and retrieval) and AI training. It is a declaration, not an access control: crawlers that ignore it are not blocked, which is why the explicit `Disallow` groups still matter.
## A real example: this site's clean-URL migration
FrontendHelpers originally served pages at `.html` addresses. Our host strips the extension automatically, but with a **307**, a temporary redirect. Google doesn't treat temporary redirects as a canonical signal, so the old `.html` URLs could have stayed the indexed versions indefinitely. We changed three things together:
- **Explicit 301s** from every `.html` path to its clean URL, so the move is declared permanent.
- **Self-referencing canonicals** on every page, pointing at its clean URL, so each page names itself as the preferred version.
- **An updated sitemap** and internal links that list only clean URLs, so the weaker signals agree with the strong ones.
Just as important was what we didn't do: block the `.html` paths in robots.txt. Google has to crawl an old URL to see its redirect.
## Checking your work in Search Console
**URL Inspection** answers the per-page questions: whether crawling is allowed, whether a `noindex` was detected, and both the canonical you declared and the one Google selected. Use "Test live URL" after a change.
The **Page indexing** report shows the patterns. Statuses worth knowing:
- _Discovered – currently not indexed_: Google found the URL but hasn't crawled it yet.
- _Crawled – currently not indexed_: crawled, but Google chose not to index it.
- _Indexed, though blocked by robots.txt_: the crawling-versus-indexing trap above, in practice.
- _Duplicate, Google chose different canonical than user_: your canonical hint lost to other signals.
- _Alternate page with proper canonical tag_: working as intended, no action needed.
If a page you expected to rank is missing altogether, work through [why your website isn't on Google](https://frontendhelpers.com/why-is-my-website-not-on-google) for the full checklist.
## Frequently asked questions
### Does robots.txt stop a page from being indexed?
No. robots.txt only stops compliant crawlers from fetching the page. Google can still index a disallowed URL if other pages link to it, and show it in results without a description. To keep a page out of the index, allow crawling and add a noindex rule, or put the page behind a password.
### Can I use noindex and a canonical tag on the same page?
Not when the canonical points to a different URL. The canonical says this page is a duplicate whose signals belong elsewhere, while noindex says drop this page entirely, so the two conflict. For duplicates, Google recommends rel=canonical rather than noindex. A self-referencing canonical on a noindexed page is harmless but does nothing useful.
### Why is Google ignoring my canonical tag?
Because rel=canonical is a hint, not a rule. Google weighs it alongside redirects, sitemap entries, internal links and HTTPS, and picks the URL it judges most useful. If your internal links, sitemap or redirects point at a different URL from your canonical tag, make them agree. URL Inspection shows both the canonical you declared and the one Google selected.
### Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a control token, not a separate crawler. Disallowing it tells Google not to use your content for training future Gemini models or for grounding, and Google states it has no effect on inclusion or ranking in Google Search. Blocking Googlebot is what removes you from search.
## Keep going
- [robots.txt Generator](https://frontendhelpers.com/robots-txt-generator): Build a valid robots.txt with crawler groups and your sitemap.
- [Meta Tag Generator](https://frontendhelpers.com/meta-tag-generator): Robots, canonical, Open Graph and Twitter tags in one go.
- [Why is my website not on Google?](https://frontendhelpers.com/why-is-my-website-not-on-google): Seven checks, in the order that finds the cause fastest.