Robots.txt vs Noindex: Which Does What, and the Classic Mistake

Robots.txt tells crawlers which URLs they may fetch. Noindex tells search engines not to list a page they’ve fetched. To keep a page out of Google, use noindex and leave the page crawlable. If you block it in robots.txt as well, Google can’t see the noindex, and the URL can stay in results. Most people assume the opposite, because “block” sounds stronger.

What each one does

Google’s robots.txt introduction says the file “is not a mechanism for keeping a web page out of Google.” A URL blocked there can still appear in results if other pages link to it. It just appears with no description, since Google never read the page.

The classic mistake

Here is how it usually plays out. Someone notices that /internal-reports/ shows up in Google, and takes two steps to be thorough. First they add a noindex to the pages:

<meta name="robots" content="noindex">

And then they add a block to robots.txt:

User-agent: *
Disallow: /internal-reports/

Weeks later, the pages are still in the results. The second step undid the first. Google can’t fetch the pages, so it never sees the noindex. Its documentation is explicit: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file.” Otherwise “the crawler will never see the noindex rule, and the page can still appear in search results.”

In Search Console, this shows up as Indexed, though blocked by robots.txt. Google’s own advice for that status is to remove the block and use noindex.

The fix:

1. Remove the Disallow rule covering those URLs.
2. Keep the noindex on the pages.
3. Let Google recrawl them. Using URL Inspection on a few important ones can speed things up.
4. Confirm in Search Console that the status has changed to “Excluded by ‘noindex’ tag”.
5. Only then, if you want to save crawling, consider adding the block back. Be aware of the risk: if new links point to those URLs, they can be listed again with no description. Most sites should leave the block off.

Which to use

Keep a page out of search results: noindex. Think thank-you pages, internal search results, thin tag archives, and staging copies that have to stay public for some reason.

Keep content private: neither. Use a login or other access control. A robots.txt file is public, and listing your sensitive paths there tells everybody where to look. Well-behaved crawlers honour it, but it’s a request and not a barrier. Google’s documentation notes that password protection generally removes a page from the index too.

Stop crawlers wasting time on endless URLs: robots.txt. Filter combinations, calendar pages that go on forever, and session parameters can produce huge numbers of URLs. Blocking those is what the file is for, particularly on large sites.

Page is gone for good: return 404 or 410. Google drops the URL once it recrawls it. Don’t block it, or Google won’t see the status code.

Page has moved: a 301 redirect. If you’ve blocked the old URL, the redirect can’t be seen.

Remove something from Google urgently: the Removals tool in Search Console. It hides a URL temporarily, so you still need a noindex, a removal of the content, or a login behind it before the removal expires. Otherwise the URL can return.

Duplicate content: a canonical, not either of these. Google says not to use robots.txt for canonicalization and doesn’t recommend noindex for it within a site. See the canonical tags guide.

Writing noindex correctly

The meta tag goes in the <head>:

<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">

The first applies to all crawlers that support it, and the second only to Google. The response header does the same job and is the only option for non-HTML files. Google says the two “have the same effect”:

X-Robots-Tag: noindex

Here’s an nginx example for PDFs in one folder:

location /downloads/ {
    add_header X-Robots-Tag "noindex";
}

Some pitfalls:

• Don’t count on JavaScript. Google warns that on seeing noindex, “it may skip rendering and JavaScript execution”, so removing the tag with script may not work. Adding it with script works only if the script gets run.
• Noindex takes time. It applies when Google next crawls the page, which may be weeks on a low-priority URL.
• Accidental noindex is a costly launch mistake. A CMS setting or staging template can carry it into production. Because the header isn’t visible in page source, check response headers as well after each launch.

Writing robots.txt correctly

Robots.txt has to be at the root of each host, for example https://example.com/robots.txt. Subdomains need their own.

User-agent: *
Disallow: /search
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml

Details worth knowing, from Google’s specification:

• Rules match by prefix. Disallow: /search also blocks /search-tips. Use /search/ if you mean the folder.
• Size limit. Google reads only the first 500 KiB of the file.
• Caching. Google generally caches the file for up to 24 hours, so changes aren’t immediate.
• Status codes matter. Most 4xx responses count as “no robots.txt”, meaning everything is allowed. A 5xx response is more serious. For the first 12 hours, Google stops crawling the site while it keeps trying to fetch the file. It may then use a cached copy for up to 30 days. So a broken robots.txt can stop crawling across your whole site.
• noindex inside robots.txt doesn’t work. Google stopped supporting that unofficial rule on September 1, 2019. Any old guide suggesting Noindex: /page is out of date.
• Don’t block CSS or JavaScript that the page needs. Google renders pages, and blocking these can make it misread your layout and content.

The most damaging robots.txt is only two lines long, and is often left over from a staging setup:

User-agent: *
Disallow: /

A check of this file belongs on every launch checklist.

Other crawlers

These rules cover search engines that follow them, and support differs by crawler. Not every bot reads the googlebot meta tag, and hostile scrapers ignore robots.txt altogether. To stop unwanted bots, you’ll need server-side controls such as rate limiting or firewall rules. A robots.txt rule can’t enforce anything.

How to check your setup

Our Robots.txt Analyzer fetches a site’s file, parses its groups and tests whether a path is allowed for a given bot. Our Robots Meta Checker reports the robots and googlebot meta tags and the X-Robots-Tag header for a live URL. Run both on any page you think is excluded. If the first says blocked and the second says noindex, you’ve found the classic mistake.

These tools fetch pages with their own user agent and read the HTML as served. If your server treats Googlebot differently, they won’t reveal that. Use the URL Inspection tool in Search Console for the final word on how Google sees the page, and see why your page isn’t indexed for the wider list of causes.