X-Robots-Tag

HTML meta tags only work inside HTML files. Non-HTML resources, PDFs, images, videos, JSON API responses, can't use <meta> tags, leaving a gap in indexing control. X-Robots-Tag fills that gap by applying to any HTTP response. Search Engine Land has documented e-commerce cases where tens of thousands of PDF catalogs got indexed and hurt rankings as duplicate content, fixed in one shot with X-Robots-Tag.

Critical distinction: robots.txt says "don't crawl," while Meta Robots and X-Robots-Tag say "don't index." To block indexing, Googlebot must actually reach the page to read the directive. Blocking in robots.txt stops the crawl entirely, so Google never sees the indexing instruction.

For snippet-level control inside an HTML page, use data-nosnippet on the exact element you want excluded from search snippets. That is different from X-Robots-Tag because it hides only selected text rather than changing the whole resource's indexing state.

Block PDF indexing (Apache .htaccess):

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Block an image directory (Nginx):

location /private-images/ {
  add_header X-Robots-Tag "noindex";
}

Target a specific crawler (Googlebot only):

X-Robots-Tag: googlebot: noindex

Time-limited indexing:

X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 GMT

Element-level snippet exclusion (HTML):

<p data-nosnippet>Do not show this sentence in Google snippets.</p>

Don't combine with robots.txt disallow: If robots.txt blocks the URL, Google can't read the header at all. To block indexing, allow crawling in robots.txt and use X-Robots-Tag noindex.

Requires server configuration: Unlike meta tags, X-Robots-Tag is configured at the web server level (Apache, Nginx, Cloudflare Workers). CMS platforms don't always handle it automatically.

Verify with Search Console or curl: Check that the header actually shows up with Google Search Console's URL Inspection tool or curl -I https://example.com/file.pdf.

Snippet controls are not privacy controls: nosnippet and data-nosnippet change how Google displays a result, but the underlying content remains publicly accessible. Use authentication or server-side access control for private content.