X-Robots-Tag: indexing control for PDFs and non-HTML files
A meta robots tag only works in HTML. Here is how the X-Robots-Tag HTTP header controls indexing for PDFs, images, and files search and AI crawlers fetch.
A meta robots tag has to sit inside an HTML head. That works for pages, but it does nothing for a PDF whitepaper, a product image, or a downloadable spreadsheet, because none of those files have a head to put it in. The X-Robots-Tag HTTP header solves that gap. It carries the same directives as a meta robots tag, but it rides on the server response instead of the markup, so it works on any file type a crawler can fetch.
What the header actually does
X-Robots-Tag is an HTTP response header. When a crawler requests a file, the server can return a header like X-Robots-Tag: noindex alongside the file itself. Googlebot, Bingbot, and most AI crawlers read response headers the same way they read HTML, so the directive applies before the crawler even opens the file. That matters for AI answer engines in particular: many of them fetch PDFs and slide decks directly when a page links to one, and a header is the only way to tell them not to index that file.
- PDFs, such as whitepapers, spec sheets, and reports
- Images and video files with no HTML wrapper
- Word documents, spreadsheets, and other downloads
- Any non-HTML resource served directly by your server or a CDN
How it differs from a meta robots tag
The directives are identical: noindex, nofollow, noarchive, nosnippet, and the rest all work the same way. The difference is only where you put them. A meta robots tag lives in the page's head, and only a browser or crawler that parses HTML will ever see it. An X-Robots-Tag header goes out with every response for a given URL, HTML or not, which is what makes it useful for files. If a page and its linked PDF both need to stay out of the index, you need a meta tag on the page and a header on the PDF. One does not cover the other.
Where to set it
You set the header at the server or CDN level, not in the file itself. On Apache, it is usually added through a rewrite or headers directive scoped to a file type, so every PDF in a directory gets the same rule without editing each file. On Nginx, the same thing happens with a location block matched by file extension. Most CDNs and static hosts, including the major cloud storage providers, offer a way to attach custom response headers to a file or a whole path. The common thread is that you are configuring the server, not the document, which means one rule can cover thousands of files at once.
Mistakes that cancel it out
The header only works if the crawler can actually request the file and read the response. A few setups quietly break that.
- Blocking the file in robots.txt. A disallowed URL is never fetched, so the crawler never sees the noindex header, and the file can still appear in search results from other signals like links.
- Applying the rule too broadly. A directory-wide header meant for PDFs that also matches HTML pages in the same folder will deindex pages you wanted to keep.
- Assuming a CDN passes headers through unchanged. Some caching layers strip or override custom headers unless you explicitly configure them to forward.
- Forgetting that noindex needs to be crawlable to take effect, the same rule that trips people up with the HTML meta tag.
How to check it
Confirming the header is live does not require special tools.
- Request the file's headers directly rather than opening it in a browser, using any HTTP client that shows response headers.
- Look for X-Robots-Tag in the response and confirm the value matches what you intended.
- Check the URL in Search Console's URL inspection tool, which reports whether Google sees the file as indexable.
- Re-check after any CDN or hosting change, since a cache configuration update can silently drop custom headers.
A directive that never reaches the crawler is the same as no directive at all.
If your site links out to PDFs, spec sheets, or other downloads that should not show up in search or get pulled into an AI answer, check whether those files carry any indexing signal at all. Most sites never think about it, because the meta robots tag trains you to look inside the document. For anything that is not HTML, the fix lives in the server config instead.