X-Robots-Tag: Crawler Directives in HTTP Headers
The X-Robots-Tag is an HTTP response header that applies crawler directives like noindex or nofollow to any URL.
X-Robots-Tag is an HTTP header that applies crawler directives to a URL.
What It Is
The X-Robots-Tag is an HTTP response header that tells crawlers how to treat a URL.
It supports the same directives as the robots meta tag, like noindex and nofollow.
Why Use It
- It works on file types that have no HTML, like PDFs and images.
- It applies to whole URL patterns when set at the server level.
- It can override or complement meta tags.
How It Works
- The server sends the header with each response.
- Values can be set per file type, like PDF or image.
- Header rules and meta tags both apply when both are present.
Best Practices
- Use it to remove non-HTML files from the index.
- Be careful with sitewide headers that block whole sections.
- Verify the header with URL Inspection or curl.
- Combine with the robots meta tag when both make sense.
Before: thousands of PDF reports are appearing in search results.
After: an X-Robots-Tag with noindex is set for all PDF responses.
The result: the PDFs drop out of the index without changing a single page.
The lesson: headers control what the HTML never touches.
Quick Tip
The X-Robots-Tag is the right tool for non-HTML files that the robots meta tag cannot reach.
Frequently Asked Questions
X-Robots-Tag, in Short
Some files never see a meta tag.
The header reaches them anyway.