X-Robots-Tag

X-Robots-Tag is an HTTP response header that communicates search indexing or presentation directives for a requested resource. It can be useful for files such as PDFs or images, where an HTML robots meta tag may not apply. A service company might use an appropriate noindex directive for a document it does not want listed in search results, while still allowing authorized crawlers to retrieve it. The header is different from robots.txt, which addresses crawling access. A blocked crawler may never see the directive, and noindex is not a security measure for private files. Server configuration and testing are essential before applying rules broadly.