Robots.txt

A robots.txt file gives instructions to compliant web crawlers about which paths they may access on a site. It is usually located at the root of the domain and can help keep crawlers away from areas that do not need frequent requests. It does not provide password protection and is not a reliable way to keep a URL out of search results: a blocked address can still be discovered through links elsewhere. For a draft page that must not be public, access controls are more appropriate. A noindex directive is different because it addresses indexing, but crawlers generally need to access the page to see that directive.