The directives you specify in a robots.txt file are only requests. Although most search engines and many web crawlers respect these directives, they are not obligated to do so. Therefore, you should never rely on the robots.txt file to hide content you do not want indexed.
Developer corner
Controlling search engines and web crawlers using the robots.txt file
Master search engine control with our concise robots.txt guide, featuring a complete walkthrough and code snippets.
You can specify which sections of your site you would like search engines and web crawlers to index, and which sections they should ignore. To do this, you specify directives in a robots.txt file, and place the robots.txt file in your document root directory.
In this example, any crawler (specified by the User-agent directive and the asterisk wildcard) can access any file on the site.
In this example, all crawlers are instructed to ignore all files on the site.
In this example, all crawlers are instructed to ignore the scripts directory.
In this example, all crawlers are instructed to ignore the documents/index.html directory.
In this example, all crawlers are instructed to wait at least 30 seconds between successive requests to the web server.