Crawl-control guide
robots.txt does not keep a page out of Google
Google's robots.txt introduction says the file tells crawlers which URLs they can access, and that it does not keep a page out of Google.
Source Google Search Central robots.txt introduction, pulled .
What a robots.txt file tells a crawler
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. Google says this is used mainly to avoid overloading your site with requests.
It does not keep a page out of Google
A robots.txt file is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
Do not use a robots.txt file as a means to hide your web pages from Google Search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. To block the page from search results, use another method such as password protection or noindex.
If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description.
A disallowed URL can still be found
Google will not crawl or index the content blocked by a robots.txt file. Google might still find and index a disallowed URL if it is linked from other places on the web. The URL, and other public information such as anchor text in links to the page, can still appear in Google Search results.
To prevent the URL from appearing in Google Search results, password-protect the file, use the noindex meta tag or response header, or remove the page.
What the file cannot force
robots.txt rules may not be supported by all search engines. The instructions cannot enforce crawler behavior. It is up to the crawler to obey them. Googlebot and other respectable web crawlers obey the instructions in a robots.txt file. Other crawlers might not. To keep information secure from web crawlers, use another blocking method, such as password-protecting private files.
Respectable web crawlers follow the rules in a robots.txt file, and each crawler might interpret the rules differently. Know the proper syntax for addressing different web crawlers, because some might not understand certain instructions.
Combining multiple crawling and indexing rules might cause some rules to counteract other rules.
FAQ
Does robots.txt remove a page from Google?
No. Google says a robots.txt file is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
Can a blocked URL still appear in search results?
Yes. If a web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description. Google might still index a disallowed URL when other pages link to it.
Do all crawlers obey robots.txt?
No. Google says the instructions cannot enforce crawler behavior. Googlebot and other respectable web crawlers obey them. Other crawlers might not. Different crawlers might also interpret the rules differently.