robots.txt
A file defining fetch rules for crawlers that follow them. It is not protection for private data. A blocked address can still appear in results; blocking does not replace authentication or noindex.
Sources and review
Reviewed:
A file defining fetch rules for crawlers that follow them. It is not protection for private data. A blocked address can still appear in results; blocking does not replace authentication or noindex.
Reviewed: