Robots.txt Settings
What is Robots.txt?
Robots.txt is a file that tells search engine bots which parts of your site they can and cannot crawl. It must be located in the root directory of your site.
Warning
Robots.txt is not sufficient to protect private content. To protect sensitive content, use server-side authentication.