What is Robots.txt
Robots.txt is a file that tells search engines not to index certain pages on your website. You may use robots.txt to block access to specific pages of your site if they contain sensitive information. For example, you might want to prevent search engine spiders from crawling a page containing your credit card number.
You can use robots.txt to tell search engines not to crawl parts of your site, but you cannot use robots.txt to stop search engines from displaying your site map.
In order to use robots.txt, you need to create a special type of web page called a robots.txt file. A robots.txt file contains instructions that tell search engines what to do with each page on your site.
A robots.txt file looks something like this:
User-agent: *
Disallow: /cgi-bin/
Allow: /index.html
The first line of the robots.txt file tells search engines that they should ignore any files named cgi-bin. The second line tells them to ignore any files named index.html.
To make sure that search engines don't display your site map, you have to add a robots.txt file to the root directory of your site. If you don't add a robots.txt to the root directory, then search engines will display your site map.
If you want to allow search engines to display your site map, then you don't need to add a robots.text file to the root directory. Instead, you can just remove the robots.txt file from the root directory.
