September 22, 2017 at 7:41 pm #451
any one can explain me with example
September 23, 2017 at 11:05 pm #506
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
User-agent: * Disallow: /
The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.
So don’t try to use /robots.txt to hide information.
October 25, 2017 at 4:44 am #722
Robots.txt is a text (not html) file we put on our site to tell search robots which pages we would like them not to visit. Robots.txt is not mandatory for search engines but generally search engines obey what user are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from searching our site