What is a robots.txt file?


This topic contains 2 replies, has 3 voices, and was last updated by shane shane 3 weeks, 5 days ago.

  • Author
  • #451 Reply

    any one can explain me with example

  • #506 Reply
    Sam Smith
    Sam Smith

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

    It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

    User-agent: *
    Disallow: /

    The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.

    There are two important considerations when using /robots.txt:

    robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

    the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

    So don’t try to use /robots.txt to hide information.

  • #722 Reply

    Robots.txt is a text (not html) file we put on our site to tell search robots which pages we would like them not to visit. Robots.txt is not mandatory for search engines but generally search engines obey what user are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from searching our site

Reply To: What is a robots.txt file?

You can use BBCodes to format your content.
Your account can't use Advanced BBCodes, they will be stripped before saving.

Your information: