What is Robot.txt?

[binz]

hello there, Can somebody help me to explore more about robot.txt and how to that EXACTLY? Thankz...

You can get detailed explanation of robot.txt at http://www.robotstxt.org/robotstxt.html. I had the same question few weeks ago and was looking for answers. I found this website very useful.

---
http://www.moveupez.com

[SEO_Internet]

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Google specifically follows instructions given in this robots.txt file

[clearwriter]

Originally Posted by binz

hello there, Can somebody help me to explore more about robot.txt and how to that EXACTLY? Thankz...

You should add a robots.txt file to the root directory of all your websites to help control the indexing of your site by robots that ignore the <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> convention. In this file you specifically list any pages that you DO NOT want walked and indexed (such as password protected folders and folders which contain only images, etc.). The robots.txt file is very simple yet very powerful and every website should have a robots.txt file on the root directory.

Create a new file with Notepad and call it robots.txt
The two conventions used in robots.txt file are User-agent: and Disallow: /
User-agent: * By using the * or wild card you are addressing ALL robots. If you wish to address individual robots you need to list each robot separately with an individual User-agent: statement. They must be listed by their specific name or IP Address, along with a separate Disallow: / statement listing the folders and files you DO NOT want the specified robot to index.
Tip: Use the * wild card to address all robots..... it is the safest way
Disallow: / List any folders that you do not want to have indexed by robots.
Warning: Disallow: / used without any folder name tells the robot do not index ANY page of the website.
ALL Files and folders in the directory named in the Disallow: / statement as well as all of those under it will NOT be indexed by robots.
Sample of Folders that could be in this website that we would not like the spiders to index with the search engines:
Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
Example: Disallow: /tutorials/
Results: All files and sub folders located within the folder tutorials which includes all the folders listed in the above example as well as any other sub folders of the tutorials directory will not be indexed by the robots if you use this statement.

This would mean that the /meta, /images, /assets, /404redirect, AND any other folders as well as all of the files in those foldes will not be seen by indexing robots.
You may also list specific files that you do not want indexed in a robots.txt file.
Sample of Specific Files that could be in this website that we would not like the spiders to index with the search engines:
Disallow: /tutorials/meta_tags.html

see below for an example.

###############################
#
# sample robots.txt file for this website
#
# addresses all robots by using wild card *
#
User-agent: *
# list folders robots are not allowed to index

Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
#
# list specific files robots are not allowed to index
#
Disallow: /tutorials/meta_tags.html
Disallow: /tutorials/custom_error_page.html
#
# End of robots.txt file
#
###############################thanks,
http://www.cleararticles.com/

[Jeffmakepeace]

I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now.