Go Back   Web Design & SEO Company > SEO

SEO Search Engine Optimization, this section lists articles and tutorials on Search Engine Optimization for various Search Engines including Google, Yahoo & MSN. We provide SEO hints, tips and other free goodies to help you optimize your site and to start ranking well in the Search Engines.

Reply
 
  #1  
Old 12-21-2007, 06:38 PM
Junior Member
 
Join Date: Dec 2007
Posts: 9
Default What is Robot.txt?

hello there, Can somebody help me to explore more about robot.txt and how to that EXACTLY? Thankz...
Reply With Quote
Top SEO Tool
Harvester and Mass Blog Commenter
Blog Comment Software

  #2  
Old 03-03-2008, 09:06 PM
moveupez
Guest
 
Posts: n/a
Default Robot.txt

You can get detailed explanation of robot.txt at http://www.robotstxt.org/robotstxt.html. I had the same question few weeks ago and was looking for answers. I found this website very useful.

---
http://www.moveupez.com
Reply With Quote
  #3  
Old 03-07-2008, 12:52 PM
Member
 
Join Date: Mar 2008
Location: Canada
Posts: 30
Default ROBOTS Search engine optimization

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Google specifically follows instructions given in this robots.txt file
Reply With Quote
  #4  
Old 03-26-2008, 12:58 PM
Junior Member
 
Join Date: Mar 2008
Posts: 11
Default

Quote:
Originally Posted by binz View Post
hello there, Can somebody help me to explore more about robot.txt and how to that EXACTLY? Thankz...
You should add a robots.txt file to the root directory of all your websites to help control the indexing of your site by robots that ignore the <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> convention. In this file you specifically list any pages that you DO NOT want walked and indexed (such as password protected folders and folders which contain only images, etc.). The robots.txt file is very simple yet very powerful and every website should have a robots.txt file on the root directory.

Create a new file with Notepad and call it robots.txt
The two conventions used in robots.txt file are User-agent: and Disallow: /
User-agent: * By using the * or wild card you are addressing ALL robots. If you wish to address individual robots you need to list each robot separately with an individual User-agent: statement. They must be listed by their specific name or IP Address, along with a separate Disallow: / statement listing the folders and files you DO NOT want the specified robot to index.
Tip: Use the * wild card to address all robots..... it is the safest way
Disallow: / List any folders that you do not want to have indexed by robots.
Warning: Disallow: / used without any folder name tells the robot do not index ANY page of the website.
ALL Files and folders in the directory named in the Disallow: / statement as well as all of those under it will NOT be indexed by robots.
Sample of Folders that could be in this website that we would not like the spiders to index with the search engines:
Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
Example: Disallow: /tutorials/
Results: All files and sub folders located within the folder tutorials which includes all the folders listed in the above example as well as any other sub folders of the tutorials directory will not be indexed by the robots if you use this statement.

This would mean that the /meta, /images, /assets, /404redirect, AND any other folders as well as all of the files in those foldes will not be seen by indexing robots.
You may also list specific files that you do not want indexed in a robots.txt file.
Sample of Specific Files that could be in this website that we would not like the spiders to index with the search engines:
Disallow: /tutorials/meta_tags.html

see below for an example.

###############################
#
# sample robots.txt file for this website
#
# addresses all robots by using wild card *
#
User-agent: *
# list folders robots are not allowed to index

Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
#
# list specific files robots are not allowed to index
#
Disallow: /tutorials/meta_tags.html
Disallow: /tutorials/custom_error_page.html
#
# End of robots.txt file
#
###############################thanks,
http://www.cleararticles.com/
Reply With Quote
  #5  
Old 04-21-2008, 06:31 PM
Junior Member
 
Join Date: Apr 2008
Posts: 3
Default Robot

I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now.
Reply With Quote
  #6  
Old 08-08-2008, 10:54 AM
Member
 
Join Date: Jul 2008
Posts: 77
Default

I think Robot.txt is usedto prevent crawlers crawl on a certain page...
Reply With Quote
  #7  
Old 08-08-2008, 11:30 AM
Junior Member
 
Join Date: Jul 2008
Posts: 16
Default

that my question too.. i don't know either how exactly robot.txt works.. hmmmm...
Reply With Quote
  #8  
Old 08-27-2008, 12:19 PM
Member
 
Join Date: May 2008
Posts: 38
Default

this is used in meta tag in you HTML coding this allow crawler to indexed or not your page in database .
Reply With Quote
  #9  
Old 09-02-2008, 10:12 AM
Junior Member
 
Join Date: May 2008
Posts: 9
Default

A file written and stored in the root directory of a website that restricts the Search Engine spiders from indexing certain pages of the website.
Reply With Quote
  #10  
Old 09-03-2008, 08:17 AM
Member
 
Join Date: Sep 2008
Posts: 46
Default

A file written and stored in the root directory of a website that restricts the Search Engine spiders from indexing certain pages of the website.
Reply With Quote
Top SEO Tool
Harvester and Mass Blog Commenter
Blog Comment Software

SEO - Top



Web Design & SEO Forums