robots.txt

Wed May 18, 2011 6:48 pm

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
sarah.brooks
 
Posts: 137
Joined: Mon Apr 11, 2011 9:23 pm

Fri Jun 03, 2011 5:45 am

Hello Friends........

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Thanks
carl16
 
Posts: 6
Joined: Thu Jun 02, 2011 7:02 am
Location: Canada

Fri Jul 01, 2011 7:53 am

Robots.txt is a regular text file that through its name, has special intending to the giant majority of honorable robots on the net. By defining a few rules in this text file, you can instruct robots to not crawl and index definite files, directories within your site, or at all.
georgebiran
 
Posts: 4
Joined: Fri Jul 01, 2011 7:47 am

Mon Oct 10, 2011 9:46 am

Robot.txt is a text file that is used in a website in order to hide pages. The pages where robot.txt is applied, don't show.
James-Smith2011
 
Posts: 3
Joined: Mon Oct 10, 2011 9:37 am

Thu Nov 24, 2011 9:18 am

The Benefits of the Robots.txt file are:
- Avoid the things that not be shown to search engine.
- Safe guard from the spam spot.
- You can easily handle what to show Google..
mojicerock
 
Posts: 2
Joined: Thu Nov 24, 2011 6:56 am

Thu Mar 01, 2012 6:12 pm

Are there other methods other than robots.txt to prevent search engine bots from crawling sections of your site?
StangMan23
 
Posts: 8
Joined: Fri Feb 10, 2012 4:09 pm
Location: Saint George, UT

Thu May 17, 2012 7:29 am

Basically, robots.txt is a plain text file which is placed in a server's root directory it includes information on whether search engine robots should index the site or parts of the site. This text file contains commands recognized by the search engines that you'll use as rules - rules for handling your site that will help you promote it.
lupinesteve
 
Posts: 13
Joined: Tue May 15, 2012 12:35 pm

Reply

Thu Jan 03, 2013 9:55 am

I gotta bookmark this post of this it seems extremely helpful and very much handy. Actually You have brought up a very good details, thanks you a lot and I truly appreciate this post. Thanks a lot my friend.
cloud_server
 
Posts: 3
Joined: Wed Jan 02, 2013 8:15 pm

Sat Jan 12, 2013 7:01 am

The robots.txt file tells search engines which directories to crawl and which not to. You can use it to block crawlers from looking at your image directory if you don't want your images showing up on google search.
Be careful not to use this to try and block people from directories you want to keep secret. Anyone can view you robots.txt file. Make sure you password protect directories that need to be secured.
martinsherman
 
Posts: 33
Joined: Fri Nov 23, 2012 9:34 am
Location: India

Fri Jun 28, 2013 11:19 am

Oh now I come to know why my SEO guy suggested me to add robots.txt file. Thanks for your information about robots.txt file.
RavinAegis
 
Posts: 9
Joined: Tue Mar 19, 2013 5:31 am

Thu Dec 12, 2013 6:54 am

robot.txt file is used in our website to provide restrictions that what we want to show to search engine crullers on our website and provide restrictions on search engines to show our website for some or to all search engines.
craigkohler
 
Posts: 8
Joined: Tue May 07, 2013 4:44 am

Return to General Affiliate Discussion

Who is online

Users browsing this forum: No registered users and 4 guests