RSS

The robots.txt file


Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many web sites don't offer this file.

If your web site doesn't have a robots.txt file Till, read on to learn how to create one. If you already have a robots.txt file, read our tips to make sure that it doesn't contain errors.

What is robots.txt?

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which web pages of your site should be indexed and which web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.abc.com/robot.txt

How do I create a robots.txt file?

As mentioned above, the robots.txt file is a simple text file. Open a simple text editor to create it. The content of a robots.txt file consists of so-called "records".

A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

Things you should avoid

>> Don't use comments in the robots.txt file
>> Don't use white space at the beginning of a line. For example, don't write
>> Don't change the order of the commands. If your robots.txt file should work, don't mix it up.
>> Don't use more than one directory in a Disallow line. Do not use the following "Disallow: /support /cgi-bin/ /images"
>> Be sure to use the right case.The file names on your server are case sensitve


0 Responses to "The robots.txt file"

Post a Comment

 
Home Design Ideas Fashion Style Trends Home and Garden Gallery Home Designs Arditourism Modestop Topmodest Free Beauty Tips Shitazine Sitahouseblog Izhealthy Shitadesign Sitahouse Topmodest Sitazine Irezine Home- decor-design Shitadesign Shitadesign-dot-com Inhomeland inhouseland ghiuldenika inebautystyle enterarteiuna radiodominiopublico biberealexandru fabiobernadosdi geeklore31 syahadah-alhafeezah peaceloveiman bigwideempty techandchefadventures kaosgunlugu fikirkulubleri duazinciri theinternationalrules morefortngintn monkeymanzach lifeatthegreenys photokid84 chelsea-settles chasingmommyhood sexyhotactresseswallpaper tophotnewstodays avvani urbantrendsussex voychic reisyamode sitazine reportmy homedesignideasx mystylepinterest fashionstyleco popularnewstips isexy-hot-girls sexywomengirl trendymodeku derumahdesign womenhairs hackoblogger gadiswomen homeinteriordesignideas1s homedesignideasa uniquehomedesignsw enterpreneurforbloggers earningblogtipss lilyandgiasmom onfeatureblog aboutadsen onblogsocial tecnologiku onlinekutip akuhub caramakemoney youseoneed blogukus newbieblogit adsenseguideto triond-adsense inseoservice intipseoblog otherseoblog freeseoguid seoblognets tawaguideblog seosecretip buzdesign modewomen gayawoman cantikide tipsehat bebeautyps tipsjilbabcantiks idekontenislami healthfitnessnutritionsz jilbab-tips tips-home-decor caramemakaijilbabv beritaremajaterkinie allisoncrewnews mypinspinterest mypinspinterest gogirlsgallery seyhotgirls topmodest worldotgirls seksigirlsblog yourgirlsnet sexywomensia decorativehousedesigns simplebloggertipsz aboutadsense09s marketingner freeseotipstricksz womensiana chicstylegallery oblogkudewe colhomegallery marketserve onlibusiness artodesign designinguide oninteriorhome interiormu intutorblog blogoffeed newbloggerhint enginseo dibisnistips internetmublog onroomdesign homegardenco
Return to top of page Copyright © 2010 | Flash News Converted into Blogger Template by HackTutors