{"id":764,"date":"2009-12-14T10:12:02","date_gmt":"2009-12-14T14:12:02","guid":{"rendered":"http:\/\/pro-webs.net\/blog\/?p=764"},"modified":"2010-10-05T06:33:55","modified_gmt":"2010-10-05T10:33:55","slug":"zen-cart-robots-txt-tutorial","status":"publish","type":"post","link":"https:\/\/pro-webs.net\/blog\/2009\/12\/14\/zen-cart-robots-txt-tutorial\/","title":{"rendered":"Zen Cart robots.txt Tutorial"},"content":{"rendered":"

<\/strong><\/p>\n

<\/strong>\"robots.txt<\/strong>
Zen Cart robots.txt<\/figcaption><\/figure>\n

Is there a robots.txt with Zen Cart?<\/strong><\/p>\n

Do you have a default robots.txt?<\/strong><\/p>\n

The answer is no… But here is why.\u00a0 Every cart is different, the very nature of Zen Cart’s flexibility prevents the ability to create a robots.txt that can be applied to even most carts. Things like easy pages and additional pages and installation paths make the process basically a custom procedure for each and every shop. The robots protocol isn’t really difficult, as a matter fact it’s quite logical once you get to know it a bit.<\/p>\n

First, lets cover the basic function and ability of a robots.txt. <\/strong><\/h2>\n

A robots.txt is a text document called robots.txt (exactly), which resides in the root of the domain. You can actually hide your robots.txt from basic Internet users by using a server side application such as Apache to deliver only qualified user agents access to the location and name of the robots directive on your site. This however, is really more work than necessary as password protect directories, such as login admin areas cannot be indexed unless there is a crawl able link to them somewhere.<\/p>\n

robots.txt is an exclusion protocol for well behaved web spider or robots to access in order to receive directives regarding where they may and may not crawl your site’s pages. Blocking a page(s) from crawl in robots.txt WILL NOT prevent them from being indexed or referenced in the search engines results…. It only blocks them from being crawled.<\/p>\n

I will cover the most common uses of robots.txt and their directives below for your Zen Cart.<\/p>\n

First to cover is that there is NO Allow directive in robots.txt, this is in fact the default behavior… So only Disallow is valid. So if I have a page foo.html which I wish not to be crawled by any behaved crawlers, then the syntax will be as follows:<\/p>\n

User-agent: *
\nDisallow: \/foo.html<\/em><\/p>\n

If I wish that a specific crawler, such as GoogleBot not crawl this page, then the syntax is as follows:<\/p>\n

User-agent: Googlebot
\nDisallow: \/foo.html<\/em><\/p>\n

You can view a\u00a0 list of crawlers in the robots database here<\/a>. But for reference her are your most common ones.<\/p>\n