http://pro-webs.net/blog/2009/10/14/crawling-indexing-robotstxt/ E-Commerce Tips, Tricks and Tribulations Tue, 16 Feb 2010 17:25:56 +0000 http://wordpress.org/?v=2.9.2 hourly 1 http://pro-webs.net/blog/2009/10/14/crawling-indexing-robotstxt/comment-page-1/#comment-465 SEO - Myth, Method or Discipline | E-Commerce for All Sat, 31 Oct 2009 15:43:20 +0000 http://pro-webs.net/blog/?p=641#comment-465 [...] My robots.txt will prevent indexing of the pages blocked…. Just plain not true. Further more, for some lousy search engines it’s a joke and less than effective in even preventing crawling of blocked pages. [...] [...] My robots.txt will prevent indexing of the pages blocked…. Just plain not true. Further more, for some lousy search engines it’s a joke and less than effective in even preventing crawling of blocked pages. [...]

]]>