{"id":641,"date":"2009-10-14T08:30:24","date_gmt":"2009-10-14T12:30:24","guid":{"rendered":"http:\/\/pro-webs.net\/blog\/?p=641"},"modified":"2009-10-14T08:30:24","modified_gmt":"2009-10-14T12:30:24","slug":"crawling-indexing-robotstxt","status":"publish","type":"post","link":"https:\/\/pro-webs.net\/blog\/2009\/10\/14\/crawling-indexing-robotstxt\/","title":{"rendered":"Crawling vs. Indexing – robots.txt Debate"},"content":{"rendered":"

This is really a long and tiresome debate… I personally have argued this point for a long time to no avail with Zen Cart forum<\/a> posters… Even linked to Sebastian<\/a> for verification. But, alas, the debate drags on.<\/p>\n

Much to my absolute delight, Matt Cutts recently (10\/6\/2009) came right out and said… what we have been saying all along… Blocking a page with your robots.txt DOES NOT prevent indexing<\/strong>, only crawling. The correct way to prevent a page from being indexed is with a Meta robots noindex tag… That’s all… and here’s Matt to verify these truths.<\/p>\n

\n<\/embed><\/object><\/p>\n

One final thing…. Your sitemap is not a command for Google or any other search engine to index your pages, it is simply a xml document to assist the search engine spiders in better locating your content for crawl. Google and other search engines are not required to index your pages and alas, you cannot force them either.<\/p>\n","protected":false},"excerpt":{"rendered":"

This is really a long and tiresome debate\u2026 I personally have argued this point for a long time to no avail with Zen Cart forum posters\u2026 Even linked to Sebastian for verification. But, alas, the debate drags on.<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[1305,1304,72,1303,1294,464,1293,1292,689,1291,1300,1298,2626],"class_list":["post-641","post","type-post","status-publish","format-standard","hentry","category-ecommerce-seo","tag-blocking","tag-crawling","tag-google","tag-indexing","tag-meta-robots","tag-meta-tag","tag-noindex-tag","tag-robots-noindex","tag-robots-txt","tag-search-engine-spiders","tag-sebastian","tag-xml-document","tag-zen-cart"],"_links":{"self":[{"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/posts\/641","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/comments?post=641"}],"version-history":[{"count":0,"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/posts\/641\/revisions"}],"wp:attachment":[{"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/media?parent=641"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/categories?post=641"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pro-webs.net\/blog\/wp-json\/wp\/v2\/tags?post=641"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}