Thursday, March 5, 2009

How to change robots.txt in blogger


When I started using Google webmaster tools , one day I got to see something like this:

URLs restricted by robots.txt : 24 urls

When I saw the details, then I came to know that most of the restricted urls are nothing but the "labels" I created on my blog. Did a lot of Google and finally I learnt that we cant change the robots.txt file in Blogger (unless we host our blog in our server).

The I looked at my rebots.txt (http://all-mixed.blogspot.com/robots.txt) which looks like this:

-----------------------------
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search

Sitemap: http://all-mixed.blogspot.com/feeds/posts/default?orderby=updated
-----------------------------

So,
"Disallow: /search" seems to be fine as it just not going to index the search result pages (labels too), which are kind of duplicate (as individual post url is going to be indexed anyway, if no error)

** In the above default site map, the line
"User-agent: *" means that allow all types of user agents to index your blog content.

If the information I put above seems wrong, please correct me. thanks.

5 comments:

Vietnamese Beauties said...

I recently discovered this when I visited the Google Webmaster Tool Central.
This really affected the traffic to my site.
http://vietnamesebeauties.blogspot.com

Peter J said...

same for me, it had such a bad affect that i lost my ranking on google.

http://www.ideasbubble.blogspot.com

Undeadstories said...

My adsense blocked by robot txt. Any solution to actived my blog again ? Please share

This my site : www.undeadstories204.blogspot.com

Sanjito Kurniawan said...

Hate blogger for disallowing their users to edit robots.txt...
But we could still change them with robots meta tag, and it seems to be working on mine.
- http://www.s4nji-blog.com

Manipal King said...

My blogDental Surgeon Blog is also suffering from the same problem