How To Add Custom Robots.txt File In Blogger

Today once again I am going to share a post related to blogging and improving the appearance on the search engines like Google,Yahoo,Bing and others so that you can boost the traffic to your blog and increase the rank.So many of our readers know what is the robots.txt file and why it is important in search engine optimization.

Image Source:Google Image Search

What is Robots.txt:

Robots.txt is a text file of a website where there are few commands for search engine bots to permit and to deny the pages to be crawled and indexed. In robots.txt we also can add the sitemap of our blog so that search engine robots can index all the pages from our it.
The robots.txt file looks like following:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

So what does these lines mean:

• User-agent: Mediapartners-Google

This code is for Google Adsense robots due to which they can serve relevant ads on your blog.Leave this line as it is although you are not using adsense.

• User-agent: *

This is for all robots marked with asterisk (*). Don't worry about this, leave it as it is.

• Disallow: /search

That means the links having keyword search just after the domain name will be ignored. See below example which is a link of label page named Blogging Tips.

http://www.tricksdroid.com/search/label/Blogging%20Tips

And if we remove Disallow: /search from the above code then crawlers will access our entire blog to index and crawl all of its web pages.

• Allow: /

Here Allow: / refers to the Homepage that means web crawlers can crawl and index homepage of your blog.

Disallow Particular Post
Now suppose if we want to exclude a particular post from indexing then we can add below lines in the code.

Disallow: /yyyy/mm/post-url.html

Here yyyy and mm refers to the publishing year and month of the post respectively.Here is a example:

Disallow: /2013/03/post-url.html

Just copy the link of post and remove the first part having name of your blog.

• Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

This line tells the crawlers about sitemap of your blog. If you add this line with your sitemap, the entire blog will be crawled by the robots of search engines.
This sitemap mentioned above contains only recent 25 posts. If you want to increase the number of link in your sitemap then replace default sitemap with below one. It will work for first 500 recent posts

http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you have more than 500 published posts in your blog then you can use two sitemaps like below:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

Must Read:
500+ High Pagerank Forums To Get Powerful Dofollow Backlinks And Boost TrafficHow to Make Blog Popular in Search Engines

How To Add Robots.txt File In Blogger:

Just follow the steps given below:

• Go to your blogger blog.

• Then Navigate to Settings -- Search Preferences -- Crawlers and indexing -- Custom robots.txt -- Enable/Edit -- Yes.

• Now paste robots.txt file code in the box we just created.

• Click on Save Changes button.

• Check it by visiting http://yourblog.com/robots.txt

That's it you have successfully added a Robots.txt file to your blogger and in some time you will notice that traffic to your blog is being increased.

For any questions, queries and suggestions leave a comment below. Be connected and keep visiying for more such tricks and tips.

Content Worth A Share ?

• Facebook
• Twitter
• Google+
• Pinterest
• StumbleUpon
• Dig
• LinkedIn

Written by Unknown

Satyam Bora Is 18 Years Old Boy Addicted To Blogging And Technology.He Is A Part Time Blogger And Student Of Civil Engineering. This Guy Is Always Here To Help You :)