What is a robots.txt file and How to add in the website

  • Home
  • How To
  • What is a robots.txt file and How to add in the website
Robots_txt file

If you are a blogger or you have just started blogging, then you must have heard about the robots.txt file in Blogger or WordPress, you already know about robots. This robot is a machine of spring and it follows our command the same work of robots.txt file, whatever command we will give to this file, it sends this command directly to the search engine, as you know, the job of a search engine is to search for your post logo. Look at it and index your post, so let’s know about it better.

What is a robots.txt file

As you know from the name itself, .txt is an extension of a text file in which we can only write text, similarly, robots.txt is a file in which we write some text or messages, in this message we will send parts of our blog. Let’s write about which part to show in the search engine and which not!

For example, if you do not want to show pages from your blog in the search engine, or if you do not want to index the category or tags in your blog, then you can write all these things inside the robots.txt file. Any search engine like google, yahoo, bing will start indexing your blog post, then it will send a message to all these search engines from robots.txt file to index all these things from the blog and do not index all these things, it benefits this makes your blog SEO friendly and it makes it easier for search engines to an index which part and which not.

How to Add Robots.txt File in Blogger

In Blogger, you will find the option of robots.txt file under search preferences, when we make our Blogspot blog SEO friendly, then we get the option of this robots.txt file, which most people leave like this in blogger. If you leave the option, blogger automatically makes the best robots.txt file for you, but if you want some parts of your blog to be high in search engine then you can do it but if you are taking blogging seriously and you later migrated your Blogspot blog to WordPress, then you must use this feature. Now let’s learn how to add this file to the Blogspot blog. Before that, let us know about some things that you will use inside this file.

Allow: If you allow something inside the text file, then it means that you want this part to be visible in the search engine.

Disallow: If you disallow a post, it means that you do not want this part to appear in the search engine.

User-Agent: By using this option, you can show your visitors what you want to show on your site, as if the source of the image is this file, where is it saved, etc.

User-agent: Mediapartners-Google: With its use, you can hide the Adsense code from the user, if you disallow it, then Google Adsense will not be able to show ads in your site, that is why you always allow it.

1. Go to Blogger’s settings, click on search preferences, then click on robots.txt, then click on edit.

2. Now you have to copy the code given below and paste it inside the box and save it.

User-agent: Mediapartners-Google
Disallow: / search
Allow: /
Sitemap: http://www.capatintechno.in/sitemap.xml

Note: Put your sitemap URL in place of site map and save the setting.

This way you can heighten and show some posts of your blog by using a robots.txt file in your BlogSpot blog if you are using a blogger, then it is better to not use this feature and it will be good, if you are using WordPress, then this feature must be used.

How to add Robots.txt file in WordPress

1. If you use WordPress, then you must have installed Yoast SEO, then you have to go to Yoast SEO and click on Tools then click on File editor.

yoast robots_txt file

2. Now copy and paste the code below into the robots.txt file and then save it by clicking on save changes to Robots.txt

User-agent: *
Disallow: / wp-admin /
Disallow: / cgi-bin /
Disallow: / comments / feed /
Disallow: / trackback /
Disallow: /xmlrpc.php
Allow: /wp-admin/admin-ajax.php

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /
Sitemap: https://www.captaintechno.in/sitemap_index.xml

Note: Replace sitemap URL with your sitemap

yoast robots txt

Tips: If you use the Blogspot blog then you do not add robots.txt file, then Google automatically creates the best robots file for you and if you use WordPress, then you must use this robot file again. If you want to ask anything, then you can ask in the comment box below.


Roshan Mehta

Get the latest technology news and updates across the world on captaintechno. Our Blog will provide you all the latest technology-related updates from all over the world.

Blog Comments

thanks bro

Thanks For sharing This information.good information to provide us

“Nice info!”
“Great share!”
“Useful post”
“Amazing write-up!”

Leave a Comment