{"id":36472,"date":"2021-12-20T11:51:09","date_gmt":"2021-12-20T06:06:09","guid":{"rendered":"https:\/\/rarathemes.com\/blog\/?p=36472"},"modified":"2024-08-21T11:52:07","modified_gmt":"2024-08-21T06:07:07","slug":"optimize-wordpress-robots-txt","status":"publish","type":"post","link":"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/","title":{"rendered":"How to Optimize WordPress Robots.txt For SEO With Example"},"content":{"rendered":"\n<p>SEO has been a top priority for website owners. With top-five results gaining <a href=\"https:\/\/blog.hubspot.com\/insiders\/seo-facts\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">75%<\/a> of the clicks, the race for higher ranking has been a challenge. There are various off-page and <a href=\"https:\/\/rarathemes.com\/blog\/on-page-seo-techniques\/\">on-page SEO techniques<\/a> that come into play. Optimizing the Robot.txt file is one of those tricks which can strengthen the SEO for your site.<\/p>\n\n\n\n<p>However, most beginners don\u2019t know how to optimize WordPress Robots.txt for SEO. Some might not even know what a robots.txt file is in the first place.<\/p>\n\n\n\n<p>Luckily, this article is at your rescue.&nbsp;<\/p>\n\n\n\n<p>Here, we will explain what a robots.txt file is, how to create the robots.txt file, and how to optimize the robots.txt file for SEO. We have also added some FAQs at the end. So make sure you are with us till the end.&nbsp;<\/p>\n\n\n\n<p>Now, let\u2019s get started!&nbsp;<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title ez-toc-toggle\" style=\"cursor:pointer\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#What_is_a_Robotstxt_file\" >What is a Robots.txt file?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#What_Does_a_Robotstxt_File_Look_Like\" >What Does a Robots.txt File Look Like?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Why_Do_You_Need_a_Robotstxt_File_in_WordPress\" >Why Do You Need a Robots.txt File in WordPress?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#How_to_Create_a_Robotstxt_for_Your_Website\" >How to Create a Robots.txt for Your Website?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Method_1_Creating_Robotstxt_file_using_Rank_Math_Plugin\" >Method 1: Creating Robots.txt file using Rank Math Plugin<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Method_2_Creating_Robotstxt_file_using_FTP\" >Method 2: Creating Robots.txt file using FTP<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Where_to_Put_Your_Robotstxt_File\" >Where to Put Your Robots.txt File?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#How_to_Test_The_Robotstxt_File\" >How to Test The Robots.txt File?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Pros_of_Robotstxt_File\" >Pros of Robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Cons_of_robotstxt_File\" >Cons of robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Frequently_Asked_Questions_FAQs\" >Frequently Asked Questions (FAQs)<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#What_is_web_crawling\" >What is web crawling?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#What_is_indexing\" >What is indexing?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#What_is_the_crawl_budget\" >What is the crawl budget?<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/rarathemes.com\/blog\/optimize-wordpress-robots-txt\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_is_a_Robotstxt_file\"><\/span>What is a Robots.txt file?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Robots.txt in WordPress is a plain text file that informs the search engine bots which pages on your site to crawl and index. You can also format the file to exclude the pages from indexing.&nbsp;<\/p>\n\n\n\n<p>The robots.txt is stored in the WordPress root directory. This file has a basic format that looks like this:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: &#91;user-name]\nDisallow: &#91;URL of the string not to be crawled]\nAllow: &#91;URL of the string to be crawled]\nSitemap: &#91;URL of your WordPress XML Sitemap]<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_Does_a_Robotstxt_File_Look_Like\"><\/span>What Does a Robots.txt File Look Like?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A robots.txt file is simply a few lines of text that directs search engines on how and what to crawl on your site. You have multiple lines on this file to allow or restrict specific page URLs. You can also add multiple sitemaps. If you disallow a URL, the search engine bots don\u2019t crawl that page.&nbsp;<\/p>\n\n\n\n<p>Here is an example of the robots.txt file:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-Agent: *\nAllow: \/wp-content\/uploads\/\nDisallow: \/wp-content\/plugins\/\nSitemap: https:\/\/URL\/sitemap_index.xml<\/code><\/pre>\n\n\n\n<p>Now, let\u2019s learn about these four major terms that you can find in a robots file:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>User-agent<\/strong>: User-agent is the specific web crawler to which you give crawl instruction. It is usually a search engine.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Allow<\/strong>: Allow is the command that tells Googlebot that it can access a page or subfolder even though its subfolder or a parent page may be disallowed.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Disallow<\/strong>: The disallow command tells a user-agent not to crawl a particular URL. You can only include one &#8220;Disallow:&#8221; line for each URL.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Sitemap<\/strong>: The sitemap is used to call out the location of any XML sitemap(s) that is related with this URL. This command is only supported by Google, Bing, Ask, and Yahoo.<\/li><\/ul>\n\n\n\n<p>In the above example, we allowed search bots to crawl and index the uploads folder but have restricted the plugins folder. In the end, we allowed indexing of the XML sitemap.<\/p>\n\n\n\n<p>The asterisk sign after <strong><em>User-Agent <\/em><\/strong>means the text applies to all the search engine bots. Each search engine has its user agent to index sites. Here are some user-agents of some popular search engines:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Google: Googlebot<\/li><li>Googlebot Images: Googlebot-Image<\/li><li>Googlebot News: Googlebot-News<\/li><li>Googlebot Video: Googlebot-Video<\/li><li>Yahoo: Slurp Bot<\/li><li>Bing: Bingbot<\/li><li>Amazon\u2019s Alexa: ia_archiver<\/li><li>DuckDuckGo: DuckDuckBot<\/li><li>Yandex: YandexBot<\/li><li>Baidu: Baiduspider<\/li><li>Exalead: ExaBot<\/li><\/ul>\n\n\n\n<p>There are so many user-agents out there nowadays. If you want to focus on a particular search engine, you can even specify it on the robots.txt file. That is, to set up instructions for the Googlebot, you can write the first line of your robots.txt file as:&nbsp;<\/p>\n\n\n\n<p><strong><em>User-agent: Googlebot<\/em><\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Why_Do_You_Need_a_Robotstxt_File_in_WordPress\"><\/span>Why Do You Need a Robots.txt File in WordPress?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Search bots will crawl and index your site even if you don\u2019t have a robots.txt file. So, why would you really need it? Well, with the robots.txt file, you can direct search engines to prioritize indexing on certain pages. Also, you can exclude pages or folders that you don\u2019t want to index.&nbsp;<\/p>\n\n\n\n<p>This does not have a bigger impact when you first<strong><em> <\/em><\/strong>start a blog and don\u2019t have a lot of content. However, after your site grows and has a lot of content, you may need to prioritize indexing on a certain page\/post\/folder and negate some.&nbsp;<\/p>\n\n\n\n<p>You see, there is a quota for each search bot for each website. That means bots crawl on a certain number of pages at a certain crawl session. If the bots don\u2019t finish crawling in that particular session, the bot comes back and resumes crawling in the next session. This slows your website indexing rate.<\/p>\n\n\n\n<p>So, by using the robots.txt file, you can direct search bots on which pages, files, and folders you want indexing and which to include. By doing so, you save the crawl quota. It may not be the safest way to hide your content from the public, but it prevents those excluded contents from appearing in the SERPs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_Create_a_Robotstxt_for_Your_Website\"><\/span>How to Create a Robots.txt for Your Website?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You can create robots.txt in two different ways. We will explain those methods below. You can then follow the method you think will work the best for you.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Method_1_Creating_Robotstxt_file_using_Rank_Math_Plugin\"><\/span>Method 1: Creating Robots.txt file using Rank Math Plugin<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Here, we will show you how to create a robots.txt file using the popular <a href=\"https:\/\/rankmath.com\/\" rel=\"noopener\">Rank Math<\/a> plugin.&nbsp;<\/p>\n\n\n\n<p>Rank Math is an SEO WordPress plugin that makes it easy to optimize your website content with built-in suggestions based on widely-accepted best practices. Using this handy tool, you can easily customize important SEO settings and control which pages are indexable.<\/p>\n\n\n\n<p>In order to create a robots.txt file using the Rank Math plugin, first, download the plugin by going to <strong><em>Plugins -&gt; Add New<\/em><\/strong> from your WordPress Admin Dashboard.&nbsp;<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"665\" height=\"269\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/01\/add-new-plugin-from-dashboard.png\" alt=\"add new plugin from dashboard\" class=\"wp-image-31787\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/01\/add-new-plugin-from-dashboard.png 665w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/01\/add-new-plugin-from-dashboard-300x121.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/01\/add-new-plugin-from-dashboard-148x60.png 148w\" sizes=\"auto, (max-width: 665px) 100vw, 665px\" \/><\/figure><\/div>\n\n\n\n<p>Now, search for Rank Math in the Search bar and click the <strong><em>Install Now <\/em><\/strong>and <strong><em>Activate<\/em><\/strong> button once you find the plugin.&nbsp;<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"593\" height=\"296\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin.png\" alt=\"installing and activating the Rank Math plugin\" class=\"wp-image-33556\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin.png 593w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin-300x150.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin-120x60.png 120w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin-476x238.png 476w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/installing-and-activating-the-Rank-Math-plugin-358x179.png 358w\" sizes=\"auto, (max-width: 593px) 100vw, 593px\" \/><\/figure><\/div>\n\n\n\n<p>Once you activate the plugin, head over to <strong><em>Rank Math -&gt; General Settings -&gt; Edit robots.txt <\/em><\/strong>from the WordPress Dashboard.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"333\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Rank-Math-Robots.txt-Settings.jpg\" alt=\"Rank Math Robots.txt Settings\" class=\"wp-image-36475\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Rank-Math-Robots.txt-Settings.jpg 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Rank-Math-Robots.txt-Settings-300x143.jpg 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Rank-Math-Robots.txt-Settings-126x60.jpg 126w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p>Now, enter the following code or copy the code from mentioned in the previous example in the blank space and click the <strong><em>Save Changes<\/em><\/strong> button.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"510\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Robots.txt-Code.jpg\" alt=\"Save Robots.txt Code\" class=\"wp-image-36476\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Robots.txt-Code.jpg 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Robots.txt-Code-300x219.jpg 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Robots.txt-Code-82x60.jpg 82w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p>There you go! You have successfully created a robots.txt file using the Rank Math plugin.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Method_2_Creating_Robotstxt_file_using_FTP\"><\/span>Method 2: Creating Robots.txt file using FTP<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>The second method involves creating a robots.txt file manually on a local computer and then uploading it to your Wordpress site\u2019s root folder.&nbsp;<\/p>\n\n\n\n<p>You will need to access your WordPress hosting using an FTP client. After logging in to your FTP client, you can see the robots.txt file in the website\u2019s root folder. Right-click and select the <strong><em>Edit <\/em><\/strong>option.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"479\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Creating-Robots.txt-file-using-FTP.png\" alt=\"Creating Robots.txt file using FTP\" class=\"wp-image-36480\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Creating-Robots.txt-file-using-FTP.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Creating-Robots.txt-file-using-FTP-300x205.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Creating-Robots.txt-file-using-FTP-88x60.png 88w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p>Now, add your custom rule to the file and save the changes. For example:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php<\/code><\/pre>\n\n\n\n<p>If you don\u2019t see the file in the root folder, you can create it on your local computer using the notepad. Add your rules on it and then upload it to your rooter folder using the FTP.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Where_to_Put_Your_Robotstxt_File\"><\/span>Where to Put Your Robots.txt File?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You should always place your robots.txt file in the root directory of your website. For example, if your site is <strong>domainname.com<\/strong>, then your robots.txt file domain will be <strong><em>https:\/\/domainname.com\/robots.txt.<\/em><\/strong><\/p>\n\n\n\n<p>In addition to adding the robots.txt file in your root directory, here are some extra tips:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>It is a must to name the file \u201crobots.txt.\u201d<\/li><li>The file name is case-sensitive. Thus, you need to get it right or else it won\u2019t work.<\/li><li>You need to add a new line for each new rule.<\/li><li>Add a \u201c$\u201d sign to signify the end of a URL.<\/li><li>Use each user-agent only once.&nbsp;<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_Test_The_Robotstxt_File\"><\/span>How to Test The Robots.txt File?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>After creating the robots.txt file, you can test it using the <a href=\"https:\/\/search.google.com\/search-console\/about\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Google Search Console<\/a>. Here, you will find the <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Open robot.txt tester tool.<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Select your property from the given drop-down list.&nbsp;<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"334\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Selecting-property-from-the-robot-tester-tool.png\" alt=\"\" class=\"wp-image-36482\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Selecting-property-from-the-robot-tester-tool.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Selecting-property-from-the-robot-tester-tool-300x143.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Selecting-property-from-the-robot-tester-tool-126x60.png 126w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p>The tool fetches your site\u2019s robots.txt file and highlights the errors and warning if it finds any.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"640\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/testing-the-robots-txt-file.png\" alt=\"\" class=\"wp-image-36483\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/testing-the-robots-txt-file.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/testing-the-robots-txt-file-300x274.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/testing-the-robots-txt-file-66x60.png 66w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p><strong><em>Note<\/em><\/strong>: The drop-down property option appears only if you have linked your WordPress site with Google Search Console.<\/p>\n\n\n\n<p>If you haven\u2019t done it, you can quickly link your site as follows:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Log in to <a href=\"https:\/\/search.google.com\/search-console\/about\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Google Search Console<\/a> and create an account. You can set up your account using either your domain or URL prefix. It is recommended to go with the URL prefix as it is comparatively easier.&nbsp;<\/li><li>Enter your website\u2019s URL.&nbsp;<\/li><li>Now, click on the \u201cContinue\u201d button.&nbsp;<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"497\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/setting-up-a-Google-Search-Console-account.png\" alt=\"setting up a Google Search Console account\" class=\"wp-image-33416\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/setting-up-a-Google-Search-Console-account.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/setting-up-a-Google-Search-Console-account-300x213.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/08\/setting-up-a-Google-Search-Console-account-85x60.png 85w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>You should now verify the site\u2019s ownership. You can do this using any of the given options. However, we recommend using the HTML tag option.<\/li><li>Copy the given code and then head over to your WordPress dashboard.<\/li><\/ul>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"609\" height=\"620\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/copying-the-code-for-verification.png\" alt=\"copying the code for verification\" class=\"wp-image-36487\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/copying-the-code-for-verification.png 609w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/copying-the-code-for-verification-295x300.png 295w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/copying-the-code-for-verification-59x60.png 59w\" sizes=\"auto, (max-width: 609px) 100vw, 609px\" \/><\/figure><\/div>\n\n\n\n<p>You can now verify the code using the Insert Headers and Footers plugin. To verify:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Go to <strong><em>Plugins -&gt; Add New<\/em><\/strong> from your WordPress Admin Dashboard.<\/li><li>Now, search for Insert Headers and Footers plugin in the search bar and then Install and Activate the plugin.&nbsp;<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"403\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/09\/activate-insert-header-and-footer-plugin.png\" alt=\"activate insert header and footer-plugin\" class=\"wp-image-33923\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/09\/activate-insert-header-and-footer-plugin.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/09\/activate-insert-header-and-footer-plugin-300x173.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/09\/activate-insert-header-and-footer-plugin-104x60.png 104w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>After that, redirect to <strong><em>Settings -&gt; Header and Footer<\/em><\/strong> from your WordPress Dashboard and then paste the copied code in the Head and footer tab. Once you are done, click the Save button.&nbsp;<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"398\" src=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Code-On-Insert-Headers-and-Footers-Plugin.png\" alt=\"Save Code On Insert Headers and Footers Plugin\" class=\"wp-image-36488\" srcset=\"https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Code-On-Insert-Headers-and-Footers-Plugin.png 700w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Code-On-Insert-Headers-and-Footers-Plugin-300x171.png 300w, https:\/\/rarathemes.com\/blog\/wp-content\/uploads\/2021\/12\/Save-Code-On-Insert-Headers-and-Footers-Plugin-106x60.png 106w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Go back to the Google Search Console tab on your browser. Then, click on the verify button.<\/li><li>Google checks for the ownership and displays the completed message once it is done.<\/li><li>Click on the \u201cGo to property\u201d option to access the Google Search Console dashboard.<\/li><\/ul>\n\n\n\n<p>Now, go back to the <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Open robot.txt tester tool<\/a> and test your robots.txt as mentioned above.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Pros_of_Robotstxt_File\"><\/span>Pros of Robots.txt File<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Here are the benefits of using the Robots.txt file in SEO context:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>It helps to optimize the search engine\u2019s crawl budgets by skipping indexing on the pages that you don\u2019t want the bots crawling. This creates a priority for the search engine to crawl only the pages that are important for you.<\/li><li>It helps optimize the web server by restricting the bots that are unnecessarily consuming resources.<\/li><li>It helps to hide the login pages, landing pages, thank you pages from indexing by search engines. Thus, it optimizes the website\u2019s performance.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Cons_of_robotstxt_File\"><\/span>Cons of robots.txt File<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Creating a robots.txt file is pretty easy, as we have explained above. However, the robots.txt file also includes the URLs to your internal pages that you don\u2019t want crawl bots to index, such as the login page. This makes your website vulnerable to security breaches.<\/li><li>While creating a robots.txt file is easy, making even a simple mistake can pull all your effort down the drain. For example, if you add or misplace a single character in the file, it messes up all your SEO efforts.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Frequently_Asked_Questions_FAQs\"><\/span>Frequently Asked Questions (FAQs)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1639974519844\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"What_is_web_crawling\"><\/span>What is web crawling?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Search engines send their web crawlers (also known as bots or spiders) across the web. These bots are smart software that navigates the entire web to discover new pages, links, and websites. This discovering process is known as web crawling.\u00a0<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1639974526171\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"What_is_indexing\"><\/span>What is indexing?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>When web crawlers discover your website, they arrange your pages into a usable data structure. This organizing process is called indexing.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1639974534866\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"What_is_the_crawl_budget\"><\/span>What is the crawl budget?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>A crawl budget is a limit to how many URLs a search bot can crawl in a given session. Every site has a specific crawl budget allocation. So you need to make sure that you spend it in a way that is most beneficial to your site.<\/p>\n<p>For example, if you have multiple pages on your site, you need to allocate the crawl budget to the most valuable pages. Thus, you can mention the page URL in your robots.txt file.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>When working on a website\u2019s SEO, most of the users focus on creating sitemaps, working on backlinks, researching keywords, and optimizing the content. Only a few webmasters pay attention to the robots.txt file.&nbsp;<\/p>\n\n\n\n<p>The robots.txt file may not play much of a part when you are starting a site. However, as your website grows, paying attention to the robots.txt file yields better results.&nbsp;<\/p>\n\n\n\n<p>Also, there is a myth that blocking category, archive pages, and tags increase the crawl rate and promote faster indexing. It\u2019s false.&nbsp;<\/p>\n\n\n\n<p>Follow the methods we mentioned above to create your robots.txt file.<\/p>\n\n\n\n<p>We hope this article helped you sort your query regarding how to create and optimize WordPress robots.txt for SEO.<\/p>\n\n\n\n<p>If you want to read more useful articles related WordPress, you may read following articles:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/rarathemes.com\/blog\/how-to-create-sitemap-in-wordpress\/\">How To Create a Sitemap in WordPress?&nbsp;<\/a><\/li><li><a href=\"https:\/\/rarathemes.com\/blog\/how-to-start-wordpress-blog\/\">How to Start a WordPress Blog?<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>SEO has been a top priority for website owners. With top-five results gaining 75% of the clicks, the race for higher ranking has been a challenge. There are various off-page &hellip; <\/p>\n","protected":false},"author":15,"featured_media":37009,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_be_single_description":"","_be_image_uploader":0,"_be_meta_button_label":"","_be_meta_button_url":"","_be_meta_button_newtab":false,"_be_meta_rel_attribute":"","footnotes":""},"categories":[6,3],"tags":[],"class_list":["post-36472","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blogging","category-tutorials","latest_post"],"_links":{"self":[{"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/posts\/36472","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/comments?post=36472"}],"version-history":[{"count":0,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/posts\/36472\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/media\/37009"}],"wp:attachment":[{"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/media?parent=36472"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/categories?post=36472"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rarathemes.com\/blog\/wp-json\/wp\/v2\/tags?post=36472"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}