Best Robots.txt File For WordPress

A robots.txt file is a plain text file in the root directory of your website that tells the search engine crawlers which sections of your website should be crawled and indexed and which should be hidden.

Using a correct robots.txt file is a must to optimize your WordPress site for search engines.

If you don’t create a robots.txt file for your WordPress site manually, then WordPress will use a virtual robots.txt file. In simple words, there won’t be any robots.txt file in the root directory of your website but if you visit this url:

http://www.yourwebsite.com/robots.txt

You’ll see that that the file contains these rules for robots:

These rules do not block anything from search engine crawlers, and thus they let them index everything from your WordPress website, including admin urls, category, tags and other archives, creating security issues, duplicate content issues and many more other SEO issues.

To fix it, you must use a custom robots.txt file. For that create a plain text file with the name robots.txt and paste the following rules in it:

Now upload this robots.txt file to the root directory of your website. This custom robots.txt file for WordPress ensures that when search engine crawlers visit your WordPress website, they will find and index only that content which will improve your rankings in search engines.