Wordpress duplicate content - Solving an SEO problem with Robots.txt

Posted on March 10, 2013

Duplicate content during SEO optimization is a major problem faced by WordPress users. It can be a major factor in your loss of rank with Google, all because of the inherent category and tag pages that are built into WordPress. This simple robots.txt file will help eliminate those errors.  


But you can fix  this issue by using  robots.txt.what is robots txt? Robots txt is a text file, which come in use to keep control over search engines bot while they crawling your blog or website.you can check your robots.txt by typing http://www.yoursite.com/robots.txt . if don’t have create one and uploaded to your website root directory.


Thanks to Mike at WebTricksBlog for this awesome Robots.txt tip.

Just copy and paste the following text into the robots.txt file in the root of your WordPress site. Don't forget to change the last line of the code to your domain!



  • Paras Sharma says:

    Thanks vinny for such a won derful post, could you please also let me know how we can allow some popular bots. like bing

  • Richard Joe says:

    This topics is very interesting and important for every SEO, thanks for sharing this post.
    best seo service india

  • Christine Williams says:

    Thanks for sharing such useful information!
    You may want to exclude any pages that contain duplicate content. For example, if you offer “print versions” of some pages, you wouldn’t want Google to index duplicate versions as duplicate content could hurt your rankings.

    However, keep in mind that people can still visit and link to these pages, so if the information is the type you don’t want others to see, you’ll need to use password protection to keep it private.

    It’s because there are probably some pages that contain sensitive information you don’t want to show on a SERP.

    Here’s a great guide on Robots.txt file: https://webris.org/wiki/robots-txt/

Leave a Reply

Your email address will not be published.

Contact Us

Have a question? Send us a message. We'll get back to you soon.