Modernizing WordPress robots.txt

[djbaxter]

WordPress robots.txt Example
by Yoast
February 5, 2015

No longer is Google the dumb little kid that just fetches your sites HTML and ignores your styling and JavaScript. It fetches everything and renders your pages completely. This means that when you deny Google access to your CSS or JavaScript files, it doesn?t like that at all. My recent post about Google Panda 4 shows an example of this.

To see whether your site can be fully rendered, you can do a Fetch & Render in Google Webmaster Tools crawl section (as it happens, we?ll do a post on that tomorrow in our series on Google Webmaster Tools, so check back then).

This means that the old often heard best practice of having a robots.txt that blocks access to your <code>wp-includes</code> directory and our own old best practice of blocking your plugins directory are no longer valid. This is why, in WordPress 4.0, I opened the issue and wrote the patch to remove <code>wp-includes/*</code> from the default WordPress robots.txt.

Something else is very important to keep in mind. If you block a URL with your site?s robots.txt, search engines will not crawl those pages. This also means that they cannot distribute the link value pointing at those URLs. So if you have a section of your site that you?d rather not have showing in the search results, but does get a lot of links, don?t use the robots.txt file. Instead, use a robots meta tag with a value <code>noindex, follow</code>. This allows search engines to properly distribute the link value for those pages across your site.

Read more...