djbaxter

Administrator
Administrator
Moderator
Joined
Jun 28, 2012
Messages
3,515
Reaction score
1,661
WordPress robots.txt Example
by Yoast
February 5, 2015

No longer is Google the dumb little kid that just fetches your sites HTML and ignores your styling and JavaScript. It fetches everything and renders your pages completely. This means that when you deny Google access to your CSS or JavaScript files, it doesn?t like that at all. My recent post about Google Panda 4 shows an example of this.

To see whether your site can be fully rendered, you can do a Fetch & Render in Google Webmaster Tools crawl section (as it happens, we?ll do a post on that tomorrow in our series on Google Webmaster Tools, so check back then).

This means that the old often heard best practice of having a robots.txt that blocks access to your <code>wp-includes</code> directory and our own old best practice of blocking your plugins directory are no longer valid. This is why, in WordPress 4.0, I opened the issue and wrote the patch to remove <code>wp-includes/*</code> from the default WordPress robots.txt.

Something else is very important to keep in mind. If you block a URL with your site?s robots.txt, search engines will not crawl those pages. This also means that they cannot distribute the link value pointing at those URLs. So if you have a section of your site that you?d rather not have showing in the search results, but does get a lot of links, don?t use the robots.txt file. Instead, use a robots meta tag with a value <code>noindex, follow</code>. This allows search engines to properly distribute the link value for those pages across your site.

Read more...
 
Similar threads
Thread starter Title Forum Replies Date
Garrett Sussman The Modern Loyalty Program for Small Businesses Local Reviews 6
rich_marlatt What Wordpress plugins do you recommend for SEO and Schema Markup? Ask a LocalU Expert [PRIVATE] (LocalU) 4
djbaxter WordPress: Contact Form 7 File Upload Vulnerability Websites, Software, and Security 0
djbaxter WordPress: The NoneNone Brute Force Attacks: Currently Active Websites, Software, and Security 0
djbaxter New features in WordPress 5.6 Websites, Software, and Security 0
djbaxter PHP 8: What WordPress Users Need to Know Websites, Software, and Security 0
djbaxter Speed Test for WordPress sites Websites, Software, and Security 1
djbaxter New WordPress Toolkit from cPanel Websites, Software, and Security 0
djbaxter Facebook & Instagram embeds on WordPress will break soon Websites, Software, and Security 1
djbaxter WordPress 5.5 update breaks plugins: Here’s the fix Websites, Software, and Security 6
Andrew Scherer Hacking QDF with WordPress Plugins Local Content 2
djbaxter Bing URL Submissions Plugin For WordPress Websites, Software, and Security 2
A What Wordpress Website Builder & Theme do you use? (Page speed in mind) Organic SEO 4
A What Wordpress Speed Optimizing Plugins do you use? Websites, Software, and Security 7
djbaxter WordPress Sites Targeted in Large-Scale Attacks Websites, Software, and Security 0
C Don’t we all want to develop fast websites? WordPress fastest page speeds using background images <srcset>, <img>, <picture>, @media, @2x retina Websites, Software, and Security 0
djbaxter Ninja Forms WordPress Plugin: High Severity Vulnerability Patched Websites, Software, and Security 0
djbaxter Test your Contact Form 7 on WordPress sites! Recycle Bin 4
djbaxter Critical security flaw in WordPress Jetpack plugin Websites, Software, and Security 0
djbaxter Site Kit by Google WordPress plugin Websites, Software, and Security 11
djbaxter WordPress Rich Reviews Plugin Under Active Attack Websites, Software, and Security 1
djbaxter Malicious WordPress Redirect Campaign Attacking Several Plugins Websites, Software, and Security 1
P New wordpress website Websites, Software, and Security 5
Chris Ratchford Anyone familiar w/ Advanced Custom Fields (for WordPress)? Consultant's Corner 6
brettmandoes Speed Plugin for WordPress sites Websites, Software, and Security 19
Jo Shaer Auto publishing from Wordpress blog to GMB post Google My Business & Google Maps 5
D Google Reviews Widget for WordPress Recycle Bin 0
djbaxter Security vulnerability in WordPress Slick Popup Plugin Websites, Software, and Security 1
djbaxter Stay current with the latest WordPress and Plugins Security Issues with this newsletter Websites, Software, and Security 0
djbaxter Urgent! Serious Security Threat Found in WordPress Plugin Yuzo Related Posts Websites, Software, and Security 1
djbaxter Grammarly Adds Junk Code to WordPress Posts and Pages Websites, Software, and Security 4
djbaxter Vulnerabilities in Two WordPress Plugins Websites, Software, and Security 2
djbaxter WordPress 5.1.1 Patches Critical Vulnerability: Update now Websites, Software, and Security 0
R Free Webinar: Wix, Weebly, Squarespace and WordPress - Which Is Best? Events 0
JoyHawkins Who is Switching to Wordpress 5.0? Websites, Software, and Security 13
djbaxter Google WordPress Plugin to Integrate Analytics, Search Console, AdSense, PageSpeed Websites, Software, and Security 4
Dan Foland Has anyone else been experiencing a rise in Brute Force attacks on WordPress recently? Websites, Software, and Security 4
djbaxter WordPress Alert: PHP 5.6 and 7.0 reach EOL December 2018 Websites, Software, and Security 2
Josh Gill New Wordpress Plugin to Manage GMB Posts Google My Business & Google Maps 12
P Broadband provider blocks wordpress site: Effect on SEO / local search? Local Search 2
P Guide/suggestions for improving my GMB local listing? wordpress plugins? Local Search 14
djbaxter Google Reviews Widget for WordPress Websites, Software, and Security 8
djbaxter WordPress 4.9.8 Released, May Need to Install the Classic Editor Websites, Software, and Security 8
djbaxter WordPress vulnerability, all versions: Check your Author and higher role permissions Websites, Software, and Security 0
djbaxter How to Disable Gutenberg in WordPress Websites, Software, and Security 3
Caroline Google My Business Website PLUS a WordPress one Local Search 8
P Wordpress template for business company Local Search 7
djbaxter WordPress 4.9.4 Fixes Critical Auto Update Bug in 4.9.3 Websites, Software, and Security 0
djbaxter Check your WordPress plugins! Organic SEO 0
dannanelli WordPress Footer Section as a Page Organic SEO 7

Similar threads

Login / Register

Already a member?   LOG IN
Not a member yet?   REGISTER

Most UpVoted Answers

LocalU Podcasts

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...
Google Product Exert


Top Bottom