Google Webmaster Tools Issue

[Dev]

Hello everyone,

I hope this is in the correct place. I'm having an issue with GWT and it has me pretty worried..

A few days back I logged in to see that a warning message had appeared saying Googlebot cannot access certain pages on my site when it tries to crawl them.. I used Wordpress and the Google XML Sitemap plug in, as well as Yoast. I've dug and dug and dug and I cannot find any reason why this is happening. My goal for the new year is to completely clean up my website from back end to front. Currently I'm running way too many plug ins and my sitemap is a mess, but that's beside the point..

I also found that about a week ago (roughly 1 week before GBot couldnt access parts of site) my index status doubled.. I haven't added any new content in a couple months... I do have a staging server set up but its noindexed so it can't be that (I hope)...

Here are my 2 sitemaps..
http://thebestguitarlessons.com/sitemap.xml
http://thebestguitarlessons.com/sitemap_index.xml

I don't have 2 intentionally.. I've had these plug ins for so long I can't even remember why I added the XML plug in if yoast generates one, too..

GWT message:

Over the last 24 hours, Googlebot encountered 14 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 36.8%

Index status:

There are also pages that I don't see in my sitemaps that GBot is trying to crawl and returning 404s to me for...

I know this isn't related to local SEO but any help would be GREATLY appreciated!! I'm pretty nervous this is going to result in a ding...

 

[Dustybones]

Hey Dev,

You should definitely clean up the site map. You might have conflicts between the plugins. Download this plugin called Hobo Bot. It shows you what Google is crawling. It might show you some trends, like Google only crawling sitemap pages. Putting aside that Google is confused on the sitemap, I would focus on guiding Google to crawl the pages you want by re-organizing the robots.txt. help Google crawl the pages you want. Also, Noticed you have a bunch of 307 re-directs. Change those to 301's.

 

[Dev]

Hi Dusty,

Thank you for your response! Truth be told I have very little experience with setting up any redirects and haven't ever done any on my site- including the 307's that you've found. I'm not saying they aren't there, but I didn't put them there- at least not intentionally. Could you tell me where you found that and how I may go about changing them to 301s?

I'm thinking of deleting the XML sitemaps plug in and just using the Yoast sitemap feature. Will also definitely look into Hobo Bot.

Thanks again!

 

[Dustybones]

Morning Dev. I use a plugin called Redirection to manage my redirects. Here is a tutorial for Redirection and Yoast. I would stick with Yoast as it is a respected plugin. Delete any plugins that will conflict with it. Hope that helps.

 

[Dev]

Thanks for the reply. I recvently deleted the XML sitemaps plugin and only use the Yoast one now. I also deleted the old maps from GWT and submitted the Yoast sitemap. I'm hoping this will help to solve the index status issue. Traffic for my site also saw a huge bump right after the new year, going from an avg 250 hits a day to 350 fairly consistently so far. I'm hoping this is as a result of something good I've done and not just a 'mistake' that will end in a 'correction' that brings me to less avg traffic than I used to get. Anyway, I'm enjoying the traffic spike no-less, temporary or not...

I actually just had a thought that I can't believe I didn't think of earlier...

I use the plug in Pretty Links on my site to create natural looking affiliate links.. Maybe those are the redirects you're seeing? For a while the links were normal but a few weeks ago I finally got around to nofollowing all of them..