Dev

Member
Joined
Dec 29, 2014
Messages
21
Reaction score
3
Hello everyone,

I hope this is in the correct place. I'm having an issue with GWT and it has me pretty worried..

A few days back I logged in to see that a warning message had appeared saying Googlebot cannot access certain pages on my site when it tries to crawl them.. I used Wordpress and the Google XML Sitemap plug in, as well as Yoast. I've dug and dug and dug and I cannot find any reason why this is happening. My goal for the new year is to completely clean up my website from back end to front. Currently I'm running way too many plug ins and my sitemap is a mess, but that's beside the point..

I also found that about a week ago (roughly 1 week before GBot couldnt access parts of site) my index status doubled.. I haven't added any new content in a couple months... I do have a staging server set up but its noindexed so it can't be that (I hope)...

Here are my 2 sitemaps..
http://thebestguitarlessons.com/sitemap.xml
http://thebestguitarlessons.com/sitemap_index.xml

I don't have 2 intentionally.. I've had these plug ins for so long I can't even remember why I added the XML plug in if yoast generates one, too..

GWT message:
Over the last 24 hours, Googlebot encountered 14 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 36.8%

Index status:
index-issue.jpg


There are also pages that I don't see in my sitemaps that GBot is trying to crawl and returning 404s to me for...
404-pages.jpg


I know this isn't related to local SEO but any help would be GREATLY appreciated!! I'm pretty nervous this is going to result in a ding... :eek:
 
Last edited by a moderator:
Joined
Aug 7, 2013
Messages
751
Reaction score
404
Hey Dev,

You should definitely clean up the site map. You might have conflicts between the plugins. Download this plugin called Hobo Bot. It shows you what Google is crawling. It might show you some trends, like Google only crawling sitemap pages. Putting aside that Google is confused on the sitemap, I would focus on guiding Google to crawl the pages you want by re-organizing the robots.txt. help Google crawl the pages you want. Also, Noticed you have a bunch of 307 re-directs. Change those to 301's.
 
  • Like
Reactions: Dev

Dev

Member
Joined
Dec 29, 2014
Messages
21
Reaction score
3
Hi Dusty,

Thank you for your response! Truth be told I have very little experience with setting up any redirects and haven't ever done any on my site- including the 307's that you've found. I'm not saying they aren't there, but I didn't put them there- at least not intentionally. Could you tell me where you found that and how I may go about changing them to 301s?

I'm thinking of deleting the XML sitemaps plug in and just using the Yoast sitemap feature. Will also definitely look into Hobo Bot.

Thanks again!
 

Dev

Member
Joined
Dec 29, 2014
Messages
21
Reaction score
3
Thanks for the reply. I recvently deleted the XML sitemaps plugin and only use the Yoast one now. I also deleted the old maps from GWT and submitted the Yoast sitemap. I'm hoping this will help to solve the index status issue. Traffic for my site also saw a huge bump right after the new year, going from an avg 250 hits a day to 350 fairly consistently so far. I'm hoping this is as a result of something good I've done and not just a 'mistake' that will end in a 'correction' that brings me to less avg traffic than I used to get. Anyway, I'm enjoying the traffic spike no-less, temporary or not...

I actually just had a thought that I can't believe I didn't think of earlier...

I use the plug in Pretty Links on my site to create natural looking affiliate links.. Maybe those are the redirects you're seeing? For a while the links were normal but a few weeks ago I finally got around to nofollowing all of them..
 
Similar threads
Thread starter Title Forum Replies Date
S Odd Links to my website found in google webmaster tools Local Search 28
J So I was checking my site on Google Webmaster Tools Local Search 2
B Google Webmaster Tools & Local SEO Local SEO Tools & Software 7
Greg Schueler Google Webmaster Tools 'Average Position' is factoring in Ad positions? Organic SEO 2
mborgelt New Google Webmaster Tools Features Organic SEO 5
Dustybones Google Webmaster Tools API Update 2014 Local Search 0
Jo Shaer Webmaster Tools to Google My Business Success Google My Business & Google Maps 0
C Google webmaster tools showing links that I don't have. Help & Support for Google Local 3
C Google webmaster tools problem Organic SEO 1
djbaxter Webmaster Tools and Google Analytics: Dangerous Security Flaw? Websites, Software, and Security 3
Conor Treacy Google Webmaster Conference Mountain View Break Room: Chat and Off Topic 1
Eric Rohrback Rendering AJAX-crawling pages - From Google Webmaster Blog Organic SEO 0
djbaxter Google Webmaster Guidelines Updated Again Organic SEO 2
Dustybones Google Webmaster Guideline Update for Website Content Local Search 2
Bishop Google Webmaster "Average Position" makes no sense! Local Search 1
Linda Buquet Important New Google Webmaster Guideline Updates re Backlinks Organic SEO 1
djbaxter Link Madness: On Google outsourcing to webmasters Organic SEO 0
Linda Buquet NEW Google Webmaster Guidelines & Rich Snippet Guidelines Local Search 2
E "Duplicate" Profile That Google Claims We Already Manage? Google Duplicates & Merges 1
T Can a Credit Union Create Google Posts Without Falling Under Prohibited Content? Google My Business & Google Maps 1
djbaxter Google My Business Bug Dropping SAB Businesses In Independence, Kansas Service Area Businesses 0
JamesGreenaway Google Insights query Google My Business & Google Maps 1
RobertCarnitas Google Posts Live on GMB, But Not Showing in GMB Dashboard Google My Business & Google Maps 1
rickvidallon How Do You Access The Buttons In Google My Business For Online Services? Local Search 4
NexstarNickR Google Star Rating: LSA vs. GMB All Things Google My Business [PRIVATE] (LocalU) 2
Phil Rozek Google Maps reviews get "New" label for reviews less than 1 month old Local Reviews 5
JoyHawkins Get Live Inventory Displayed on Your Google My Business Listing Google My Business & Google Maps 6
JoyHawkins Google Testing 2-Packs on Mobile Google News: Important Changes & Features 4
Paul Gregory Google Maps default location bias distance and the Google Maps Places API Google Algorithm Discussions 2
Chris Ratchford Google Analytics - setting up goals for click-to-call Organic SEO 2
L Google ranking in Legal Field Organic SEO 1
Cherie Dickey Something funny going on in Google Maps... am I missing something? Google My Business & Google Maps 1
hajnasiewicz Google Marketing Kit: Create Video Local Content 3
S Google LSA Lead Board - How to manage? Paid Search and Local Service Ads 1
krystaltaing Google My Business Bulk Verification Limit Reached? Multi-Location Issues 0
K Unable to get verified in Google for SAB Local Search 14
Matt Chauhan Google Maps showing category that doesn't even exist Local Search 1
djbaxter Brightlocal Tool shows Large Google Local Search Algorithm Update March 10, 2021 Google Algorithm Discussions 1
D Local Services and Google my Business Paid Search and Local Service Ads 2
F Google Maps Citation confusion? Google My Business & Google Maps 3
K Google requiring storefront photos Google Duplicates & Merges 4
Rich Owings Google community challenge to update 100,000 businesses Google My Business & Google Maps 0
cleverlyengaged Advice on Landmarks/POI/Unclaimed Google Map Locations? Google My Business & Google Maps 6
R What steps should I take to get my suspended and denied Google My business approved? Help & Support for Google Local 1
Jac Google My business Photos - Uploaded but not reflecting on contribution page Google My Business & Google Maps 3
AndySimpson Google My Business Opening Date - What Date Should You Use? Google My Business & Google Maps 2
Lanerizz Google Analytics 4 Organic SEO 1
M Google Local Services Ads - Lack of Leads Paid Search and Local Service Ads 14
rustybrick Google Local ranking update? Google Algorithm Discussions 12
harmony How Does Google Choose Service Options in a Review? Local Reviews 6

Similar threads

Login / Register

Already a member?   LOG IN
Not a member yet?   REGISTER

Most UpVoted Answers

LocalU Podcasts

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...
Google Product Exert


Top Bottom