More threads by lancemoore22

Joined
Mar 26, 2015
Messages
36
Reaction score
13
Hi!

I have a client's website that isn't indexed in Google, however it has previously been indexed. And on occasion, the homepage might be indexed for a day then disappear.

Google search console said there is a crawl anomaly. I've done the research on that and here's what all I've troubleshot.

  • I tested it through many crawl tools and had no issue with it being crawled. (i.e. Fetch & Render | TechnicalSEO.com, SEMrush, etc.)
  • It's not on a bad server. All of the other websites on the same server are indexed.
  • DNS is functioning properly.
What other possible reason could this website not be indexed?
 
Have you tried inspecting one of the URLs using Google Search Console to determine why Google is telling you the page(s) aren't indexable? I would start there.

They should provide more specifics about the crawl anomaly. If you have some more details come back here but can't help much with what you have provided thus far.
 
I've used it a few times. This is the error message.

Crawl anomaly: An unspecified anomaly occurred when fetching this URL. This could mean a 4xx- or 5xx-level response code; try fetching the page using the URL Inspection tool to see if it encounters any fetch issues. The page was not indexed.

1572536796404.png
 
Strange! I wonder if the server was having issues on 10/29 when Google last crawled the page? Or, do they see those crawl anomalies every time they crawl no matter what?
 
I believe they are having issues every time they crawl it. When it's indexed it's only the homepage and only for a day. It's bizarre.
 
I've used it a few times. This is the error message.

Crawl anomaly: An unspecified anomaly occurred when fetching this URL. This could mean a 4xx- or 5xx-level response code; try fetching the page using the URL Inspection tool to see if it encounters any fetch issues. The page was not indexed.
I'm a bit confused: Is that the error message you see generally or an error message when you try to use the URL Inspection Tool?

Have you tried running it through this utility?

 
What do you have in the robots.txt file?

What do you have in your .htaccess file?
 
I have triple checked it. I see there's a crawl delay. That wasn't there before. Maybe that's the root issue. Going to remove that delay.
 
Last edited:
Please post your robots.txt and htaccess files.
 
You can add website RSS feed in sitemap submission, Text files with links etc too. See if that can help a bit.
 
I believe you don’t have an issue with robots.txt, htaccess as well as meta robots. All look good right?

Can you check your sitemap again sitemap.xml or sitemap_index.xml? Once you got your sitemap indexed your pages start getting indexed in Google.

What is your referring page under your URL Inspection when you retrieve data from Google index?
 
Hi Suresh and Vivekpatel,

Thanks for your input.

Since my last post, I have created a new domain for Google to index. Google has successfully indexed the new domain. Still working to get it the visibility that it needs though.
 
If you would like to PM, you can. :)

Hi Suresh and Vivekpatel,

Thanks for your input.

Since my last post, I have created a new domain for Google to index. Google has successfully indexed the new domain. Still working to get it the visibility that it needs though.
 

Login / Register

Already a member?   LOG IN
Not a member yet?   REGISTER

LocalU Event

Trending: Most Viewed

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...
Top Bottom