Google Crawl Challenges: Seeking Advice for Overcoming Persistent Neglect Despite '200 OK' Response

[ganapathy]

Initially, Google crawled the page and encountered a 'Not Found' status on October 6, 2023, with the Desktop bot, and on October 13, 2023, with the Smartphone bot. However, the page is currently returning a '200 OK' response, yet Google appears to be neglecting further crawls.

The initial '404' error was attributed to a server stability issue on our website during website deployments. However, after reviewing the server log file, no issues are currently detected.

We manually requested indexing, ensured the URL is present in the XML sitemap, provided internal links to the page, created few external links (external links are indexed), and updated the content.

Question: What advice do you have for prompting Google to recrawl the particular page?
Please let me know if you need any additional information.

 

[JeffClevelandTN]

We've encountered this only once in the past, where Google simply would not recrawl/index a previously problematic page. We wound up copying the content to a new page, redirecting the old page to the new page, changing all internal links to the new page. We then manually submitted a recrawl and everything was good. Being that the page is not indexed, you really have nothing to lose at this point by republishing the page.

 

[ganapathy]

We've encountered this only once in the past, where Google simply would not recrawl/index a previously problematic page. We wound up copying the content to a new page, redirecting the old page to the new page, changing all internal links to the new page. We then manually submitted a recrawl and everything was good. Being that the page is not indexed, you really have nothing to lose at this point by republishing the page.

Thanks for sharing your experience, @JeffClevelandTN .
I have already recommended this practice—creating a new page and redirecting the old page—to our development team. We have kept it as the final option.
I will update you on the status.

 

[ganapathy]

@JeffClevelandTN Here's the update: We successfully utilized the Google Index API with the assistance of our Python developer to perform a fresh crawl. This allowed us to index previously unindexed pages, and the process was a success.

 

[JeffClevelandTN]

@JeffClevelandTN Here's the update: We successfully utilized the Google Index API with the assistance of our Python developer to perform a fresh crawl. This allowed us to index previously unindexed pages, and the process was a success.

Thanks for giving an update! Just an FYI, RankMath provides built-in functionality to request fast indexing via the Google API too. Glad you got your problem resolved!