j_holtslander

Member
Joined
Feb 5, 2019
Messages
133
Reaction score
110
Background
I have a server that runs many staging sites for many client websites as individual subdomains. eg: client1.stagingserver.com, client2.stagingserver.com, client3.stagingserver.com, etc.

Each site contains both the production server's robots.txt file as well as a special robots-staging.txt file that is purposed with preventing the staging sites from being indexed.

The way it works is that if the domain that the website is running on is a subdomain of [stagingserver.com] then the robots-staging.txt file is "substituted" for robots.txt by a mod_rewrite rule for Apache in the site's .htaccess file.
More info.

So (in theory) if a crawler finds [client1.stagingserver.com] and tries to read robots.txt they'll be served the robots-staging.txt file instead which reads:
Code:
User-agent: *
Disallow: /
Noindex: /

But if a crawler finds the exact same website running on [client1.com] it gets the robots.txt file as normal which reads as:
Code:
User-agent: *
Disallow: /.git
Disallow: /cgi-bin
Disallow: /config

Sitemap: https://[client1.com]/sitemap.xml

The Problem
Google seems to somehow be ignoring the rewrite rule.

Humans are definitely getting the robots-staging.txt file's contents when requesting robots.txt from staging, but several client websites have had their staging sites indexed by Google anyway and are now duplicate content. (Canonicals for the URLs are defined by the Wordpress site's WP_SITEURL)

Anyone have any ideas why Google would be indexing these sites? Or a solution?

Yes, we could instead set the staging sites to "Do not index" within Wordpress' general settings for the staging site. But that setting could easily be accidentally cloned from Staging to Production which would be pretty disastrous.
We need a "set it and forget it" solution that's immune to memory lapses which is why we've gone this auto-substitution route.
 

Yan Gilbert

Administrator
Moderator
Local Search Expert
LocalU Faculty
Joined
Oct 15, 2016
Messages
452
Solutions
1
Reaction score
520
Can you not just put the robots.txt file in the staging directory directly? I assume then you run into the same issue about having to update the file when you push the staging to live.

Not sure of the answer, but does it take into account any www or non-www versions?

Does this page help: Correctly redirect bot requests to static version of a website
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,950
Reaction score
579
1) Google seems to deindex slowly. Did you accidentally allow them to be indexed and you're now waiting for them to drop from the index?

2) Have you tried using a header for no index?
 

j_holtslander

Member
Joined
Feb 5, 2019
Messages
133
Reaction score
110
does it take into account any www or non-www versions?

Yeah, no issues there.


Nope. Afraid not.

Did you accidentally allow them to be indexed and you're now waiting for them to drop from the index?

Unclear TBH, but yes currently waiting for them to be dropped after requesting removals.

2) Have you tried using a header for no index?

Unsure how to do that where it'd apply only to Staging and not to Production. Ideas?
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,950
Reaction score
579
That may be the problem. Maybe Google is honoring the noindex, only it just now showed up. So they're dropping it from the index. I can tell you from experience that takes a long time sometimes. Months.

Yeah, I'm not sure how to do it either. If it's in a subdirectory you can do it. I did it awhile back but now I've forgotten how.
 
Similar threads
Thread starter Title Forum Replies Date
Linda Buquet Technical Local SEO: Beyond Schema - Semantic Search Strategy Local Search 3
djbaxter Webinar Technical SEO Help Hour Oct. 28 Events 0
JoyHawkins [Technical Issue] Don't Change the Address on Service Area Business Listings Google News: Important Changes & Features 5
hajnasiewicz Non Technical info re: Mobile First Indexing Marketing 2
JoyHawkins Technical Issues Impacting Review Replies Local Reviews 2
JoyHawkins Known Technical Issue Impacting GMB Photos Google My Business & Google Maps 24
KevinWag Is this technically against Google's guidelines? Local Search 7
C Citation Clean Up - How to Overcome Technical Issues? Citations 14
Dustybones White Board Friday: What You Need to Know About Technical SEO Organic SEO 1
Dustybones Understanding the technical side of Google Panda Local Search 2
Linda Buquet Amazing List of Schema, Microdata & Technical SEO Analysis Tools Local SEO Tools & Software 0
W Local Search Ecosystem Diagram Local SEO Tools & Software 1
M Google Search Console Errors "Coverage and Breadcrumbs"? Google My Business & Google Maps 4
D Knowledge Panel for Competitor Showed Up for Branded Search Google Duplicates & Merges 6
djbaxter Brightlocal Tool shows Large Google Local Search Algorithm Update March 10, 2021 Google Algorithm Discussions 1
Matt Chauhan Knowledge Panel Not Appearing for Brand Name Search Google My Business & Google Maps 5
djbaxter Webinar Get together with Google Search at Search Central Live - February 23, 2021 Events 0
djbaxter Passage based rankings, core web vitals, responsive search ads: A rant Google Algorithm Discussions 0
djbaxter New Local SEO tool from BrightLocal: Local Search Grid Local SEO Tools & Software 7
djbaxter Microsoft Bing: 15% Of Searches Are Misspelled So Speller100 To The Rescue Organic SEO 1
Lluc Google launches Years in Business in Search Google My Business & Google Maps 3
djbaxter Enhanced Search added. Forum Software Updates and Technical Issues 0
B GMB listing missing from search, dashboard and GSC. Yikes!!! Help & Support for Google Local 3
Elmer .S What is the most accurate search strings to use when doing manual citation audit? Citations 7
J Wrong GMB Listing Appearing when Searching for Physician by Name Google My Business & Google Maps 4
N Dedicated local search Local Search 12
sixthcitysarah Specific location search shows results for other franchise location Multi-Location Issues 6
djbaxter Webinar The State of Local Search 2021 [Webinar + Q&A] Events 0
Dino Closed business showing in branded search Multi-Location Issues 1
SpectateSwamp How about discussing local search... Like the Spectate one; everything a 50+ years of programming Geezer wants Break Room: Chat and Off Topic 3
djbaxter Google services down Dec 14, 2020: Google Ads, Google Analytics, Search Console, more Google News: Important Changes & Features 1
A How to change GMB ticket field in search results knowledge panel NAP, Categories, Dashboard Data 4
Eoghan_MFeed Looking for a SaaS but more importantly Local Search experienced product managers 🔍 Local SEO Job Board 0
Y Radius of searches Service Area Businesses 4
pbarnhart Google Search Console Merging UTM'd URLs into Canonical URLs - Suggestions? Local SEO Tools & Software 5
djbaxter Google Search Console: New and improved crawl stats for your site Organic SEO 0
JoyHawkins Search Console vs Google My Business (GMB) Insights: A Large Case Study Google My Business & Google Maps 4
G Registering a DBA name as a Search Term Spam on Google 5
RattlingTram Wild rank variations for plural v singular & same intent single keyword searches Ranking Puzzles 2
djbaxter A (home) movie about how Google Search works Organic SEO 0
U Alternative to Search Console URL Inspection Fetch Organic SEO 5
Brian Barwig How to Win at Local Search with Service Area Businesses Service Area Businesses 3
Jeffrey GMB Listing Not Showing For Direct Searches Google My Business & Google Maps 7
Barb Davids Google Search Console Bug? Local SEO Tools & Software 2
djbaxter Google Updates Search Quality Raters Guidelines Google News: Important Changes & Features 0
Creativedge Question on duplicate content and search ranking for dentists Local Search 13
D Dynamic local city webpage to satisfy [search term] + "near me" queries Service Area Businesses 2
Tony Wang Search Terms is fading away :-( Paid Search and Local Service Ads 3
djbaxter A bit of a new look for the Local Search Forum Forum Software Updates and Technical Issues 2
wesduplantier Favicons in Maps search menu? Google News: Important Changes & Features 5

Similar threads

Login / Register

Already a member?   LOG IN
Not a member yet?   REGISTER

Most UpVoted Answers

Trending: Most Viewed

LocalU Podcasts

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...
Google Product Exert


Top Bottom