DanLeibson

LocalU Faculty
Joined
May 17, 2013
Messages
229
Reaction score
142
While robots.txt may allow crawling, it's only half of the equation. The other half is whether Google can actually crawl the data or not.

A lot of data we see as the end user is query based and have temporary URL's that can't be replicated by Google. Google needs a permanent or "hard" URL to crawl. That's why you rarely see a URL from a Google SERP with programming syntax (might be wrong vernacular here) with "?", "&" etc. Those seem to be a dead giveaway that the URL you're looking at is query generated and therefore, "temporary" for lack of a better term.

Just FYI, URL parameters get crawled and indexed all the time. They are a huge problem for site performance, because of all the content duplication that they can cause, and part of the reason Google released rel="canonical" and gave webmasters the ability to set URL parameters in Google Search Console. But Google absolutely can and does crawl and index them, often prolifically.
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,961
Reaction score
584
Just FYI, URL parameters get crawled and indexed all the time. They are a huge problem for site performance, because of all the content duplication that they can cause, and part of the reason Google released rel="canonical" and gave webmasters the ability to set URL parameters in Google Search Console.

URL parameters, thanks Dan. That's what I was looking for :)

While they may get crawled and indexed, I personally would not say it's often. We typically see when building citations that many URL's with parameters do not get indexed. In fact, calling it 50/50 would be generous from my experience I think. Again, just my experience. And the ones that do get indexed only have maybe 1-2 parameters. Most government websites I've dealt with have many, many parameters.

Just food for thought.
 

DanLeibson

LocalU Faculty
Joined
May 17, 2013
Messages
229
Reaction score
142
URL parameters, thanks Dan. That's what I was looking for :)

While they may get crawled and indexed, I personally would not say it's often. We typically see when building citations that many URL's with parameters do not get indexed. In fact, calling it 50/50 would be generous from my experience I think. Again, just my experience. And the ones that do get indexed only have maybe 1-2 parameters. Most government websites I've dealt with have many, many parameters.

Just food for thought.

We work with a lot of large IYPs and URL parameters are specifically a huge problem for them ;) I have seen strings of several parameters where different orders of the same parameters were indexed for multiple URLs. It's not as uncommon as you think.
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,961
Reaction score
584
We work with a lot of large IYPs and URL parameters are specifically a huge problem for them ;) I have seen strings of several parameters where different orders of the same parameters were indexed for multiple URLs. It's not as uncommon as you think.

We do a ton of indexing work specifically with citations and the only issues we have pertaining to indexing are multiple URL parameters. They are a huge pain.

We may have to agree to disagree on this one :)
 

DanLeibson

LocalU Faculty
Joined
May 17, 2013
Messages
229
Reaction score
142
We do a ton of indexing work specifically with citations and the only issues we have pertaining to indexing are multiple URL parameters. They are a huge pain.

We may have to agree to disagree on this one :)

Fair enough, but the issue does seem pretty settled. That being said, a low quality site with lots of pages isn't going to have the crawl budget for Google to try to figure out all their URL parameters.
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,961
Reaction score
584
Fair enough, but the issue does seem pretty settled. That being said, a low quality site with lots of pages isn't going to have the crawl budget for Google to try to figure out all their URL parameters.

I had not thought about that. Very interesting point. Thanks for bringing it to my attention.

For us though, we're not waiting for Google to crawl the site and pick it up on their own. We're doing direct indexing for a specific URL and Google can't pick it up.

But that is something I will definitely keep in mind in the future.
 

JoyHawkins

Administrator
Administrator
Moderator
LocalU Faculty
Joined
Jul 18, 2012
Messages
4,437
Solutions
10
Reaction score
3,868
Okay here is the "official" response from Google: Most of our data comes from third party feeds and other licencors, which could include data from government sources.
 

JoshuaMackens

Local Search Expert
Joined
Sep 12, 2012
Messages
1,961
Reaction score
584
Okay here is the "official" response from Google: Most of our data comes from third party feeds and other licencors, which could include data from government sources.

What an awesome, vague answer ;) shouldn't expect anything less haha.

I think it makes sense for them to use government sources to grab info for their database but I doubt it helps ranking at all.

Now, a government website citation on the other hand...well, it definitely can't hurt.
 

Login / Register

Already a member?   LOG IN
Not a member yet?   REGISTER

LocalU

LocalU Nov2021

Most UpVoted Answers

Trending: Most Viewed

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...
Google Product Exert


Top Bottom