SEMRush Bot seen on results page

[Ana]

Hi Guys

Recently, I've a SEMRush BOT appear on my server search results, instantly placed a #SEO block then after a number of days unblocked it. Question is, why are they here?

Thanks

 

[James Watt]

When you say you saw it in the server search results, you mean in your log files on your host? Is your question, just wondering why SEMRush was crawling your site?

There's a huge number of bots out there, apparently over half of all internet traffic is from bots. Most friendly bots crawl your site for the same general reason humans do... to find information. I don't know about SEMRush, but a common reason for bots to crawl might be to index pages, or find the link structure connecting your site with other sites. SEMRush likely isn't hurting anything, but (unless you're using their service) they probably aren't helping anything either. Your data's just being used to help others make informed marketing decisions, if you care enough to block, there's no downside for you at least. Unless you have reason to believe the bots are using up more than their fair share of server load though, there's not really any reason to take the time to block them.

 

[Ana]

Re: SEMRush Bot seen on results LOG files

I think I know very good SEO. I actually taught myself via the white hat method, but have poor CSS, so I won't show my website. Yes in the log files. I did some reading on their

https://www.semrush.com/bot/

and others then decided afterwards to re-block them in my /robots.txt as basically they are criminals pinching other people's hard work for their Adsense benefit. Someone stated a long while ago: Thou shalt not steal.

i believe the reason they are here is because of my high ranking for some top quality keywords i an using.

Thanks

 

[Eric Rohrback]

Re: SEMRush Bot seen on results LOG files

They are visiting your site to document for their tools, not adsense. SEMrush is a reporting software that helps monitor keyword performance for organic ranking and paid search. They are one of the better companies out there to help understand site performance.

I've used both SEMrush and SpyFu, and never had a problem with them. If you block all bots from your site, then hopefully you have developed some internal/proprietary software for your own use. Otherwise you're going to have a tough time understanding the full market landscape.

 

[Ana]

Eric, they are here like you say and because I have mastered (this area) how to fully optimize rank on page 1 of Google and Bing for some seriously juicy keywords and they want all the cake, but I have blocked the Semrushbot and SemRushbot-SA plus a few others just in case.

 

[Eric Rohrback]

@Ana - Nice trolling. We'll consider this a case closed subject since you obviously are set on what you want to do and do not want to engage in any real discussion.

Please contribute in a meaningful way to threads in the future.