Forums » Suggestions & Feature requests



Exclude Google bot traffic

Hi, today we had 350 users which appear to be from Google. I guess they are bots, so is there a way to exclude them from our traffic. Also each visit from Google comes from another IP adres, excluding manually is not an ideal solution.

Posted Sun Nov 7 2010 11:04a by Ald***


Some example data:

7:55pm 66.249.68.217 1 action 10s
7:54pm 66.249.67.197 1 action 10s
7:51pm 66.249.67.236 1 action 10s
7:49pm 66.249.67.214 1 action 10s
7:49pm 66.249.68.130 2 actions5m 35s
7:25pm 66.249.68.183 1 action 10s
7:24pm 66.249.68.202 1 action 10s
7:23pm 66.249.68.234 1 action 10s
7:21pm 66.249.68.184 1 action 10s
7:20pm 66.249.67.130 1 action 10s
7:18pm 66.249.67.196 1 action 10s
7:17pm 66.249.68.129 1 action 10s
7:16pm 66.249.68.219 1 action 10s
6:57pm 66.249.68.236 1 action 10s
6:52pm 66.249.68.219 1 action 10s
6:50pm 66.249.67.229 2 actions55s
6:24pm 66.249.68.218 1 action 10s
6:20pm 66.249.68.234 1 action 10s

Posted Sun Nov 7 2010 11:07a by Ald***


Googlebot rarely shows up in our stats. I think we are actually blocking some of their IPs. But it looks like alll of those are either 66.249.68.* or 66.249.67.* - just add those two IPs to your block list and they'll stop showing up.

Posted Wed Nov 10 2010 3:29p by Your Friendly Clicky Admin


Just got us a pro account and installed the code.
But as the Google bot visits our websites quite often we had already 813 visits from Google recorded within 20 minutes after placing tracking code online.

I suggest to do this according tho Google own guidelines: http://www.google.com/support/webmasters/bin/answer.py?answer=80553

They always identify as Google.
Could you please filter them out, our stats are useless this way.

When I look for "Visitors who are from Google" under Locale -> Orginizations there are way more ip's then the two ranges listed here above.

Posted Wed Nov 17 2010 6:09a by tixo***


See here: 60 IPs that are 100% sure Google bot!

I filtered these unique IPs from the ones that visited one of our site today. Really believe it would be best to filter them out as per Google describes in the link above.

209.85.226.85
209.85.228.80
209.85.228.81
209.85.228.82
209.85.228.84
209.85.228.85
64.233.172.20
66.249.82.1
66.249.85.65
72.14.192.65
72.14.212.80
72.14.212.81
72.14.212.82
72.14.212.83
72.14.212.84
72.14.212.86
74.125.112.82
74.125.112.83
74.125.112.85
74.125.126.81
74.125.154.80
74.125.154.81
74.125.154.82
74.125.154.83
74.125.154.84
74.125.154.85
74.125.154.86
74.125.156.80
74.125.156.81
74.125.156.82
74.125.158.80
74.125.158.81
74.125.158.82
74.125.44.80
74.125.44.81
74.125.44.82
74.125.44.83
74.125.44.84
74.125.44.85
74.125.44.86
74.125.44.87
74.125.46.80
74.125.46.81
74.125.46.82
74.125.46.83
74.125.46.84
74.125.46.85
74.125.46.87
74.125.64.80
74.125.64.81
74.125.64.82
74.125.64.83
74.125.64.84
74.125.64.85
74.125.64.86
74.125.64.87
74.125.66.80
74.125.66.81
74.125.66.82
74.125.75.19

Posted Wed Nov 17 2010 10:35a by tixo***


This is apparently from the new "instant preview" feature on google.com. They have a unique user agent string for this and it's already on my todo list to filter this traffic out.

Posted Wed Nov 17 2010 12:33p by Your Friendly Clicky Admin


NIce work... just out of interest, i told the woopra people about this bot four months ago, and they've done NOTHING to filter it out. Same IP Range. Useless as tits on a bull, bunch of monkeys!

Posted Sat Nov 20 2010 10:05p by antec***


I'm starting get some traffic from the addresses above and I'm not sure why - is the filter still implemented from Nov 2010?

Posted Mon Feb 14 2011 11:15a by mwcsta***


Hey Your Friendly Clicky Admin,

It's still f#cking up our result every day. We update our adwords campaigns daily after which Google-bot will come to our site to check these pages for QS etc.

See this traffic spike: http://cl.ly/0F3T0G0c2b1a1A3A0820
That's the google bot that visits our site on monday morning after we updated our adwords campaigns.

You guys have to filter this out, it's making a mess of our statistics.

Posted Tue May 10 2011 8:07a by tixo***


Oh and see this url for info about the Google Instantpreview :
http://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews

When we generate previews on-the-fly based on user requests, we use the "Google Web Preview" user-agent, currently:
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

Posted Tue May 10 2011 8:16a by tixo***


+1 for this. My stats are useless. Currently 31% of my visits for the last 7 days are these types of Google visits.

I'd live with a blanket "ignore all Google visitors" option. Thanks.

Posted Tue May 10 2011 11:32a by cpilk***


We have been filtering out "Google Web Preview" since my last post. So if they're showing up then they're sending a fake or different user agent.

Posted Tue May 10 2011 5:55p by Your Friendly Clicky Admin


Here is the last 5000 Google visitors that I've seen.
http://saunascape.com/saunascape.com--visitors-list--2011-04-12,2011-05-11--1.csv

All but 3 of them report that they are running IE6 on Win XP. I see them no more than once per day in groups of 100+ visits within a 1 hour period, then nothing for a day or more. This seems to be related to Adwords changes more so than web preview.

Thanks.

Posted Wed May 11 2011 1:50p by cpilk***


Ok. I'll see if I can get anything useful from that.

Posted Wed May 11 2011 5:31p by Your Friendly Clicky Admin


Still not solved ? It has been 6 months.. I'm happy to pay for a service, but please make it work. Right now it's useless.
Take a look in our accounts, and see the traffic spikes in the morning. This is right after we updated our adwords campaigns.

Posted Wed May 18 2011 9:56a by tixo***


@tixor: See this new thread I started: '"Spoofed" Google bot visits are from AdWords API calls'.

Are you using the AdWords tab on Salesforce.com? That is what I have traced my traffic spikes to. I've logged a case with SF to get some clarification on this. You should log one too. It looks like they are passing an IE6 user agent string to the Adwords API (which can't be filtered) instead of something unique (that can be).

Posted Thu May 19 2011 7:08a by cpilk***


@cpilko

No the problem is known and described before.

After you upload new url's to adwords, google will send out a spider to visit these pages. That's all, getclikcy seems unable to remove these visits from there logs. Which is strange.

@getclikcy you have our emailadress get in touch and let us solve this issue, we have detailed log files on our own servers and can help you get every data you need.

Posted Sat May 21 2011 3:12a by tixo***


Your Friendly Clicky Admin not interested in solving this issue at all? Send me an email and we will give you everything you need regarding the bot details.

Posted Wed May 25 2011 12:54a by tixo***


Hi All

We're getting a similar issue too. Most mornings we seem to get a traffic spike from different visitors and the referrer being google.com. What's slightly weird is that next to the referrer entry it has the search term, but then this search term is always one of the 8 or 9 terms that we are trying to optimize for...

Posted Wed May 25 2011 9:08a by davbe***


I've added the IPs that you listed way above, and the ones from the CSV that cpilko posted (they were pretty much the same), but I can't upload it until our current code branch is merged.

Posted Wed May 25 2011 11:59a by Your Friendly Clicky Admin


There are a ton more IP's then the one listed above. I've already blocked these all myself.

Posted Fri May 27 2011 12:54a by tixo***


Ok, but do you understand how difficult this is, since Google is sending a fake user agent string? And they could always add more IPs in the future. They may constantly switch between various groups of them to make it harder to auto-detect. The user agent is key so them sending a fake one makes this impossible to fix 100%. I'm just saying, because you seem to think we don't care. We do care, but Google is making is hard.

Posted Sat May 28 2011 5:26p by Your Friendly Clicky Admin


They show up as Google!

See a screenshot with some visits today. I'm happy to help you and solve this problem, just send me an email guys!
http://cl.ly/2J3p181m0F1Y0d1j1l2v

Posted Mon May 30 2011 8:25a by tixo***


So no one at Google's offices browses the web? We should just automatically ignore all of them?

Posted Mon May 30 2011 9:44a by Your Friendly Clicky Admin


@You're NOT SO Friendly Clicky Admin

How many of the real Google Employees will browse the web with IE 6.0 at 1024x768 ?
That's what all these 'visitors' have in common.

But just tell me if you don't want to change this at all, I'm happy to cancel our paid plan then and stop bothering you with simple - but for you extreme difficult - questions.

Posted Fri Jun 3 2011 10:01a by tixo***


no update on this ?

Posted Tue Jun 7 2011 2:20a by tixo***


Same here, any updates on weather or not you're going to improve the way you handle bot traffic ?

I see new bots showing up in my reports almost every day, and filtering them all manually is not that convenient.

Posted Fri Jun 10 2011 8:28a by linuxfac***


Working on this change right now. If OS==XP && Browser==IE6 && Res=1024 && Org==Google|Microsoft - no loggy! (Microsoft uses the same faux user agent exactly!)

Should go live today!

Posted Fri Jun 10 2011 4:23p by Your Friendly Clicky Admin


It's live! Let me know if you're still seeing any of this nonsense.

Posted Fri Jun 10 2011 5:43p by Your Friendly Clicky Admin


Oh, it's working, 3 servers have already filtered a few hits from bots. Yay :D

Posted Fri Jun 10 2011 5:51p by Your Friendly Clicky Admin


It worked for a week or two.. but since this week Adwords bots shows up in our stats again..

http://cl.ly/2P1S3s2x3H40281B0X3K

In the image above you can see the spike, that is 900% sure the adwords bot visiting the urls from campaigns that we did submit.

Posted Wed Jun 29 2011 3:28a by tixo***


Operating system: Windows XP
Web browser: Internet Explorer 7.0
Resolution: 1024x768
Javascript: Enabled

It seems they 'upgraded' their bots to IE 7.0 this used to be IE 6.

Posted Wed Jun 29 2011 3:31a by tixo***


Yup, spotted that yesterday, uploaded a fix about 8 hours ago.

Posted Wed Jun 29 2011 7:21a by Your Friendly Clicky Admin


I don't understand why you should filter these google out in the first place. I am glad that this still works with the php script.

Posted Tue Jan 24 2012 3:44a by digital***


What I prefer seeing, which is possibly a much better solution, is adding these bots to a new tab somewhere. I think its very interesting to know about bot behavior/activity on my site.

Posted Tue Jan 24 2012 6:09a by digital***


I only have a few sites and they're all small personal sites but one of them is my professional services page and I get about three hits from Google a day, via my AdWords ad for the page, which haven't changed in over a month. Not only does Clicky record the visit, but it also registers it as via an ad.

Don't get me started on Clicky's attitude to displaying a visit as having come from an ad (they do it even if the visitor clicked on an ad once, three months ago, which is utterly useless to me as a stat and makes me think my ads are bringing more people to my site than they actually are, but anyway...) - the point here is it's not a valid visit and it's not an ad-click.

So how do I filter out all these Google visits?

Posted Wed Jun 18 2014 8:31a by owen***


One of the things I like about Clicky is I can spot when Mountain View visits and thus check my sites are being crawled as often as I'd like. Can you let me know if you're filtering and if yes, whether it's possible not to!?

Posted Mon Jul 21 2014 9:05a by Gid***


I have mostly China and Russian IPs 10s views one page

Posted Mon Dec 8 2014 7:37a by adami***


Im new to Clicky Analytics I test the free premium monthly membership, I was blown away of the simplicity and I preferred this over Google Analytics, but now all I get is 10 second visits from Google, I almost thought something is causing bounce rate, after searching on Google I found this thread, so did you find any solution for this? Please help, it doesn't make sense to check analytics after these Google Bots.

Posted Mon Jan 9 2017 1:58a by Hilali1***


You must be logged in to your account to post!