Exclude Google bot traffic

Hi, today we had 350 users which appear to be from Google. I guess they are bots, so is there a way to exclude them from our traffic. Also each visit from Google comes from another IP adres, excluding manually is not an ideal solution.

Some example data:

7:55pm 66.249.68.217 1 action 10s
7:54pm 66.249.67.197 1 action 10s
7:51pm 66.249.67.236 1 action 10s
7:49pm 66.249.67.214 1 action 10s
7:49pm 66.249.68.130 2 actions5m 35s
7:25pm 66.249.68.183 1 action 10s
7:24pm 66.249.68.202 1 action 10s
7:23pm 66.249.68.234 1 action 10s
7:21pm 66.249.68.184 1 action 10s
7:20pm 66.249.67.130 1 action 10s
7:18pm 66.249.67.196 1 action 10s
7:17pm 66.249.68.129 1 action 10s
7:16pm 66.249.68.219 1 action 10s
6:57pm 66.249.68.236 1 action 10s
6:52pm 66.249.68.219 1 action 10s
6:50pm 66.249.67.229 2 actions55s
6:24pm 66.249.68.218 1 action 10s
6:20pm 66.249.68.234 1 action 10s

Googlebot rarely shows up in our stats. I think we are actually blocking some of their IPs. But it looks like alll of those are either 66.249.68.* or 66.249.67.* - just add those two IPs to your block list and they'll stop showing up.

Just got us a pro account and installed the code.
But as the Google bot visits our websites quite often we had already 813 visits from Google recorded within 20 minutes after placing tracking code online.

I suggest to do this according tho Google own guidelines: http://www.google.com/support/webmasters/bin/answer.py?answer=80553

They always identify as Google.
Could you please filter them out, our stats are useless this way.

When I look for "Visitors who are from Google" under Locale -> Orginizations there are way more ip's then the two ranges listed here above.

See here: 60 IPs that are 100% sure Google bot!

I filtered these unique IPs from the ones that visited one of our site today. Really believe it would be best to filter them out as per Google describes in the link above.

209.85.226.85
209.85.228.80
209.85.228.81
209.85.228.82
209.85.228.84
209.85.228.85
64.233.172.20
66.249.82.1
66.249.85.65
72.14.192.65
72.14.212.80
72.14.212.81
72.14.212.82
72.14.212.83
72.14.212.84
72.14.212.86
74.125.112.82
74.125.112.83
74.125.112.85
74.125.126.81
74.125.154.80
74.125.154.81
74.125.154.82
74.125.154.83
74.125.154.84
74.125.154.85
74.125.154.86
74.125.156.80
74.125.156.81
74.125.156.82
74.125.158.80
74.125.158.81
74.125.158.82
74.125.44.80
74.125.44.81
74.125.44.82
74.125.44.83
74.125.44.84
74.125.44.85
74.125.44.86
74.125.44.87
74.125.46.80
74.125.46.81
74.125.46.82
74.125.46.83
74.125.46.84
74.125.46.85
74.125.46.87
74.125.64.80
74.125.64.81
74.125.64.82
74.125.64.83
74.125.64.84
74.125.64.85
74.125.64.86
74.125.64.87
74.125.66.80
74.125.66.81
74.125.66.82
74.125.75.19

This is apparently from the new "instant preview" feature on google.com. They have a unique user agent string for this and it's already on my todo list to filter this traffic out.

NIce work... just out of interest, i told the woopra people about this bot four months ago, and they've done NOTHING to filter it out. Same IP Range. Useless as tits on a bull, bunch of monkeys!

I'm starting get some traffic from the addresses above and I'm not sure why - is the filter still implemented from Nov 2010?

Hey Your Friendly Clicky Admin,

It's still f#cking up our result every day. We update our adwords campaigns daily after which Google-bot will come to our site to check these pages for QS etc.

See this traffic spike: http://cl.ly/0F3T0G0c2b1a1A3A0820
That's the google bot that visits our site on monday morning after we updated our adwords campaigns.

You guys have to filter this out, it's making a mess of our statistics.

Oh and see this url for info about the Google Instantpreview :
http://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews

When we generate previews on-the-fly based on user requests, we use the "Google Web Preview" user-agent, currently:
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

+1 for this. My stats are useless. Currently 31% of my visits for the last 7 days are these types of Google visits.

I'd live with a blanket "ignore all Google visitors" option. Thanks.

We have been filtering out "Google Web Preview" since my last post. So if they're showing up then they're sending a fake or different user agent.

Here is the last 5000 Google visitors that I've seen.
http://saunascape.com/saunascape.com--visitors-list--2011-04-12,2011-05-11--1.csv

All but 3 of them report that they are running IE6 on Win XP. I see them no more than once per day in groups of 100+ visits within a 1 hour period, then nothing for a day or more. This seems to be related to Adwords changes more so than web preview.

Thanks.

Ok. I'll see if I can get anything useful from that.

Still not solved ? It has been 6 months.. I'm happy to pay for a service, but please make it work. Right now it's useless.
Take a look in our accounts, and see the traffic spikes in the morning. This is right after we updated our adwords campaigns.

@tixor: See this new thread I started: '"Spoofed" Google bot visits are from AdWords API calls'.

Are you using the AdWords tab on Salesforce.com? That is what I have traced my traffic spikes to. I've logged a case with SF to get some clarification on this. You should log one too. It looks like they are passing an IE6 user agent string to the Adwords API (which can't be filtered) instead of something unique (that can be).

@cpilko

No the problem is known and described before.

After you upload new url's to adwords, google will send out a spider to visit these pages. That's all, getclikcy seems unable to remove these visits from there logs. Which is strange.

@getclikcy you have our emailadress get in touch and let us solve this issue, we have detailed log files on our own servers and can help you get every data you need.

Your Friendly Clicky Admin not interested in solving this issue at all? Send me an email and we will give you everything you need regarding the bot details.

Hi All

We're getting a similar issue too. Most mornings we seem to get a traffic spike from different visitors and the referrer being google.com. What's slightly weird is that next to the referrer entry it has the search term, but then this search term is always one of the 8 or 9 terms that we are trying to optimize for...

I've added the IPs that you listed way above, and the ones from the CSV that cpilko posted (they were pretty much the same), but I can't upload it until our current code branch is merged.

There are a ton more IP's then the one listed above. I've already blocked these all myself.

Ok, but do you understand how difficult this is, since Google is sending a fake user agent string? And they could always add more IPs in the future. They may constantly switch between various groups of them to make it harder to auto-detect. The user agent is key so them sending a fake one makes this impossible to fix 100%. I'm just saying, because you seem to think we don't care. We do care, but Google is making is hard.

They show up as Google!

See a screenshot with some visits today. I'm happy to help you and solve this problem, just send me an email guys!
http://cl.ly/2J3p181m0F1Y0d1j1l2v

So no one at Google's offices browses the web? We should just automatically ignore all of them?

@You're NOT SO Friendly Clicky Admin

How many of the real Google Employees will browse the web with IE 6.0 at 1024x768 ?
That's what all these 'visitors' have in common.

But just tell me if you don't want to change this at all, I'm happy to cancel our paid plan then and stop bothering you with simple - but for you extreme difficult - questions.

no update on this ?

Same here, any updates on weather or not you're going to improve the way you handle bot traffic ?

I see new bots showing up in my reports almost every day, and filtering them all manually is not that convenient.

Working on this change right now. If OS==XP && Browser==IE6 && Res=1024 && Org==Google|Microsoft - no loggy! (Microsoft uses the same faux user agent exactly!)

Should go live today!

It's live! Let me know if you're still seeing any of this nonsense.

Oh, it's working, 3 servers have already filtered a few hits from bots. Yay :D

It worked for a week or two.. but since this week Adwords bots shows up in our stats again..

http://cl.ly/2P1S3s2x3H40281B0X3K

In the image above you can see the spike, that is 900% sure the adwords bot visiting the urls from campaigns that we did submit.

Operating system: Windows XP
Web browser: Internet Explorer 7.0
Resolution: 1024x768
Javascript: Enabled

It seems they 'upgraded' their bots to IE 7.0 this used to be IE 6.

Yup, spotted that yesterday, uploaded a fix about 8 hours ago.

I don't understand why you should filter these google out in the first place. I am glad that this still works with the php script.

What I prefer seeing, which is possibly a much better solution, is adding these bots to a new tab somewhere. I think its very interesting to know about bot behavior/activity on my site.

I only have a few sites and they're all small personal sites but one of them is my professional services page and I get about three hits from Google a day, via my AdWords ad for the page, which haven't changed in over a month. Not only does Clicky record the visit, but it also registers it as via an ad.

Don't get me started on Clicky's attitude to displaying a visit as having come from an ad (they do it even if the visitor clicked on an ad once, three months ago, which is utterly useless to me as a stat and makes me think my ads are bringing more people to my site than they actually are, but anyway...) - the point here is it's not a valid visit and it's not an ad-click.

So how do I filter out all these Google visits?

One of the things I like about Clicky is I can spot when Mountain View visits and thus check my sites are being crawled as often as I'd like. Can you let me know if you're filtering and if yes, whether it's possible not to!?

I have mostly China and Russian IPs 10s views one page

Im new to Clicky Analytics I test the free premium monthly membership, I was blown away of the simplicity and I preferred this over Google Analytics, but now all I get is 10 second visits from Google, I almost thought something is causing bounce rate, after searching on Google I found this thread, so did you find any solution for this? Please help, it doesn't make sense to check analytics after these Google Bots.