Exclude Google bot trafficHi, today we had 350 users which appear to be from Google. I guess they are bots, so is there a way to exclude them from our traffic. Also each visit from Google comes from another IP adres, excluding manually is not an ideal solution.
Posted Sun Nov 7 2010 11:04am by Aldo
Some example data:
7:55pm 220.127.116.11 1 action 10s
7:54pm 18.104.22.168 1 action 10s
7:51pm 22.214.171.124 1 action 10s
7:49pm 126.96.36.199 1 action 10s
7:49pm 188.8.131.52 2 actions5m 35s
7:25pm 184.108.40.206 1 action 10s
7:24pm 220.127.116.11 1 action 10s
7:23pm 18.104.22.168 1 action 10s
7:21pm 22.214.171.124 1 action 10s
7:20pm 126.96.36.199 1 action 10s
7:18pm 188.8.131.52 1 action 10s
7:17pm 184.108.40.206 1 action 10s
7:16pm 220.127.116.11 1 action 10s
6:57pm 18.104.22.168 1 action 10s
6:52pm 22.214.171.124 1 action 10s
6:50pm 126.96.36.199 2 actions55s
6:24pm 188.8.131.52 1 action 10s
6:20pm 184.108.40.206 1 action 10s
Posted Sun Nov 7 2010 11:07am by Aldo
Googlebot rarely shows up in our stats. I think we are actually blocking some of their IPs. But it looks like alll of those are either 66.249.68.* or 66.249.67.* - just add those two IPs to your block list and they'll stop showing up.
Posted Wed Nov 10 2010 3:29pm by Your Friendly Clicky Admin
Just got us a pro account and installed the code.
But as the Google bot visits our websites quite often we had already 813 visits from Google recorded within 20 minutes after placing tracking code online.
I suggest to do this according tho Google own guidelines: http://www.google.com/support/webmasters/bin/answer.py?answer=80553
They always identify as Google.
Could you please filter them out, our stats are useless this way.
When I look for "Visitors who are from Google" under Locale -> Orginizations there are way more ip's then the two ranges listed here above.
Posted Wed Nov 17 2010 6:09am by tixor
See here: 60 IPs that are 100% sure Google bot!
I filtered these unique IPs from the ones that visited one of our site today. Really believe it would be best to filter them out as per Google describes in the link above.
Posted Wed Nov 17 2010 10:35am by tixor
This is apparently from the new "instant preview" feature on google.com. They have a unique user agent string for this and it's already on my todo list to filter this traffic out.
Posted Wed Nov 17 2010 12:33pm by Your Friendly Clicky Admin
NIce work... just out of interest, i told the woopra people about this bot four months ago, and they've done NOTHING to filter it out. Same IP Range. Useless as tits on a bull, bunch of monkeys!
Posted Sat Nov 20 2010 10:05pm by anteck-DELETED
I'm starting get some traffic from the addresses above and I'm not sure why - is the filter still implemented from Nov 2010?
Posted Mon Feb 14 2011 11:15am by mwcstats
Hey Your Friendly Clicky Admin,
It's still f#cking up our result every day. We update our adwords campaigns daily after which Google-bot will come to our site to check these pages for QS etc.
See this traffic spike: http://cl.ly/0F3T0G0c2b1a1A3A0820
That's the google bot that visits our site on monday morning after we updated our adwords campaigns.
You guys have to filter this out, it's making a mess of our statistics.
Posted Tue May 10 2011 8:07am by tixor
Oh and see this url for info about the Google Instantpreview :
When we generate previews on-the-fly based on user requests, we use the "Google Web Preview" user-agent, currently:
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
Posted Tue May 10 2011 8:16am by tixor
+1 for this. My stats are useless. Currently 31% of my visits for the last 7 days are these types of Google visits.
I'd live with a blanket "ignore all Google visitors" option. Thanks.
Posted Tue May 10 2011 11:32am by cpilko
We have been filtering out "Google Web Preview" since my last post. So if they're showing up then they're sending a fake or different user agent.
Posted Tue May 10 2011 5:55pm by Your Friendly Clicky Admin
Here is the last 5000 Google visitors that I've seen.
All but 3 of them report that they are running IE6 on Win XP. I see them no more than once per day in groups of 100+ visits within a 1 hour period, then nothing for a day or more. This seems to be related to Adwords changes more so than web preview.
Posted Wed May 11 2011 1:50pm by cpilko
Ok. I'll see if I can get anything useful from that.
Posted Wed May 11 2011 5:31pm by Your Friendly Clicky Admin
Still not solved ? It has been 6 months.. I'm happy to pay for a service, but please make it work. Right now it's useless.
Take a look in our accounts, and see the traffic spikes in the morning. This is right after we updated our adwords campaigns.
Posted Wed May 18 2011 9:56am by tixor
@tixor: See this new thread I started: '"Spoofed" Google bot visits are from AdWords API calls'.
Are you using the AdWords tab on Salesforce.com? That is what I have traced my traffic spikes to. I've logged a case with SF to get some clarification on this. You should log one too. It looks like they are passing an IE6 user agent string to the Adwords API (which can't be filtered) instead of something unique (that can be).
Posted Thu May 19 2011 7:08am by cpilko
No the problem is known and described before.
After you upload new url's to adwords, google will send out a spider to visit these pages. That's all, getclikcy seems unable to remove these visits from there logs. Which is strange.
@getclikcy you have our emailadress get in touch and let us solve this issue, we have detailed log files on our own servers and can help you get every data you need.
Posted Sat May 21 2011 3:12am by tixor
Your Friendly Clicky Admin not interested in solving this issue at all? Send me an email and we will give you everything you need regarding the bot details.
Posted Wed May 25 2011 12:54am by tixor
We're getting a similar issue too. Most mornings we seem to get a traffic spike from different visitors and the referrer being google.com. What's slightly weird is that next to the referrer entry it has the search term, but then this search term is always one of the 8 or 9 terms that we are trying to optimize for...
Posted Wed May 25 2011 9:08am by davbel
I've added the IPs that you listed way above, and the ones from the CSV that cpilko posted (they were pretty much the same), but I can't upload it until our current code branch is merged.
Posted Wed May 25 2011 11:59am by Your Friendly Clicky Admin
There are a ton more IP's then the one listed above. I've already blocked these all myself.
Posted Fri May 27 2011 12:54am by tixor
Ok, but do you understand how difficult this is, since Google is sending a fake user agent string? And they could always add more IPs in the future. They may constantly switch between various groups of them to make it harder to auto-detect. The user agent is key so them sending a fake one makes this impossible to fix 100%. I'm just saying, because you seem to think we don't care. We do care, but Google is making is hard.
Posted Sat May 28 2011 5:26pm by Your Friendly Clicky Admin
They show up as Google!
See a screenshot with some visits today. I'm happy to help you and solve this problem, just send me an email guys!
Posted Mon May 30 2011 8:25am by tixor
So no one at Google's offices browses the web? We should just automatically ignore all of them?
Posted Mon May 30 2011 9:44am by Your Friendly Clicky Admin
@You're NOT SO Friendly Clicky Admin
How many of the real Google Employees will browse the web with IE 6.0 at 1024x768 ?
That's what all these 'visitors' have in common.
But just tell me if you don't want to change this at all, I'm happy to cancel our paid plan then and stop bothering you with simple - but for you extreme difficult - questions.
Posted Fri Jun 3 2011 10:01am by tixor
no update on this ?
Posted Tue Jun 7 2011 2:20am by tixor
Same here, any updates on weather or not you're going to improve the way you handle bot traffic ?
I see new bots showing up in my reports almost every day, and filtering them all manually is not that convenient.
Posted Fri Jun 10 2011 8:28am by linuxfacile
Working on this change right now. If OS==XP && Browser==IE6 && Res=1024 && Org==Google|Microsoft - no loggy! (Microsoft uses the same faux user agent exactly!)
Should go live today!
Posted Fri Jun 10 2011 4:23pm by Your Friendly Clicky Admin
It's live! Let me know if you're still seeing any of this nonsense.
Posted Fri Jun 10 2011 5:43pm by Your Friendly Clicky Admin
Oh, it's working, 3 servers have already filtered a few hits from bots. Yay :D
Posted Fri Jun 10 2011 5:51pm by Your Friendly Clicky Admin
It worked for a week or two.. but since this week Adwords bots shows up in our stats again..
In the image above you can see the spike, that is 900% sure the adwords bot visiting the urls from campaigns that we did submit.
Posted Wed Jun 29 2011 3:28am by tixor
Operating system: Windows XP
Web browser: Internet Explorer 7.0
It seems they 'upgraded' their bots to IE 7.0 this used to be IE 6.
Posted Wed Jun 29 2011 3:31am by tixor
Yup, spotted that yesterday, uploaded a fix about 8 hours ago.
Posted Wed Jun 29 2011 7:21am by Your Friendly Clicky Admin
I don't understand why you should filter these google out in the first place. I am glad that this still works with the php script.
Posted Tue Jan 24 2012 3:44am by digitalux
What I prefer seeing, which is possibly a much better solution, is adding these bots to a new tab somewhere. I think its very interesting to know about bot behavior/activity on my site.
Posted Tue Jan 24 2012 6:09am by digitalux