Google's new Ajax-powered search results breaks search keyword tracking for everyone

Do you run a web site? Do you use a service such as Clicky, Statcounter, AWStats, etc, to monitor the traffic to your web site? Do you find it useful how these services can tell you what search terms are leading visitors to your site? Have you ever wondered exactly how they are able to do that?

No, you probably haven't. But that's ok. Just know this: a major update that Google is testing has completely broken the ability for any external analytics service like Clicky to determine the search query used by a visitor arriving at your web site. Why would they do such a thing? Who knows. They aren't talking.

Knowing the search terms that are driving traffic to your web site is probably the #1 reason to use any kind of traffic monitoring service. And Google owns a huge chunk of the search market. If this update goes live for everyone, it effectively means that 2/3 of all searches leading to the average web site will be a complete mystery. This is huge.

So what is the problem exactly? Normally when do you a search on Google or any other search engine, the search term used become part of the URL. A search for Clicky, for example, gives you this:

http://www.google.com/search?q=clicky

When someone clicks a search result on that page, that URL above is sent as the "referrer" to the target site. An analytics app running on the target site can parse the referrer string and extract the word "clicky", and store that as a search that occured for that site. This is obviously very useful.

Here's what the new search result URLs look like with the new "Ajax" feature:

http://www.google.com/#q=clicky

See how there's a hash mark # in there now, and the "q=test" is after it? The problem is that web browsers don't send anything after the # in the referrer string. This means organic searches from Google will now show up as just "http://www.google.com/", with no search parameters. In other words, no analytics app can track these searches anymore. I started noticing lots of hits from just "http://www.google.com/" recently in our own search logs. I thought maybe it was just a bug with Clicky. But then one of our users contacted me about this article, and my jaw about broke from hitting the floor so hard.

This change isn't live for everyone yet so if you go to google.com and do a search, you may not see it in action. But it is happening for me when I go there and do a search.

So what can we do about it? If you run a blog, write about this. Submit this story or your own story to large tech blogs like TechCrunch, CenterNetworks, ReadWriteWeb, GigaOm, etc - no large site has written about this yet, and one of them needs to. Post in Google's Web Search forums (there's no way to directly contact them about web search unfortunately). Do anything you can to spread the word and let Google know that this is not acceptable.
80 comments |   Feb 03 2009 9:50am

Clicky roadmap through March 2009

We've got a lot of buns in the oven right now, many of which are waiting for that database transform to be completed. All database servers are done, except for the one, the only, db1, which is lagging badly at barely 33% done. It should be done by this weekend though. (If you have a site on db1 and are tired of it being laggy, read this).

It's been fairly quiet around here recently, but we have in fact been working like mad behind the scenes, as usual. So I wanted to share with you the features we have planned for the next couple of months, all of which we plan to have out by the end of March 2009.

  • In depth per page analysis, e.g. for your "/contact" page, view the top countries, top searches leading to it, top referrers to it, percent of views that were landing/exit (meaning first or last action of a session), average time spent on page - ETC!
  • Filter visitors by number of actions, including support for less than, greater than, and equal to (e.g. at least 10, exactly 1, less than 3, etc)
  • Filter visitors by amount of time spent on site (at least 5 minutes, less than 1 minute, etc).
  • Filter visitors by revenue (at least $1000, exactly $100, less than $100, "any" amount of revenue, etc)
  • Filter visitors by a page URL (e.g. all visitors who viewed "/contact")
  • Goals: Support for javascript/onclick events to complete a goal
  • Goals: More up front details about revenue in views outside of the main goals page
  • Goals: View history (timeline graph) for conversion rate and/or revenue for any goal
  • API for adding data to a session after the fact (e.g. declaring a goal as completed, adding/changing revenue, etc)
  • Clicking an item under "links" or "pages" will by default invoke visitor filtering for that item, rather than going to it (e.g. clicking "/contact" would fitler your visitors by that page, instead of going to the contact page on your web site). To actually go to the link/page, there will be an arrow next to the item to click, if you want to do that instead.
  • Much better mirror/sub-domain support
  • Hourly data figures for visitors and actions
  • Track figures for page views, outbounds, downloads, and "clicks" seperately (dashboard module "the basics" will have expandable view to see this extra data for the total actions value)
  • Super Mega Ultra Spy (this will blow you away)
  • Visitor groups - kind of like a "saved" filter, e.g. all visitors from the US who arrived via Google and use Firefox. Groups will be able to be used as funnels for goals, which will be amazing.
  • Interface to update your credit card details, instead of forcing you to sign up for a new subscription when your card expires or you need to change it for whatever reason.


As you can see, that's quite a number of things coming soon. So be patient, and it will all eventually be yours!
18 comments |   Jan 26 2009 9:06am

The new user homepage




We've had all sorts of requests to improve the user homepage to give a better summary of all your sites with one glance. The screenshot above shows you what we have come up with, which you can see for yourself on your user homepage.

This is basically the Super Mega Uber Dashboard. The Dashboard of all dashboards. But hopefully not as controversial as when we launched the new site dashboard :) Our goal was to make the most amazing, user-friendly, as-much-data-as-possible-but-still-clean-and-efficient interface that could possibly be made to overview the traffic on all of your web sites on one page. There is nothing else like this on the market. Well, not yet anyways. Give Woopra and Piwik a couple of weeks - they'll soon have their own imitation of it.

Here's a quick run down of everything:

  • By default, for each site you will see today's visitor and action tally, and a graph on the right of the visitor count for the last 30 days.

  • You can customize the type of data shown for "today". Options include visitors, actions, bounce rate, and goals (which includes revenue, if applicable).

  • Clicking on any of the links under "today" will refresh the graph with that data type. By default it shows visitors but if you click actions, for example, it will refresh with the daily action tally.

  • You can change the date range for this "tally" data with the same options on the normal dashboard - 30, 60, 90, or 180 days, and weekly (26 weeks) or monthly (12 months).

  • You can sort your list of sites by alphanumeric (default), or any of the data types you have enabled for "today" - e.g. most visitors, most actions, etc. Your sort preference is remembered until you change it at a later time.

  • You can also view "popular" data types for each site. By default the options are searches, links, and pages, but you can customize it to have any and all data types available in the drop down menu.

  • You can customize how many results are displayed for popular data. Default is 10, but available options range from 5 to 50.

  • There are "global" menus (in blue) at the very top. If you change the data type to "searches" for example, all of your sites will refresh to show your top searches. Very usefeul and amazing.

  • There is a global date menu shown in this screenshot, but there is a bug with a few of the major web browsers that we haven't been able to fix yet (Surprisingly enough, MSIE is the only browser that actually works 100% correct for this menu. Is it just me or are pigs flying?). Until that is fixed, the global date menu is disabled for all types. But we know you'll find it uber awesome once we get it working right. Being able to, for example, refresh all of your sites with just a few clicks to show the monthly visitor tally, or the top searches from the last 30 days, is insanely amazing.

  • Option to hide the forums from your user homepage, if you find them distracting.

  • Customizaton, sorting, dates, and global menus are only available for premium members. Since free members are limited to 1 site and 30 days of history anyways, this really isn't much of a limitation.


Thoughts? We're really happy with it from our own internal use, but we fear our page view limit is going to drop, with so much data available on one page. We know, the page view is dead and all, but still, it's a number that's hard to let go of.


Update

By popular demand, you can now disable the line graphs. This has the unfortunate side effect of disabling the popular data options as well, since they shared the same space previously. When disabled, you will see "yesterday" and "last 7 days" instead, as shown below. This brings it back to being quite similar to the old userhome, except it looks a lot cleaner, and you still get the sorting options and the ability to choose a few additional types other than just "visitors" and "actions".

37 comments |   Dec 15 2008 8:11pm

Keep tabs on your twitter followers from Clicky

We've partnered with TwitterCounter.com to integrate their data into Clicky, via their just released API. This service monitors the number of followers you have on a daily basis. Your Twitter account isn't necessarily tied directly to your web site, but we have so many bloggers and other people who use both Clicky and twitter (us included) that it made sense to integrate this.

All we need is your twitter username, which you can enter in on the site preferences page. Then your follower count will be integrated into "the basics" dashboard module (you'll notice we also added your FeedBurner count here too):




We also created a simple dashboard module for twitter, if you want to always see the graph by default. (Hot tip: you can also click the red/green percentage next to any item on the dashboard to see its history, including twitter and feedburner):




Check out the Clicky demo to see this all in action!

Anyone have ideas for other third party services that it would make sense to integrate? The two we thought of recently:

- Google PageRank - unfortunately there is no official API to get this data and Google frowns upon any sort of hax0rz to obtain it. Also this number only changes a few times per year at the most for any given site.

- Technorati "authority" - this service used to be huge but I don't think anyone takes it seriously anymore.
18 comments |   Nov 24 2008 11:31pm

Max date range is now 1 year for all Pro members

By far one of our most requested features has been to increase the maximum date range limit of 31 days. We've been delaying this because of the performance penalty our servers take when accessing such a large range of data, but we can ignore your requests no longer. As of now the maximum date range has been increased to 1 year. However, because this does take a lot of extra processing, we can only offer it to our Pro members. And be patient. If you do an entire year, it can literally take several minutes to get the data!

We'll be monitoring how this affects the performance of traffic processing. If it's too big of a hit, we may have to decrease it to 180 days. I don't want to do that, 1 year is kind of a magic number, but we'll see what happens.

Also, the dashboard is still limited to 31 days. Pulling up an entire year's worth of data for potentially ALL data types might cause our servers to explode, so we can't offer you that. But on all individual pages other than the dashboard, and including the API, the max range for all Pro users is now 1 year. When you click the "Custom data range" link in the date drop down menu, it will tell you the max date range for that page. On the dashboard it will always say 31 days, but on all others pages it will say 366 (1 extra day for leap years).
5 comments |   Nov 08 2008 1:21pm

Next Page »




Copyright © 2017, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS