The new Clicky is good to go

January 19, 2008

It's alive! I was going to write this nice long blog post all about everything new, but I'm dead tired and need sleep. So I'm just copy/pasting my development notes into here, which was also kind of my rough draft for this post I had all planned in my head. We'll have an official press release pretty soon that will be more along the lines of what I wanted to post here, including some pics. This post is really long, but very detailed also. If you want to know everything that's new, you should read it.

All site IDs below 15,000 or so won't be ready until about 10PM USA PST on Saturday, because that server is HUGE. So big in fact that if we didn't cut out some of the really old history, we'd be here for almost twice as long. Visitor and action data more than 6 months old got the axe. Sorry, but it had to be done, and we estimate we saved more than 24 hours with this measure. 6 months has always been our official history length anyways, but this is the first time we've had to enforce it.

All other database servers are online and ready to use though - they were much smaller. There are some minor bugs, so bear with us over the next few days while we work them out. But for the most part it's working great.

To all Freewebs users, sorry but I wasn't able to manage your servers all at the same time as the normal ones, it was just overwhelming. I'll get to it on Saturday afternoon, and you should only have about 6 hours of downtime.

Please leave comments and bug reports in this post. Enjoy :)

New visitor filtering features

• Slick new filtering interface with popup menus, easy to add/change/remove filters on the fly
• When you filter visitors, it also analyzes all of them and gives you useful summary data, and compares it to the site average for the selected date range. Pro users have the ability to compare it to another sub-set of visitors too instead of just the site average.
• Filters now work across multiple days! With the old system, if you were looking at "last 30 days" and selected google.com as the referring domain, it would only show you one day of results. This was intentional because the system was not optimized to support querying such a large span of data. It is much more optimized now - BUT - we're currently limiting them to 7 days until we can get a feel for how much impact they will have on our service with real world usage.
• Some new filters: custom data (e.g. usernames or emails), cities, screen resolutions, and links. (Links meaning, the full link someone used to get to your site - before you could only do this with the domain name. Now you can filter your visitors down to just the ones who came from the exact URL you input.)
• "Anything" filters, e.g. domain= will only return visitors who arrived at your site from another site. This even works on custom data tracking, e.g. custom[username]= returns only visitors who have a name assigned to them .... Partial wildcards aren't supported yet, but we plan to add them in (e.g. domain=google won't work - yet)
• The global maps page has the same filtering interface and options as the visitors page (as it did before with the old filtering interface).
• Actions page got the new filtering interface too, but the filters are obviously different. You can filter by page URL, page title, and action type (pageview, download, outbound, click). You used to be able to filter actions by the same criteria as visitors, such as country or web browser, but that really doesn't make any sense and has been removed. Page URLs and titles support ending wildcards, e.g. "/some/path/" would only return actions on that specific page "/some/path/", while if you entered in "/some/path/*" (asterisk on the end), you will get actions returned such as "/some/path/more/stuff/" and "/some/path/another/path/". ENDING WILDCARDS ONLY! (for now)
• When you use the data export feature while you have any filters applied, those filters will apply to the export as well! Works with both actions and visitors.

New centralized area to manage all site preferences

• Managing site preferences used to be a pain, as they were scattered all over the place - some on your user page (which a lot of you never even see), and also some on the stats pages (like the dashboard prefs). It was a confusing mess, as indicated by the number of emails we got from people looking for these things. So we fixed it! There is now a new main tab on your stats page that takes you to the new site prefs section. From here, you can get info about your site (sitekey, etc), manage your dashboard, manage ip tags and filters, edit your site (timezone, domain, etc), and get your site's tracking code. All in one place. Much better.

Visitors

• Most active visitors. This shows you the people who have visited your site the most in whatever date range you have selected. If they have a name, it shows that and links to it, otherwise it shows and links to their IP. It's grouped by IP though, not username or hostname or organization. Grouping, filtering, and sorting by IP address is the most optimized way to do this, and since it's reading through so much data, it's the only realistic choice.
• When viewing visitor details, there is now a link next to their IP address "Name this visitor", which, you guessed it, takes you right to the IP tagging page with the IP address filled in. We've received billions of emails about making this easier. I agree the old way was horrible. Hopefully this will satisfy!
• When viewing your visitors list, the visitor's name is chosen in this order of priority: Username (from IP tagging or custom data tracking), Organization, Hostname, IP address. This gives you a lot more info up front about your visitors, since a list of IPs is pretty gosh dang boring and useless. When you click on a visitor, it used to always just link to their IP address, even if it was showing their username. Now it links to whatever is actually shown. If they have a name, it will filter your visitor list by that username to show all visits from that one person, even if their IP address is different between visits. If it shows an organization or hostname, clicking on that will filter down the list by that organization or hostname. If you want to see the details for just one visitor, click the "# actions" link to view their session. From there you can click the IP address to find all other visits from just that user.
• Visitors list no longer shows browser and OS icons next to each visitor, to give more room to the referrer and search data, and also the hostname/organization if applicable. You can still see anyone's computer info by viewing their details.
• Visitors list now has a visitor's search as clickable (to initiate filtering by that search), also you can hover your mouse over the search to see the full text instead of being limited to the first 20 characters or so
• Visitor hostnames - up to now you could only see the hostname for any visitor when you clicked on their IP or viewed a session. We limited it to this view because looking that info up is slow and saps resources. Now we are using a product from MaxMind to lookup the hostname for EVERY visitor automatically when we process that visitor's traffic. However, just like the geolocation product, it's not guaranteed to be 100% accurate, because IPs are always changing hands - but it should be pretty good. Also, it's not a full hostname lookup, but just the top and second level domain. For example, visitors using Comcast cable internet will show up as "comcast.net", not their full hostname which might be something like "24-60-143-200.hsd1.or.comcast.net".
• Visitor organizations - this is another product from MaxMind that tells us the organization that this IP is assigned to. For home users, this value will typically be their ISP name (e.g. "Comcast Cable"), but otherwise should be the name of the company or corporation that the IP is assigned to, e.g. "Microsoft Corp".
• You can of course filter visitors by either of these two new criteria.
• Two new sub-tabs under visitors tab to view the most popular hostnames and organizations, also added as sub-tabs to the "Locale" dashboard module.

Much more accurate numbers for searches, links, and content

• Referrer and search data is now only logged for the first action of any visit, instead of any and every action. This prevents multiple counts for people who show up at your site, click a link, press back, click a link, press back, etc etc... Each time someone presses back, their browser sends us the referrer data again, even though it's no longer "fresh". We used to log this data and increase the count for the applicable link and search, but now we don't. Potential problem: every once in a blue moon you might have a visitor who shows up at your site, then a few minutes later shows up again from a search engine or link because they searched for something on your site after seeing it on your site. This new system will not log these links and searches, but we think the trade off is well worth it, and this circumstance is so rare it should not be an issue.
• Page titles - Unique pages are determined by a combination of URL and title, but non-javascript users don't send us the title. This has been a big problem, because the pages they view have been seen as unique from the ones everyone else views. Now when we get sent a page with no title, we try to find an entry in the database for this URL that has a title and we link to that one instead.
• Page URLs - Link #hashs are stripped off the end of URLs (for both your pages, and referrers) so that they don't get counted as unique. However, they are left on the end of URLs for file downloads, outbound links, and clicks, because these may be more relevant, particularly if you are using javascript on your pages.

Searches and Links: Recent, and Newest Unique.

• Recent is just a log of all searches and links to your site, newest to oldest (similar to what used to be sub-tabs under the Actions tab, except presented in a much better fashion now. The old way was really just an ugly hack). "Newest unique" means searches and links that have led someone to your site for the very first time (since you installed Clicky). This helps you identify new traffic sources, and also makes sure you don't miss any "long tail" items. If you're looking at "today", it shows you the newest 50, no matter when they occurred. However, if you are looking at the past or a specific date range, then it will only show the ones that occurred within that time. Note that this won't work for dates prior to today, as previously we were not tracking the date/time that a link or search first occurred. This also means that searches and links you've already had will show up in "newest unique", even though they technically aren't. So this feature won't be much use for a few weeks until most of your common and popular searches/links get tagged as new. Patience!

Tracking code updates

• You can now customize the length of the "pause" when someone clicks on a download or outbound link. It currently defaults to 500 (milliseconds, aka, 1/2 second). To change, set a global javascript variable called clickypausetimer (timer, not time) to the value you want, e.g., var clickypausetimer = 200; This must go BEFORE the tracking code on your site. The shorter you set it, the more likely that the data won't be logged, because there may not be enough time for the visitor's browser to contact us before moving to the next page. It's a trade off, but now you have the power to tweak it to your liking. Previously, the only option was to completely disable it.
• Those of you using the clicky.log() function - there is a new action type, "click", that is also the new default if you don't specify a type. This makes more sense than defaulting to "pageview" because typically if you want to track when someone clicks on an item, I have trouble thinking of a scenario where that should be counted as a page view. Clicks have a new icon in the action log, they show up with the mouse icon (same icon as in the main "actions" tab). There is also a new "clicks" sub-tab under the content area, to show you the most popular clicks. Most of you probably won't be using this but some of you will.

API

• All visitor and action filters work with the API! For visitors: country, city, language, browser, os, resolution, domain, search, link, hostname, organization. For actions: title, href, action_type [pageview,outbound,download,click]). All values must be URL encoded. For "global" types of data, e.g. things that are not specific to your site only (country, city, language, browser, os), you do have to type them in exactly right, which is unfortunate since you don't necessarily know how we have them stored. e.g. you might try "browser=msie" and get no results, because we have it stored as "Internet Explorer" so you would have to do "browser=internet+explorer". We plan to add in some flexibility here, and also have a list for you to download of all the official values.
• Some of the new data types have been added into the API: hostnames, organizations, bounce-rate, and actions-average... Most active visitors, and recent/unique searches and links are MIA but will be there eventually.
• None of this has been added to the API docs yet but will be with our next major API release, which will be coming in a month or two.

Other minor features, tweaks and bug fixes

• Bounce rate - percentage of your visitors who only viewed one page. This is shown in "the basics" on the dashboard, and also when using visitor filters.
• Popular data pages (searches, content, etc) now have trend data just like the dashboard, including the ability to click the trend percentage and see daily history for that one item. Also added "next page" button so you can see beyond the top 50. I can't think of any good reason why I never did this before, because it's basically the most obvious feature in the world.
• Time has always been rounded to the nearest minute, which is fine for large values, but a bit obnoxious for smaller values such as individual visitor times. Now we always display the seconds for all time values that are less than 24 hours. Also, the minimum time for a visitor is now "5s" instead of "less than a minute".
• On the dashboard, in "the basics", previously you couldn't click the trend link for "total time spent" and "average actions/visit" to see their history. Now you can!
• When viewing daily trend data for one item on the dashboard or any of the other pages that support it, each date is now a hyperlink that will take you to the visitors page for that date with the appropriate filter to instantly view and analyze those visitors. Confused? How about an example: You are viewing daily trend data for visitors from google.com. You click the date "Mon Jan 14" - this will take you to the visitors page, with the "domain" filter set to "google.com" for Jan 14.
• Our date math for selecting entire consecutive months has been fixed. Huh? Ok, sorry this gets a bit confusing as well... Say you were viewing the entire month of Dec 2007, that's 31 days. If you viewed the actions subtab on the dashboard, it would show you the total number of actions in that period, and then going back previous periods too. Those previous periods used to be based on the number of days in the selected period, which means it would be the 31 days before Dec, 31 days before that, etc. Obviously not every month has 31 days, so the selected ranges were a bit odd. This has all been fixed so if you view an entire month, all the previous period before that are whole months as well.
• "Referrer" is now spelled correctly in all places, except API output variable names, which will be changed upon the next API release. I place all blame on Apache for corrupting my otherwise stellar spelling. I'm not alone: http://www.google.com/search?q=apache+referer+spelling
• You can enter in "*" for site mirrors, which means traffic will be logged for any domain (good for sites with lots of sub domains). Better sub-domain support will come in the future.
• Feedburner icon only shows up in the tab bar if you have it configured for that site.
• IP tagging page - now you can click on the name or IP in that list to view all visits by that name or IP.
• Dashboard trends - for "today" trends, these have always been estimates, and not hard figures. They would be too intense to calculate otherwise. This has understandably caused confusion for some of you. Now, when you hover the mouse over a trend, it will say e.g. "~500 yesterday". The ~ signifies it's an approximation and not an exact figure.
• We've moved to a third party ad system, as Google penalized us for selling static text links, which have been removed. Ads are only shown on stat pages. Free users have a banner in the header and footer, blogger premium have a banner in the footer, and Pro accounts have no ads anywhere.

The following new features are for premium members only!

But for the first 7 days, EVERYONE has access to them! (Except for the visitor comparison feature, because it's a bit intense. If you want to try it out, hit the front page of getclicky.com and click on the "demo" link to try it out.)

• Hostnames
• Organizations
• Bounce rate
• Most active visitors
• Visitor analysis when you have filters applied.
• Visitor comparison analysis (comparing two subsets of visitors ("Pro" only)

We were trying to do too much at once and had to cut some features from this release to get it out the door already. They were all mainly "Pro" features but included things like campaign and goal tracking, and reports by email. We hope to have all of these out by the end of February.

That's it!