New preferences: Log bots, and hide the current hour in hourly graphs

As part of our massive preferences overhaul, we added two new preferences that have been requested fairly often.

Bot logging

This is a new site (tracking) preference. We highly recommend leaving this OFF - log analyzers are the best way to look at bot activity on your site - but there are specific use cases where this may be useful. For example, if Clicky is logging significantly less traffic than another tracker you are using, bots are a likely cause so enabling this will let you see if that's the case or not.

Here's where to enable this new option: (Again, we highly recommend leaving it OFF...)

It's important to note that this is no guarantee that all bots will get logged. We'll only log bot activity if they interact with Javascript (a lot of them do these days) or load our 'noscript' backup tracking method.

We'll try our best to identify Google, Microsoft, and Yahoo bots, which you'll see under the new 'Robot' section of the web browsers and operating systems reports. All other robots will just be labeled 'Robot'. Bots will also be logged as normal visitors if this is on, so you will see them in your visitors report and you can filter visitors with the new robot user agents.

Hide current hour in hourly graphs

Near the beginning of the hour, the number for that hour (visitors, actions, whatever) might be quite small compared to a full hour. If you view your reports near the beginning of the hour, you will see this as a 'cliff' and perhaps panic that your traffic suddenly died.

We highly suggest checking the current time before panicking in such situations, but we get it that some people would prefer never to be in that situation. So now there's a new user preference to always hide the current hour.

Also released today were some new on-demand features for graphs, so be sure to check those out too and let us know what you think!
3 comments |   Oct 14 2014 10:51pm

New graphing features

Part of our massive preferences overhaul included chipping away some nasty cruft. One of those things was the option to allow simple HTML bar graphs to be the default graphing method. This was used by very few people so we removed it as an option to be the default graphing option.

But have no fear, we know this method is still useful even for people who never set it as their default, so we changed it to be an on-demand option that you can see in the top right corner of every graph. Whatever date range you're viewing in a graph, clicking the bar graph will show the same data, with one exception: no hourly data. Bar graphs never supported hourly data since they were created (and deprecated) well before we added support for hourly data. So they still don't do that. When viewing hourly data, if you view a bar graph it will just show the last 28 days instead.

We've also added the ability to export the raw data you see in any graph straight from the API, straight to CSV or the other formats our exporting of normal (non-graph) reports has always supported. Unlike bar graphs, this method does support hourly data, so that's good. It will honor whatever date range you are viewing in the graph.

The new-ish 'Download' menu we added for graphs has been changed to an icon indicating you can "save the graph", in there are the old options to save it in various image formats, and new options below to export the raw data itself.

Also released today were a few new features that have been requested fairly often: A site preference to allow the logging of all bot activity, and a user preference for hiding the current hour in hourly graphs, to avoid panic when seeing a 'cliff' near the beginning of the hour.

These are covered in more detail in this post, so be sure to check those out too and let us know what you think!
1 comment |   Oct 14 2014 10:47pm

Yup, we're still alive. Here's what we've been up to the last 5 months.

It's been 5 months since our last blog post. Normally we are known for our constant updates and posts, so when people started asking us recently if we were still alive, that was understandable.

I am here to assure you, we are in fact still here and things are just fine! We did try to enjoy life a little this summer so things were definitely a bit slower, but we still pushed out plenty of updates. Not many of them warranted their own post, but we figured with people wondering our status, we'd let you know what's been happening since our last post in May.

More real time!

Clicky processes data in batches, which makes things much more efficient than processing every single action as it streams in. Since the beginning (2006!) the batches have been done once a minute (excluding Spy, which is a separate system and does in fact show data live as it streams in). After nearly 8 years, we decided it was time to up the game. We started with halving the interval down to 30 seconds, and then 20. It's still at 20 right now, but the hope is we can get it down to once every 10 seconds. More testing is needed though.

We've also been testing a version of Spy that uses web sockets so data is pushed to your browser, making it truly "live" data, instead of the current "pull once every few seconds" method. Testing and debugging has taken a lot longer than expected, but we're getting there and hope to have this in production fairly soon.

First/third party cookie synchronization

We use a combination of first and third party cookies to track unique visitors to your site. Third party cookies always took precedence though, which is particularly useful for those of you tracking multiple domains (including sub-domains) under the same site ID. The problem was that if you wanted to query our API live based on their unique (cookie) ID, the cookie that your site would see for that visitor would not necessarily match the cookie we were using on our end, since your site would only see the first party cookie. We've updated the tracking so when a hit comes in, we always update the first party cookie to be the same value as the third party cookie if they don't match. This change allows you to reliably use the first party cookie ("_jsuid") value now to query our API.

PushState navigation

4-5 years ago, hashbang navigation was all the rage for making your site feel like an "app". There was simply no alternative. Then browser vendors added support for the new History API, which offers the same benefits as hashbangs, and none of the drawbacks. One day I had had enough and decided to update Clicky to this new method. It only took about a day and we couldn't be happier with the change.

Not to mention the pleasant side effect of fixing a serious bug that was nearly 4 years old that I had never been able to figure out until I was rewriting all of this code. The bug was rare but it was a really nasty one, and the moment I figured out the root of the issue was cause for celebration.

Weekly/monthly data accuracy

Clicky stores weekly and monthly data in their own tables to make querying a lot faster over large date ranges. The problem was that these tables were only updated once a day at around 4am PST (GMT -8). So data for the current week and month would always be off because they would never include "today". We updated the code to take this into account so these values should always be accurate now.

Multiple values for single custom data keys

Clicky has always logged every piece of custom visitor data you've thrown at it, but for any given key, we've always only shown the last value logged for any specific visitor. We will now show all of the values logged, both in the web UI and in API exports.

Code generator

Not everyone's a programmer. So we added a new code generator page to help you set up some of the custom tracking options, video/audio analytics, and dynamic campaigns (utm_campaign etc).

Google tag manager

Google tag manager has really taken off, and we get requests for how to use it with Clicky all the time. So we added these detailed instructions to our apps and plugins page.

More exporting options

Any line graph you see can now be exported to a variety of image formats with the new "Download" menu you will see in the top right corner of every graph in every report.

You can now also export data in "The Basics" module from our UI. This has been a sore spot for the longest time, almost every report can be exported to a variety of formats by clicking the disk icon in the top right corner, but when you're on the dashboard the only option was PDF. Now we've added a new option in the export menu, "The Basics...", when you're on the dashboard. Click that to be taken to a new page that gives you options for exporting all of that data.

Pagination and totals

At the bottom of every report we now give you options to jump to any and every page, as well as show you the total sum for the current page and all pages.

Auto-play looping HTML5 video/audio files no longer auto-tracked

Our code tries to detect all HTML5 video and audio tags and track interactions automatically, but when we added this feature we didn't take into account that sometimes those videos might auto-play and loop (big problem with background videos for those fancy sites you see sometimes). This could lead to an explosion of tens or even hundreds of thousands of actions logged for a single visitor session. Ouch.

Heatmaps on sites with multiple tracking codes

Many customers use multiple tracking codes on a single site. The problem is that when the heatmap viewing process is initiated, the code always just used the first site ID in your HTML to grab the data. That's fine if initiating from the on-site analytics widget, but if you were initiating it by clicking a heatmap link from the content report on Clicky, that wouldn't always be the same site ID, which could really break things if for example trying to view a session heatmap. Now when you click on these links, the site ID is included in the URL so it should always work.

There's been another issue with heatmaps when used with Firefox 29+, that issue being heatmaps don't work at all. We're not sure what they changed that broke this, but the good news is the developer of the heatmap javascript library we use has fixed the issue with the most recent release of his library. But it's a major-version upgrade, so compatibility issues may exist. It's near the top of our list to investigate so hopefully we'll get this upgraded soon and heatmaps will start working again in Firefox.

Graphs in PDF exports

A change we made to javascript caused a fatal error with WKHTMLtoPDF, causing the graphs to be completely broken for a few months in PDF exports. We send thousands of these things every day in automated email reports but surprisingly no one emailed us about this issue until it had already existed for about 2 months. But it's fixed now.

Coming soon

We're currently working on a major overhaul to site/user prefs, one of those things being to make the trend and graphing preferences be user preferences rather than dashboard preferences. The dashboard options will still be there but a user will be able to specify whether or not they want they want their own settings to override them.

While we're at it, we're adding two new preferences. One will be a site preference to allow bots to be logged to your stats. Most people don't want to see bot traffic so we (try to) filter it all out by default, but we do get requests to let them get logged. So that will be an option soon, disabled by default though of course. The other will be a preference to hide the current hour in hourly graphs, to avoid the "cliff" effect you see when it's near the beginning of the hour.

In addition to some other new features we'd like to add, we've also been contemplating a major UI overhaul to make our reporting prettier and more dynamic, not to mention letting Clicky take full advantage of the high-resolution screens that just about everyone has these days.
7 comments |   Oct 07 2014 2:23pm

OS X push notification alerts

We just added OS X push notification support to our alerts system. Push notifications at the OS level require
the latest version of OS X (Mavericks).

Here's what an alert looks like. Click on an alert to be taken straight to the visitor session on Clicky.

To set this up, you need to use Safari on a Mac running Mavericks. Safari is only needed for the initial authorization of push notifications, so if you use a different browser normally, don't panic.

Go to the alert setup page for any of your sites and click the link at the bottom:

You will be prompted to authorize Clicky to send push notifications to your Mac. Once you grant permission, the "device" (your Mac) will be attached at the user level so if/when you setup alerts for other sites, you will NOT need to go through the authorization process again, the device will already be available to you during the alert setup process.

You can name each Mac after you add it so if you have multiple Macs and you want to get alerts on all of them, you will know which one is which, in case you want to remove any of them later. This includes sub-user accounts; when managing alerts, you will see a list of all devices that belong to all users with access (on Clicky) to the site in question.
3 comments |   May 06 2014 4:04pm

HTML5 History API tracking

If you use the HTML5 History API on your site, good news! Clicky now automatically supports it.

This type of navigation typically only reloads a small portion of your page to inject new content, which means our tracking code (previously) would not be executed again since that part of the page would be static -- unless you manually added calls to clicky.log or clicky.pageview when you executed history.pushState or history.replaceState.

Clicky will now automatically track calls to both pushState and replaceState, as well as listen for the window.onpopstate event that occurs when a visitor clicks back or forward in their browser.

If for some reason you need to disable this new functionality, for example you were already logging calls manually, you can do so with the new clicky_custom.history_disable option.

Long term, we want to replace our own custom hashbang-ish navigation that we use on with the History API. This automatic support of logging these methods will be a good motivator.

Update, May 7: We've removed replaceState()from being auto-tracked, as its purpose is quite a bit different from pushState() and was causing some excess data to be tracked that shouldn't have been.
3 comments |   Apr 30 2014 4:15pm

Next Page »

Copyright © 2019, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS