API throttling

The analytics API has been a complete free for all in almost 6 years of existence. This has rarely been an issue, save for once or twice a year maybe, we'd have to ask someone to please relax themselves.

But recently it's become a serious ongoing problem. We've had at least 3 different people in the last few weeks all doing utterly massive exports of data, causing some of the database servers to lag quite badly (up to almost 2 hours in the most severe case).

When a server is lagging it affects thousands of customers. We can't have this anymore, so today we have implemented some API throttling functionality and it is live now.

Throttling will only apply for visitors-list, actions-list, and segmentation requests, as those are by far the biggest drain on resources. All other requests are unaffected.

Here is how it works:

  • Maximum of 1 simultaneous request per IP address per site ID at any point in time. Part of the issue recently has been people doing automated simultaneous requests for exporting data, in one case over 20 requests at the same time for the same site ID, from the same IP. This will no longer work. You will receive an API error.

  • Maximum of 500 results per request (down from 5,000), maximum date range of 3 days. This one is pretty strict and we will likely raise these limits, but we have to get API usage under control immediately. We will be monitoring things and plan to raise the limits as things calm down. UPDATE: things have been stable so we've raised the limits to 1000 results and 7 days.


To repeat, these changes only apply for visitors-list, actions-list, and segmentation requests. No other types of requests are affected by anything mentioned here.

We know this is pretty lame, but it's in the interest of keeping the service as close to real time as possible for all customers and that's important. Hope you understand.

-------
Update, Monday Feb 11:

Since we made this change on a Friday, and Friday-Sunday is a complete trickle compared to the rest of the week, it wasn't until today (Monday) that we could really see the effect of this change.

Good news: All servers are keeping up with real time no problem now. A few are 1 minute behind right now, which sometimes happens when caches expire on the servers and have to regenerate, and usually they're back up to normal shortly thereafter.

We'll likely raise the single day restriction soon, first to 3 days, and if things keep up, then probably 7 days. I don't know if we'll ever let it go beyond 7 days again though. As well, the limit of 500 per request will probably be raised to 1000, but again I'm not sure if we'll ever let it go beyond that.

----
Update, Tuesday Feb 12:

Things have been stable so we've raised the date range limit to 3 days now. We'll see how things go from here.

----
Update, Friday Feb 15:

Things continue to be stable so we've raise the date range limit to 7 days, and the result set limit to 1,000 items. We've also changed the "one request per IP address" limit, so that it's "one request per IP address, per site ID".
14 comments |   Feb 08 2013 4:11pm

Monitage: Uptime monitoring beta

Many Clicky users have asked us about site monitoring. We are happy that this is top-of-mind for some of you because its importance should not be understated. Often we will receive emails from Clicky users asking why there was a dramatic lag in visitors tracked or a complete drop-off altogether. While there can be many explanations, it is not uncommon that the site itself went down unbeknownst to the site owner. Standard web analytics will not be able to tell you this, but site uptime monitoring and alerts will.

With this in mind, we are excited to announce a closed beta in partnership with Monitage, a newly-developed site uptime monitoring service by Roxr Software (that's us). We have integrated Monitage into Clicky to give you a bigger picture of the health and activity of your web sites.




Monitage monitors web sites from five locations around the world (three in the US, one in Paris, one in Japan) and only declares a downtime event if a majority of its servers agree on it. This prevents network hiccups on the monitoring end from sending false alarms.

Pro Plus users and above receive access to the Monitage closed beta. When we officially launch, you will have the ability to create up to 30 checks per site with intervals as fast as 1 minute, but during testing we want to keep resource usage within a reasonable range. So for the time being, we are limiting to 3 checks per site with a max interval of 5 minutes. We expect to officially launch within 4 weeks, at which point Monitage will also be available as a standalone service.

To access Monitage, go to your site dashboard and click the Uptime tab. You can create checks for HTTP, HTTPS, SSH, FTP, IMAP, IMAPS, and ICMP (ping). We've also created a dashboard module.




You can also access uptime stats from the API. Check the API docs and search for "uptime". type=uptime will give you the current status of all of your checks for a site. type=uptime-list will give you a chronological list of all downtime events for your site for the date range requested.

Last, we added uptime stats as an option in email reports as well.

When Monitage officially launches, we will determine what intervals and types of tests will be included with Clicky Pro Plus plans and above.

We are asking that you test Monitage, and let us know your thoughts, what you like, don't like, and want to see. As Monitage is in its infancy, we want your feedback to help mature it into a stalwart companion to Clicky.

Note to white label customers: Monitage will be added as an option to white label service when it officially launches, but for now it is only available to Clicky users.

----
Update, Feb 4: Just pushed an update that integrates uptime monitoring into the Big Screen report. Also added web hooks to the setup page. Enter a URL and we will POST a JSON object (documented in the setup page) to that URL for events.

Also we've been getting reports of false positives since launch, we are pushing some updates later today that should fix it entirely, or at least close to.
21 comments |   Jan 22 2013 6:23pm

'GetClicky' no more!

You may have noticed that we're now live on clicky.com! Yay!

For 6 years, our brand has always been "Clicky". Excluding our domain name, nowhere on our site has the brand "GetClicky" ever been mentioned, other than some testimonials that we didn't edit. However, because of the old domain name, the majority people have always referred to us as "GetClicky". It didn't help that our Twitter handle was also @getclicky, but we wanted it to match our domain name (bad idea). It's always kind of driven us insane, but it's been an interesting lesson. If you can afford the right domain name, do yourself a favor and buy it as soon as possible. We paid a pretty freaking penny for clicky.com, but the second it was in our possession was just pure unadulterated joy.

We've had the @clicky Twitter handle from the beginning, but it was just a placeholder. We've now swapped @getclicky with @clicky so all of our followers and tweet history remains in place. Fun fact, Twitter does not offer an official method to do this - you have to actually relinquish your brand for a brief few seconds while you swap things around, which is extremely nerve racking to say the least. So @clicky became @clickyx, @getclicky became @clicky, and last @clickyx became @getclicky. Scary 20 seconds or so. Anyways, please note: If you tweet at us in the future, use @clicky, not @getclicky! But if you were already following us you will keep getting our tweets without having to do anything.

Any page viewed on getclicky.com is now redirected to clicky.com, with a little note at the top about updating your bookmarks. That note will go away in a few weeks, but for now we want as many people to notice the change as possible.

You do NOT need to update your tracking code, because getclicky.com isn't going anywhere. Your existing tracking code, along with any integrations you have with the API, widgets, or whatever via getclicky.com, all will continue to work. Going forward, getclicky.com will only be seen in the tracking code (we are using it indefinitely for tracking). And we'll continue to use it for email, for now at least, since that's a nightmare to change. So... clicky.com = web site domain, getclicky.com = tracking domain, email, and indefinite backwards compatibility.

This was not as easy as just acquiring clicky.com and pointing it to a new IP address. A service of our size and complexity, there's a LOT of things to deal with when changing your domain name. My checklist is ridiculous in length and I hope to go into it more in a later blog post.

There may be a few bugs lying around... we'll squash them tomorrow as they come to our attention.
32 comments |   Dec 19 2012 12:34am

On-site analytics update

Just pushed a much needed tracking update for sub-user accounts, so now they can use on-site analytics, and their traffic will now also be automatically ignored.

Previously this was not possible because of the way our tracking servers were caching information about each site in memcached. After banging heads on keyboards for a while, we finally figured out a way to do it (it's more complicated than it sounds). So, now sub-users will have their traffic ignored automatically, and they will see the on-site analytics widget.

One caveat - the master account's preference for disabling on-site analytics on an account wide basis will override a sub-user's preference.

Making this change also addresses a bug that popped up after the four-week heatmap trial ended a few days ago. All sub-user accounts are created as "Pro", since Pro has always had access to every feature, and a Pro account was required to have sub-user accounts. But now that heatmaps require Pro Plus or higher, heatmap tracking broke for some of you if you had created another sub-user account and assigned them admin access to a site in your account. Their account features were overriding yours on the tracking servers for the sites they had admin access to. So this update fixes that. But we went ahead anyways and made sub-users account types mirror the master accounts - any time the master account type changes, the sub-users are updated accordingly, to address any future updates that might cause this scenario to come up again.
2 comments |   Nov 16 2012 2pm

Update on heatmaps

Heatmaps launched four weeks ago. As we said in that post, everyone would have it for four weeks, but then we would be requiring a Pro Platinum account to continue usage after that point. The reason for this is because the extra bandwidth and storage requirements for this data are significant, which has proven to be the case as we've monitored resource consumption since launch.

After listening to your feedback, we've decided to add a new plan (Pro Plus) with pricing in between Pro and Pro Platinum. It is the exact same as standard Pro, except that it includes heatmaps.

As of now, these changes are live. So if you are the standard Pro plan, you will need to upgrade to at least Pro Plus in order to keep heatmap tracking going forward.

Note to white label account holders: We've had a few of you ask about heatmaps for white label, we are including this feature with all account levels.
12 comments |   Nov 12 2012 2:28pm

Next Page »




Copyright © 2017, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS