API throttling

The analytics API has been a complete free for all in almost 6 years of existence. This has rarely been an issue, save for once or twice a year maybe, we'd have to ask someone to please relax themselves.

But recently it's become a serious ongoing problem. We've had at least 3 different people in the last few weeks all doing utterly massive exports of data, causing some of the database servers to lag quite badly (up to almost 2 hours in the most severe case).

When a server is lagging it affects thousands of customers. We can't have this anymore, so today we have implemented some API throttling functionality and it is live now.

Throttling will only apply for visitors-list, actions-list, and segmentation requests, as those are by far the biggest drain on resources. All other requests are unaffected.

Here is how it works:

• Maximum of 1 simultaneous request per IP address per site ID at any point in time. Part of the issue recently has been people doing automated simultaneous requests for exporting data, in one case over 20 requests at the same time for the same site ID, from the same IP. This will no longer work. You will receive an API error.


• Maximum of 500 results per request (down from 5,000), maximum date range of 3 days. This one is pretty strict and we will likely raise these limits, but we have to get API usage under control immediately. We will be monitoring things and plan to raise the limits as things calm down. UPDATE: things have been stable so we've raised the limits to 1000 results and 7 days.

To repeat, these changes only apply for visitors-list, actions-list, and segmentation requests. No other types of requests are affected by anything mentioned here.

We know this is pretty lame, but it's in the interest of keeping the service as close to real time as possible for all customers and that's important. Hope you understand.

-------
Update, Monday Feb 11:

Since we made this change on a Friday, and Friday-Sunday is a complete trickle compared to the rest of the week, it wasn't until today (Monday) that we could really see the effect of this change.

Good news: All servers are keeping up with real time no problem now. A few are 1 minute behind right now, which sometimes happens when caches expire on the servers and have to regenerate, and usually they're back up to normal shortly thereafter.

We'll likely raise the single day restriction soon, first to 3 days, and if things keep up, then probably 7 days. I don't know if we'll ever let it go beyond 7 days again though. As well, the limit of 500 per request will probably be raised to 1000, but again I'm not sure if we'll ever let it go beyond that.

----
Update, Tuesday Feb 12:

Things have been stable so we've raised the date range limit to 3 days now. We'll see how things go from here.

----
Update, Friday Feb 15:

Things continue to be stable so we've raise the date range limit to 7 days, and the result set limit to 1,000 items. We've also changed the "one request per IP address" limit, so that it's "one request per IP address, per site ID".