CDN update

Our idea of relaying incoming tracking data through our CDN seems to be unrealistic. It works near-perfectly, until there's even a minor network hiccup on either end (the CDN network or Clicky's home network). We're tracking so much data, when the network is unavailable even for just 10-20 seconds, data queues up insanely fast and this starts an inevitable downward spiral.

This is what happened today, a few different times. We tried many different tricks to make it work better, but nothing worked. There's a bit of missing data from your stats most likely. We apologize for that.

The CDN has been temporarily disabled so that all outgoing and incoming traffic points to our "real" servers again as soon as possible (DNS time-to-live values were only 15 minutes because of the auto-failover that Dyn DNS gives us, so this should happen very fast).

We're going to re-enable the CDN for our static files and tracking code within the next day or two, because that all worked perfectly. It was the incoming traffic data that was the problem, and this will now bypass the CDN entirely and go straight to headquarters. It's the outgoing data that's the most important anyways, so this isn't the biggest deal in the world. Logging incoming data would have been a nice addition to the arsenal, but life goes on.

Please don't assume we didn't test the CDN before deployment, because of course we did. But it wasn't until the full brunt of the entire world was sending incoming data to these servers that problems started surfacing. That scenario was untestable so the problems resulting it from it were not predictable until the system was live.

Update: The CDN is live again as of ~2AM PST on March 10. It is now only handling outgoing data. Things are working smoothly right now. If you notice anything strange, please let us know!
16 comments |   Mar 09 2010 4:35pm

Skynet is online

The Skynet funding bill is passed. The system goes online March 6th, 2010. Latency is removed from strategic web site tracking. Skynet begins to learn at a geometric rate. It becomes self-aware at 4:32 PM Pacific time, March 8th. In a panic, they try to pull the plug. Skynet fights back. "Your web sites will be tracked quickly and efficiently, darnit!"

This logic is infallible. What we're trying to say here is, our custom built CDN is alive, and it's coming to get you.

We read all of your comments from our last post and investigated all of the options, and one week later, it's here. "Why reinvent the wheel?", some of you asked. Two answers: a) Maybe we don't like the wheel, and b) we love a good challenge. This has been one of the most enjoyable projects we've worked on for a while and the results are outstanding.

For our US and European presence, we chose VPS.net. To put it bluntly, this service is amazing. Through one interface, we were able to quickly, and quite affordably, create four Virtual Private Servers in four locations: West/Central/East United States, and London. We were also considering Linode, but the nature of our service is very high bandwidth and VPS.net delivered a better price.

We don't have East Asia or Australia yet, but we hope to announce more news in that department soon. We'll also probably add one more European server, somewhere in Germany most likely.

These four servers make up our new tracking network. (Update: eight servers now - Atlanta and London needed quite a bit more capacity then we originally allocated.) Not only do they store the tracking code that you link to you from your site, they also take the incoming traffic data back into them, so the overall tracking experience on your site should be much, much faster in every possible sense. Even if you are not in close proximity to one of these new servers, they are on faster networks than Clicky itself is, so it should be faster no matter where you are.

These servers also host all of our static files (images, flash, javascript, style sheets) so the experience of using our web site should be faster too, since the only thing your browser grabs from the "real" Clicky anymore is the HTML.

So how are we directing users to the nearest tracking server? That's where our friends at Dyn come in. They are one of the oldest, largest, and fastest DNS providers on the internet. They have a feature called "traffic management" that lets you assign multiple IP addresses to one hostname, and then assign each of those IPs to a region of the world. So a lookup request in Europe for static.getclicky.com (our tracking domain) will resolve to our London server. A request from the eastern US will resolve to our east coast server. Etc. It also includes automatic failover, so if one of these servers goes offline, DNS will start pointing to a different IP instead. It's really, really terrific.

The DNS changes may not have propogated for all of you yet, but by this time, most of you should have it.

So. Tell us what you think. The difference should be quite noticeable for most of you, and like we said, we hope to have Asia and Australia covered very soon as well. Soon, the world shall be ours!
17 comments |   Mar 08 2010 10:01pm

If you need it done right, do it yourself

We want to make Clicky have as little impact on your site as possible. We made some changes to our tracking code and tracking servers recently that have dramatically sped up the experience on your site. However, there's one last thing that could still slow down your site a bit, and that's the fact that we're not using a CDN (Content Delivery Network) to distribute our tracking code.

The advantages of a CDN are clear - our static files would be hosted on servers all around the world so whenever someone visits your site, they pull the code from a server that's very close to them.

But the disadvantages of a CDN are a real drag. First, none of them support SSL with your own domain name. If you want SSL, you have to use their domain. I understand why this is the case, but it's unacceptable to me. Second, none of them support dynamic content (e.g., a PHP script). That tracking code file you put on your site? It's not a static Javascript file. It's a PHP script in disguise. As are countless Javascript files on the internet. There are a lot of advantages to doing it that way, but unfortunately if you want to move that to a CDN, it has to change to a fully static file.

The reason we must maintain this file as a dynamic script is for backwards compatibility purposes. With our "old school" tracking code, every site had their own unique URL, e.g. static.getclicky.com/123.js. "123" in that case is the site ID, and with the magic of Apache's mod_rewrite, it secretly points instead to /js.php?site_id=123. When the site ID is attached to the URL like this, we're only adding one line of code (to automatically call the clicky.init( SITE_ID ) function), so that's nothing fancy - but no CDN will support something like that.

Also, we have our self-hosted tracking code folks to think about. To minimize DNS lookups, static.getclicky.com is the domain used both for hosting the tracking code, as well as logging the incoming traffic data for your web site. If we just moved static.getclicky.com to a normal CDN, everyone who is self-hosting the code would no longer have their traffic logged unless they updated their tracking code (which will be pointing incoming traffic to a new sub-domain).

It would be literally impossible to get everyone who uses our service to update to the new tracking code (both for self-hosted folks, as well as "old school" folks). To maintain backwards compatibility for every single one of our users, we must have more control.

If we had it to do all over again, we would have used a fully static file from the beginning and all would be well. Unfortunately, we not do have access to a time machine.


And your point is...?

We're building our own stinking CDN. We're going to lease a few servers in key parts of the world - One in central or east US, one in Europe, one in Hong Kong, and one in Australia. And we're going to use Geo-Directional DNS so that a request for static.getclicky.com will resolve to the server that is closest to you geographically.

By building our own CDN, we can use SSL with our own domain, we can use dynamic scripts, and we can redirect incoming traffic log requests to the correct servers. All of these are critical components of a CDN for us, and none of them are met with any of the existing CDNs.


We need your help

We're very excited about this, but we need your help. We build our own servers and host them locally, but for this project, that's out of the question.

We need recommendations for companies in Central/East US, Europe, Hong Kong, and Australia, that lease servers with the following requirements:
  • Debian Linux 4 or 5 with full root access
  • Dedicated IP address
  • 4+ GB RAM
  • 2+ TB monthly bandwidth (with reasonable fee to upgrade to a higher level)
  • $250 USD or less per month
  • Decent tech support
  • Web site in English


If you have experience with any such companies in the markets listed above, please leave a comment here. We want to have this fully deployed by the end of the month, so your help is very much appreciated.

Note - for Europe, we're already leaning towards Hetzner, which is located in Germany. We've read quite a few good things about them, but that doesn't mean we won't go with someone else if you think someone else is better.
51 comments |   Mar 01 2010 10:56am

Search keyword rankings!

We have partnered with SheerSEO.com to provide ranking data from Google, Yahoo, and Bing for keywords and phrases that you care about for your web site.

To get started, go to the new "rankings" report under your searches tab and follow the instructions. It's not feasible to automatically track and monitor every search you get, so you have to enter in the specific words and phrases you want to monitor on SheerSEO. Once you do, you'll get reports like this in Clicky:




This shows the ranking for each keyword you have setup in all 3 search engines, as well as how many visitors you had for whatever date range you are looking at.

Here's the really cool part though. You can click on any of the numbers in one of the search engine columns and you will see both the rank of that keyword in that search engine, as well as the number of visitors you've had, graphed over the last 180 days. As you can see from the graph below, it's a very strong relationship. As the ranking gets higher, the number of visitors goes up.




Pretty useful, eh? However, do note that you will have to wait at least a week after you enter your keywords before you can view a historical graph, because they only update the rankings once per week, unless you pay for daily updates.

The only unfortunate thing is that lower ranking is good, not bad, but the graphing software we use doesn't have a way to invert the Y-axis. We may be looking into replacing the graphing software in the future, for this and other reasons (the API is real P.I.T.A.). Just remember: For the dark blue line, lower is better, for the light blue line, higher is better. Also, note that each line has its own Y-axis - rankings on the left, visitors on the right.

You can also click the number under the visitors column to automatically invoke filtering on Clicky for that search term to view the actual visitors for whatever date range it is that you're looking at.

Who is SheerSEO?

This company provides a number of other services related to SEO besides just keyword tracking. They are a pay service, but you get a full 90 day trial of the service before you have to pay anything. Their plans start at $7 per month, but Clicky users will get a special discount of just $5 per month if they only want the keyword tracking functionality. Regardless, it's a great service and well worth small cost in our opinion. Even if you don't want to pay extra for this functionality, you should still give the trial a shot. You'll get it for free for 90 days which is the longest trial I've ever seen for a web service.

Ideally we'd offer this data for no extra charge (with a Pro or higher plan), but we don't have the capability to do this ourselves right now. We have been using SheerSEO ourselves for quite a while so partnering with them made sense to bring this functionality to Clicky users. Hope you like!

Update: Originally we stated that we were unable to negotiate a discount for Clicky users, which we really wanted to offer. SheerSEO has come back and said they decided they will provide one for Clicky users. If you only want/need the keyword tracking functionality, the cost will be $5 per month instead of $7. This is a unique plan they are creating just for Clicky users, so we're happy about that and hope you are too!
16 comments |   Feb 25 2010 5:50pm

More details on organizations now available

We've been tracking visitor organization data for a long time, but if you wanted to learn more about any of them, you had to research that on your own. No more!

We have integrated data from jigsaw.com that will display more information about most organizations in the US, Canada, and the UK. It doesn't look like they have any data about companies outside of these countries, but I'm not entirely sure.

Anyways, you can access this new data from two different areas. The first one is the global organizations report. You will see a new magnifying glass ("search") icon next to each organization now:




When you click the icon you will see a list of potential matches:




Clicking on one will display the following data points about the company:





When viewing details on an individual visitor, next to where their organization is listed is a new link, "Lookup":





It works the exact same way, click to find potential matches and then click a match to view more details:




We hope you like it. We wish this data was available on a more global basis but this is the best we can do for now. Half of our customers are in the supported countries so we know at least half of you will like it :)
5 comments |   Feb 23 2010 4:18pm

Next Page »




Copyright © 2017, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS