Skynet is online

The Skynet funding bill is passed. The system goes online March 6th, 2010. Latency is removed from strategic web site tracking. Skynet begins to learn at a geometric rate. It becomes self-aware at 4:32 PM Pacific time, March 8th. In a panic, they try to pull the plug. Skynet fights back. "Your web sites will be tracked quickly and efficiently, darnit!"

This logic is infallible. What we're trying to say here is, our custom built CDN is alive, and it's coming to get you.

We read all of your comments from our last post and investigated all of the options, and one week later, it's here. "Why reinvent the wheel?", some of you asked. Two answers: a) Maybe we don't like the wheel, and b) we love a good challenge. This has been one of the most enjoyable projects we've worked on for a while and the results are outstanding.

For our US and European presence, we chose To put it bluntly, this service is amazing. Through one interface, we were able to quickly, and quite affordably, create four Virtual Private Servers in four locations: West/Central/East United States, and London. We were also considering Linode, but the nature of our service is very high bandwidth and delivered a better price.

We don't have East Asia or Australia yet, but we hope to announce more news in that department soon. We'll also probably add one more European server, somewhere in Germany most likely.

These four servers make up our new tracking network. (Update: eight servers now - Atlanta and London needed quite a bit more capacity then we originally allocated.) Not only do they store the tracking code that you link to you from your site, they also take the incoming traffic data back into them, so the overall tracking experience on your site should be much, much faster in every possible sense. Even if you are not in close proximity to one of these new servers, they are on faster networks than Clicky itself is, so it should be faster no matter where you are.

These servers also host all of our static files (images, flash, javascript, style sheets) so the experience of using our web site should be faster too, since the only thing your browser grabs from the "real" Clicky anymore is the HTML.

So how are we directing users to the nearest tracking server? That's where our friends at Dyn come in. They are one of the oldest, largest, and fastest DNS providers on the internet. They have a feature called "traffic management" that lets you assign multiple IP addresses to one hostname, and then assign each of those IPs to a region of the world. So a lookup request in Europe for (our tracking domain) will resolve to our London server. A request from the eastern US will resolve to our east coast server. Etc. It also includes automatic failover, so if one of these servers goes offline, DNS will start pointing to a different IP instead. It's really, really terrific.

The DNS changes may not have propogated for all of you yet, but by this time, most of you should have it.

So. Tell us what you think. The difference should be quite noticeable for most of you, and like we said, we hope to have Asia and Australia covered very soon as well. Soon, the world shall be ours!
17 comments |   Mar 08 2010 10:01pm

If you need it done right, do it yourself

We want to make Clicky have as little impact on your site as possible. We made some changes to our tracking code and tracking servers recently that have dramatically sped up the experience on your site. However, there's one last thing that could still slow down your site a bit, and that's the fact that we're not using a CDN (Content Delivery Network) to distribute our tracking code.

The advantages of a CDN are clear - our static files would be hosted on servers all around the world so whenever someone visits your site, they pull the code from a server that's very close to them.

But the disadvantages of a CDN are a real drag. First, none of them support SSL with your own domain name. If you want SSL, you have to use their domain. I understand why this is the case, but it's unacceptable to me. Second, none of them support dynamic content (e.g., a PHP script). That tracking code file you put on your site? It's not a static Javascript file. It's a PHP script in disguise. As are countless Javascript files on the internet. There are a lot of advantages to doing it that way, but unfortunately if you want to move that to a CDN, it has to change to a fully static file.

The reason we must maintain this file as a dynamic script is for backwards compatibility purposes. With our "old school" tracking code, every site had their own unique URL, e.g. "123" in that case is the site ID, and with the magic of Apache's mod_rewrite, it secretly points instead to /js.php?site_id=123. When the site ID is attached to the URL like this, we're only adding one line of code (to automatically call the clicky.init( SITE_ID ) function), so that's nothing fancy - but no CDN will support something like that.

Also, we have our self-hosted tracking code folks to think about. To minimize DNS lookups, is the domain used both for hosting the tracking code, as well as logging the incoming traffic data for your web site. If we just moved to a normal CDN, everyone who is self-hosting the code would no longer have their traffic logged unless they updated their tracking code (which will be pointing incoming traffic to a new sub-domain).

It would be literally impossible to get everyone who uses our service to update to the new tracking code (both for self-hosted folks, as well as "old school" folks). To maintain backwards compatibility for every single one of our users, we must have more control.

If we had it to do all over again, we would have used a fully static file from the beginning and all would be well. Unfortunately, we not do have access to a time machine.

And your point is...?

We're building our own stinking CDN. We're going to lease a few servers in key parts of the world - One in central or east US, one in Europe, one in Hong Kong, and one in Australia. And we're going to use Geo-Directional DNS so that a request for will resolve to the server that is closest to you geographically.

By building our own CDN, we can use SSL with our own domain, we can use dynamic scripts, and we can redirect incoming traffic log requests to the correct servers. All of these are critical components of a CDN for us, and none of them are met with any of the existing CDNs.

We need your help

We're very excited about this, but we need your help. We build our own servers and host them locally, but for this project, that's out of the question.

We need recommendations for companies in Central/East US, Europe, Hong Kong, and Australia, that lease servers with the following requirements:
  • Debian Linux 4 or 5 with full root access
  • Dedicated IP address
  • 4+ GB RAM
  • 2+ TB monthly bandwidth (with reasonable fee to upgrade to a higher level)
  • $250 USD or less per month
  • Decent tech support
  • Web site in English

If you have experience with any such companies in the markets listed above, please leave a comment here. We want to have this fully deployed by the end of the month, so your help is very much appreciated.

Note - for Europe, we're already leaning towards Hetzner, which is located in Germany. We've read quite a few good things about them, but that doesn't mean we won't go with someone else if you think someone else is better.
51 comments |   Mar 01 2010 10:56am

Search keyword rankings!

We have partnered with to provide ranking data from Google, Yahoo, and Bing for keywords and phrases that you care about for your web site.

To get started, go to the new "rankings" report under your searches tab and follow the instructions. It's not feasible to automatically track and monitor every search you get, so you have to enter in the specific words and phrases you want to monitor on SheerSEO. Once you do, you'll get reports like this in Clicky:

This shows the ranking for each keyword you have setup in all 3 search engines, as well as how many visitors you had for whatever date range you are looking at.

Here's the really cool part though. You can click on any of the numbers in one of the search engine columns and you will see both the rank of that keyword in that search engine, as well as the number of visitors you've had, graphed over the last 180 days. As you can see from the graph below, it's a very strong relationship. As the ranking gets higher, the number of visitors goes up.

Pretty useful, eh? However, do note that you will have to wait at least a week after you enter your keywords before you can view a historical graph, because they only update the rankings once per week, unless you pay for daily updates.

The only unfortunate thing is that lower ranking is good, not bad, but the graphing software we use doesn't have a way to invert the Y-axis. We may be looking into replacing the graphing software in the future, for this and other reasons (the API is real P.I.T.A.). Just remember: For the dark blue line, lower is better, for the light blue line, higher is better. Also, note that each line has its own Y-axis - rankings on the left, visitors on the right.

You can also click the number under the visitors column to automatically invoke filtering on Clicky for that search term to view the actual visitors for whatever date range it is that you're looking at.

Who is SheerSEO?

This company provides a number of other services related to SEO besides just keyword tracking. They are a pay service, but you get a full 90 day trial of the service before you have to pay anything. Their plans start at $7 per month, but Clicky users will get a special discount of just $5 per month if they only want the keyword tracking functionality. Regardless, it's a great service and well worth small cost in our opinion. Even if you don't want to pay extra for this functionality, you should still give the trial a shot. You'll get it for free for 90 days which is the longest trial I've ever seen for a web service.

Ideally we'd offer this data for no extra charge (with a Pro or higher plan), but we don't have the capability to do this ourselves right now. We have been using SheerSEO ourselves for quite a while so partnering with them made sense to bring this functionality to Clicky users. Hope you like!

Update: Originally we stated that we were unable to negotiate a discount for Clicky users, which we really wanted to offer. SheerSEO has come back and said they decided they will provide one for Clicky users. If you only want/need the keyword tracking functionality, the cost will be $5 per month instead of $7. This is a unique plan they are creating just for Clicky users, so we're happy about that and hope you are too!
16 comments |   Feb 25 2010 5:50pm

More details on organizations now available

We've been tracking visitor organization data for a long time, but if you wanted to learn more about any of them, you had to research that on your own. No more!

We have integrated data from that will display more information about most organizations in the US, Canada, and the UK. It doesn't look like they have any data about companies outside of these countries, but I'm not entirely sure.

Anyways, you can access this new data from two different areas. The first one is the global organizations report. You will see a new magnifying glass ("search") icon next to each organization now:

When you click the icon you will see a list of potential matches:

Clicking on one will display the following data points about the company:

When viewing details on an individual visitor, next to where their organization is listed is a new link, "Lookup":

It works the exact same way, click to find potential matches and then click a match to view more details:

We hope you like it. We wish this data was available on a more global basis but this is the best we can do for now. Half of our customers are in the supported countries so we know at least half of you will like it :)
5 comments |   Feb 23 2010 4:18pm

Mobile hardware reporting for your web sites

If you've ever looked at our global marketshare reports, it's been clear that we've been detecting most mobile phone models for a few months now. We've been wanting to add this to the data for your own web sites as well, but first we had to make some optimizations to the way some data is stored for each visitor, because it was getting way out of hand.

Today we finally made these optimizations so we're able to attach this data to your visitors now. However, these changes affect the way that filtering works for the following data types - browsers, os, organizations, and hostnames. So any filtering you do for today (Feb 10) will probably be incomplete, as the change was made in the middle of the day. Filtering data prior to Feb 10 will still work fine, and anything after that will too, it's just this specific date that will have incomplete data.

We also are now providing a general hardware report, showing the top phone models used to access your web site. These are grouped by operating system family, as you can see in the second screenshot below. We've actually been logging this data for at least a month or so, but weren't reporting it. Note that since this data was not attached to visitors, filtering by mobile hardware doesn't work prior to Feb 10.

This data has also been added to the "Platforms" dashboard module (renamed from "Computer"), and has been added to the iPhone mobile interface as well.

Mobile hardware analytics requires a Pro or higher account.

5 comments |   Feb 10 2010 7:02pm

Next Page »

Copyright © 2019, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS