Advice on moving to the cloud?

From day 1 we've managed our own physical servers. We buy them and build them and configure Linux on them and throw them in the data center ourselves. This allows us to configure things exactly as needed, and is quite cost efficient. But man are we getting sick of it. So much time is spent dealing with servers!

People have often asked us why we don't use a cloud environment, such as Amazon. There's two main reasons. First of all, bandwidth - our load balancers are pumping almost 4,000 requests per second through them during peak hours, and a total of about 250,000,000 requests per day (which is almost 300GB of data - per day). Last I checked, Amazon only includes 5,000,000 requests per month, and you have to pay overages for extra. I don't know what the overage price is but considering we'll be over our limit 30 minutes into any given month, this is a big concern.

The second reason is reliability. Cloud servers are not really guaranteed from what I understand and they can pop offline at any time. This is a very big deal when it comes to large database being stored on this type of service. Some of our database servers have over 300GB of data, although the average is closer to 100GB. And we have almost 40 database servers. If our databases were in a cloud environment and one of them popped offline, it's going to need repairs done on it before coming back online, since it would not have been shut down properly. Not something I want to deal with.

So... do any of you have experience running high bandwidth services in a cloud environment? Doesn't have to be Amazon. If so, how do you find the pricing? Also, any experience with running a large database in the cloud? How have you found the reliability? Any advice from people with experience with either of these scenarios is much appreciated.

We're not necessarily going to do this. It's just under consideration. We just want some input first!

Update: The overall consensus is that we're insane to consider this, so it's unlikely to happen. It was just an option we were thinking about. Thanks for your advice.

One thing I was thinking about is what if someone like Google decided to move to a hosted cloud environment. Obviously they're 10000x bigger than us but still - that'd be insane right? To run a high bandwidth and huge I/O service, you need to go old school.
46 comments |   Oct 07 2010 9:23pm

More beta updates; and we're off to San Francisco

On Saturday, we're taking the train down to San Francisco for TechCrunch Disrupt. We'll be down there for an entire week. We'll try to be good about answering emails, but unless your message is urgent, your patience is appreciated.

We were really hoping to get the beta finished and turn it on for everyone before leaving, but it's not quite there yet. The only bugs left that we know of are very minor and are not holding us back, but there's just a bit more polishing to do. Also, we don't want to flip the switch and find out there's some major problem while we're on the road. We've been getting about 1,000 people a day consistently using it, which is great, but once that jumps to over 30,000, a serious problem may surface that we didn't know about. So, we're going to wait until we're back, which will be October 2. We'll probably spend a few more days working on it and then push it out to everyone.

Here's everything that's new since our last beta release:
  • When viewing Spy, it's always updated the document title to include the number of online visitors, but no other report did that. Now that we're always displaying this data at the top of every report (which we call "stats header" for short), it's included in the document title as well no matter what report you're viewing.
  • Speaking of stats header, Spy will now also update all of those numbers in real time, instead of just visitors online. When viewing any other report, they update once per minute instead.
  • Command-key support for Macs to open links in new windows
  • Clicking on a username or IP address in Spy now works
  • 1024 support - if your browser window is less than ~1100 pixels wide, the sidebar now auto-hides itself. Just move your mouse over it to show the full thing. This is one thing we want really feedback on. Only 7% of our users have windows 1024 or smaller, which is awesome, but we still want to provide a good experience for you. So let us know if the auto-hide thing is good for you. We think it could be better and we will improve it, but for now this is what we got. It's not really possible to move the tabs back on top, as we were originally thinking, because of the way a bunch of things work together in our new layout.
  • Fixed the popular filters form
  • Fixed bug when filtering visitors by page URL (this was pretty major - any link you clicked on after doing that was completely broken)
  • Tons of minor bug fixes

There is only one major bug right now, and that's that somehow Spy broke for MSIE. No idea why. And there's about 10 minor bugs left. We'll be fixing all of those once we get back, then putting a bit more spit and polish on this bad boy and pushing it out the door.
13 comments |   Sep 24 2010 7:29pm

Beta updates

Thank you to everyone who has tested our new beta interface and sent us great feedback on it. The vast majority of the feedback has been very positive, and for such a big change, that's very promising. If you haven't tried yet, you should.

We've been squashing tons of bugs and adding more functionality this week, and we just uploaded all of the changes. As usual, please test and report on any bugs (by leaving a comment on this post).

One of the biggest changes since the last release is our new user homepage. We heard you loud and clear, you want it to be cleaner and you want your sites above the fold. We used to allow you to hide the forum posts on the user homepage, but always left the blog there because we want as many people reading our posts as possible to stay up to date with what we're doing. With the new layout, there's no need for either of those options anymore, because those things no longer takes up unnecessary vertical space, as you can see from the screenshot below. We took all of the most important user links (upgrade, affiliate, etc) and put them in your face at the top, and put the blog/forum posts next to them. New blog and forum posts will also be highlighted in red so you can see when there's something new to read.

Changes since the last release

  • New user homepage, as mentioned above
  • Shift/control clicking a link should now open it in a new window/tab instead of still loading it via ajax
  • All links that open in a new window should now be fixed (quite a few were broken), except for links inside Spy - those are still broken but we hope to have them fixed tomorrow.
  • Many of the "next page" links at the bottom of reports didn't work, but they do now!
  • Spy basic has been added back in
  • When viewing "mega" Spy, there is no longer a dedicated Spy "version" of visitors online. It now updates the global value at the top of window instead.
  • We have quite a few javascript files. Most of them (the ones that rarely change) have been consolidated into one large file, which should result in much faster loading
  • Header stats now show up for MSIE or for any browser that loads a stats page directly (a bug was preventing this from happening before)
  • If you had scrolled down and tried to open the calendar, it didn't show up. Now it is fixed position so it scrolls up and down with you while it's open.
  • Tons of other minor bug fixes

Things coming soon

  • 1024 support
  • Sidebar menu options only change highlighting when clicked. If you click one, then click back, the old one stays highlighted. This will be fixed.
  • When viewing Spy, it updates the document title with the visitors online, but no other reports do even though we have a dedicated box at the top of every report showing this value. We'll be updating the entire reporting interface so this is done no matter what page you are viewing.
  • Links in Spy (e.g. clicking an IP address) will be fixed very soon!
  • We've had a lot of people ask us, why don't we move to a native (javascript) graphing engine and ditch the flash one while we're redoing the whole site? Well, we do want to do this, but it's a much bigger task than it sounds. There's a ton of backend stuff that would need to be updated. So I don't think this will happen before this is officially released. But we want to do this, so it will happen soon!
19 comments |   Sep 21 2010 6:13pm

Help test the new Clicky beta

The last month we have been been doing nothing but working on a massive overhaul of our entire web site. The goal was to make it faster, easier to navigate, and feel like a true web "application".

Check it out at (but please read the rest of this post too!)

What's new?
  • Every page loads via Ajax, including form submissions, which makes everything load lightning fast. (Except for IE - sorry, but this browser just has too many problems).
  • All navigation is now in a sidebar that changes depending on context. When you're viewing reports, it just contains your report options. (Click the Clicky logo to get back to your user homepage). Every other part of the site will show a different menu instead.
  • If your screen is at least ~800 px tall, the sidebar is in a "fixed" position so as you scroll up/down the screen, it's always there.
  • The sidebar helps us get important data as high up on the page as possible. For example when viewing reports, there used to be around 200 pixels of vertical space before the data you wanted was displayed. Now it's about 20 pixels.
  • When viewing reports, there are a few figures that are always at the top of every page (visitors online now, and total visitors and actions today). The visitor/action tallies have a sparkline graph next to them as well. These automatically update once per minute.
  • Lots of other stuff (I'm a bit pressed for time right now)

This has been tested extensively with all modern browsers - Firefox 3.6, Chrome 6.0, Opera 10.5, Safari 5, and IE7. We haven't tested with older versions of these browsers (or newer, in the case of Firefox 4 (in beta), and IE8/9). So we want to play around with this extensively and report on any bugs here, especially with browsers that we haven't tested.

This is not complete by any means, but we hope to have it released within the next week or two. There's still some ideas we have brewing, and there are some known bugs:

Known bugs/issues that will be fixed

  • This is slightly wider than 1024, which is the standard. Thankfully, only around 7% of our users have a screen that's 1024 or less pixels wide. But we will be adding support for 1024, don't worry. Either the sidebar will automatically hide, or we'll just move the tabs back to the top for these users.
  • The site may seem kind of slow on your first few page views, but that's because none of these files are on our CDN yet. That will change when this is released officially!
  • When viewing "Spy", the visitors online figure doesn't necessary match the global one at the top. The global one also updates slower. We'll be fixing this so when you view Spy, it will update both figures instead of just the one "inside" Spy.
  • The graphs at the top don't show up for MSIE yet
  • MSIE also has a few problems with some of the drop down menus that are loaded into the page via ajax
  • In some browsers, the "saved filters" drop down menu doesn't work right
  • The "filter results..." form at the top of any "popular data" report doesn't work
  • When you click on sidebar menu, the item you click is automaically highlighted. If you click "back" in your browser, that item remained highlighted.
  • The /user/ pages will probably get their own sidebar menu
  • If you're a white label user, there is some new and some changes CSS in here. We'll be giving you a 1-month window to create a new CSS file for the new interface, before the new interface goes live. Also, we'll be maintaining the "iframe=1" parameter, and when we see that we will automatically adjust to 1024 width, including probably putting the tabs back on top for the reporting interface).

Let us know what you think!
38 comments |   Sep 16 2010 1:25pm

Tracking server upgrades this weekend

The last few months we've been growing faster than we want to. That's a nice problem to have but it has its consequences. We've been getting a surge of high traffic sites using our service, so our tracking servers are really struggling right now. You may have noticed a bit of lag when our tracking code tries to log data to our home servers - that is a result of this. Of course, that data is logged asynchronously, which means it shouldn't interfere with anything on your site, but it's still annoying.

As we've grown there have always been software tweaks we could make to boost performance, but we're out of options now, these things are as optimized as they're going to get.

We apologize for this lag recently, but of course it's going to get fixed. This weekend, we'll be adding an additional 2 tracking servers into the mix (currently, there are 3). These 2 servers are much higher end than the existing ones, so it will be 150-200% capacity boost instead of just 67%. This will give us a lot of breathing room, which we could really use.

What's next?

The last 3 weeks we've been working on a massive overhaul to the entire web site. It started as just an overhaul to the stats interface, but we decided the entire web site could use an update. This update is awesome but it's also a lot of work. We hope to have a beta available late next week for you to play around with, as we expect there to be some bugs (particularly with older browsers), so we'll need as many people to test it as possible. We'll be announcing it here and on Twitter as soon as it's available!
21 comments |   Sep 01 2010 1:07pm

Next Page »

Copyright © 2019, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS