Better bot protection and backups

Two complaints we receive fairly often are that too many bots get logged, and backups on Friday night are annoying. Well here I am on a Friday night letting you know that things are looking up!

Bots

Let me first be clear that this problem is not unique to Clicky. Most bots don't interact with Javascript, so most are not logged by Javascript-based trackers. We also have a fairly big regular expression that aims to filter out any that do the Javascript thing, and it works pretty good. I think we are definitely one of the best at filtering out bots already, but the complaints keep coming in. People see it as a defect of Clicky, even though it affects every tracker. And the bots keep getting trickier.

Both Microsoft and Google have started sending out bots in disguise in the last year or two, the theory being that they're ensuring your content doesn't appear differently if your web site thinks it's a regular visitor instead of a crawler. These bots have "real" user agents so you can't tell they're bots. However, a few people pointed out something unique - their user agent is always Windows XP / MSIE 6.0, and they always report a screen resolution of 1024x768. That alone is not enough to filter out a visitor - chances are good someone on IE6 has a real dinosaur of a computer on their hands - but since Clicky tracks organizations, we can dig deeper. When we look up the organization info for these visitors, if it's Google or Microsoft, we can be 99.9% confident this is definitely a bot. (Because if either of these extremely rich companies still seriously have computers this horrible that are used by employees... well, they should be sued).

The problem was however, we didn't look up the organization of a visitor until after that visitor was inserted into the database. But tonight, I re-arranged some things, and now we check for those three unique factors - XP, IE6, 1024x768 - before inserting into the database. If we have a match, we'll look up the organization immediately and pull a little preg_match("#(microsoft|google)#i", $organization) magic out of our hats, and if it returns true - BAM. Not logged.

There will still be bots who sneak through, I'm sure of that. However, Google and Microsoft seem to be the biggest "problems", and I've never seen what was obviously a bot from either of them that did not have XP / IE6 / 1024. They might update that in the future to make our lives more difficult again, but for now I'm confident this will eliminate almost all of the bots that we log that shouldn't be getting logged. Yay!

Backups

We do full database backups every Friday night starting at 10pm PST (GMT -7), during which traffic processing is halted. As the databases grow in size, these take longer and longer. We were investigating improvements to this process earlier this week, and I realized I was not setting a flag that would basically cut the time needed to do the backup in half. This was a horrible oversight on my part but I'll own up to it, and this has now been fixed. Most databases complete their backup in 1-2 hours, but some that are 3-4 years old were getting near the 4 hour mark. Now, the max any of them should take is about 2 hours, and most should be an hour or less.

But wait, there's more! We're going to be moving to a new database engine in the near future (goal: 3 months) that will be much more backup friendly. We won't have to halt processing at all while the backups are taking place. That will be a nice change. ^_^

Good night!
19 comments |   Jun 10 2011 8:19pm

Clicky is a launch partner for CloudFlare Apps

We met the CloudFlare team while we were at TechCrunch Disrupt last September, and we think their product as well as their co-founders, Matthew and Michelle, are awesome! CloudFlare was runner up for best startup at the conference, and they've been growing leaps and bounds since then.

CloudFlare sits between your web site and the internet, making it faster by acting like a CDN for your entire site, and more secure by blocking evil bots and the like. The original article on TechCrunch summarizes what they do quite well. CloudFlare is completely free to sign up for. They have paid upgrades with additional features, but there aren't any limits on free accounts for things like how much traffic you have. So if you think your site is slow, we wholeheartedly recommend trying them out.

This morning at TechCrunch Disrupt NYC, they announced their latest initiative, called CloudFlare Apps, which launches on June 1. It's launching with 20 services, and Clicky is one of them.

"CloudFlare's users have asked for a realtime analytics solution since we launched," said Matthew Prince, CEO and co-founder of CloudFlare. "Rather than reinvent the wheel and build our own, we are proud to offer a great solution like Clicky that lets any website owner see who is on their site as they click from page to page."

A common problem for many web services like Clicky is that people need to copy and paste code onto their site to start using the service. CloudFlare is trying to solve this with their new Apps service, which allows anyone using CloudFlare to install web apps like Clicky with a single click of their mouse. We get a lot of emails about this from new users, so it's a very real problem, and we think this is a great solution.

How do they do it? CloudFlare serves all of your web site's traffic, so they can automatically insert additional code into your HTML as it travels through their servers. So if you opt in to Clicky, they'll add our code at the bottom of all of your pages automatically as each one is served. And if we ever update the format of our code, CloudFlare will update it too, ensuring you always have the latest and greatest.

We think this is fantastic in all regards, so we're really excited to be a launch partner!

UPDATE! Someone tweeted that they were excited about this partnership because it meant they could finally use Clicky on their Posterous blog, which currently only allows Google Analytics in terms of Javascript tracking. We hadn't considered this, but the potential here is really great. There are tons of hosted services out there that don't allow you to add third party Javascript to them, but as long as you can slap your own domain on top of it, you'll be able to do this with CloudFlare. Very cool!
13 comments |   May 25 2011 10:35am

Facebook referrers update

A request we've had quite a bit is to show the actual page that a visitor has come from on Facebook. It's not that we were trying to hide it in the first place, it just didn't show anything beyond facebook.com. We decided to dig into this further today.

When you post a link on Facebook, whether it's in your feed or on a fan page, Facebook makes that link go through a redirect script first. There are several good reasons for this that I can think of that benefit both Facebook users and Facebook Inc., but we don't need to get into that right now. All that matters is any link that someone clicks on Facebook to an external site goes through their redirect script first.

Browsers normally keep referrers intact through redirects, but Facebook is doing something different. They are using Javascript to call document.location.replace() to send you on your merry way. This method works the same as if this intermediate page instead had a link to click on, and that link sent you to the external page: the end result being that this intermediate page "becomes" the referrer.

Out of curiosity, I wondered if you had Javascript disabled, since this intermediate page appeared to only use Javascript and no other method of redirect. Well, turns out with Javascript disabled, the links on Facebook actually point directly to the third party page. But... sigh... without Javascript, Clicky can't access the referrer at all.

So I hate to be the bearer of bad news, but it looks like what we have now for Facebook referrers is the best we're going to get. If you know of an analytics service that through some minor miracle actually gives you referrer data from profiles or fan pages, let us know, but I'm pretty sure it's impossible. Sorry!
8 comments |   May 20 2011 10:19am

The user experience and psychology of color [FIXED]

One of my most favorite articles ever written about Clicky is this one, which might seem entertaining at first glance, because it's basically ripping us a new one for a design decision I made almost 7 years ago (pre-Clicky) and that I have stuck with ever since. What decision is that? That whenever I want my software to give you any kind of feedback, whether good or bad, that it will be displayed in SKULL-MELTING BOLD RED.

Now of course, I know that people associate red with bad. For example, "You totally forgot to fill out a field! Nice going!". But I make web sites for a living, and while I consider my attention to detail to be off the charts, I have caught myself missing "messages" all the time that random web site X is trying to share with me, because the message doesn't BURN MY RETINA.

So what about Joe Sixpack? How many of these "messages" is he missing? Considering my background, I would guess he misses a lot more of them than me. And since these tend to be "important" things, the fewer he sees of them, the more confused he's going to be. And what does confusion lead to? Tech support! Now don't get me wrong... of course I'm absolutely not saying anything negative about anyone who emails us for support. But, the fewer emails we get, the more time we get to spend writing code, which is the ultimate dream of any software business. That's where the fun is!

So that's been my philosophy. It may not be right, but I wanted you to know the reasoning behind it. As we continue to grow, however, the amount of "#UI #FAIL" tweets and emails we get has increased substantially, and it has started to bother me. But the real tipping point... yes, the real tipping point was today, when one of our new developers, Alexander, was playing with his local install of Clicky, and he did something that trigged a "success" message that subsequently melted his skull. He said, "Success messages are red?". I gave him the spiel. He understood, but he emphasized how much his mood was spoiled. He was truly upset that Clicky was essentially yelling at him while at the same time it was congratulating him for successfully filling out a form.

I felt bad. I almost let him go home early! But then I decided enough is enough... it was time to fix this. So I spent the last 4 hours or so doing just that. This is the result (screenshot below). And based on the comments in the article I linked above, I took the time to make it color-blind friendly (using blue instead of green for "good" messages). There are still some areas of our site where I use red to make something stand out, e.g. a specific sentence in the middle of a paragraph in our help documentation, but otherwise you should find Clicky a much friendlier companion.

15 comments |   May 18 2011 10:17pm

User homepage totals

We've added a much requested feature to the user homepage: a row at the bottom summarizing your total numbers across all of your sites. This can add a bit of loading time to your user homepage though if you have a lot of sites registered, since it's showing today, yesterday, and the last 7 days. So if you don't need this feature and you feel this page loads too slow, you can disable it on your user home customization page.

Example screenshot is below of what this looks like. (For you mathematicians out there, the numbers don't add up because this is just a snippet of the top and bottom of my home page.) I also wanted to point out that the bounce rate is a weighted average, instead of just sumOfBounces/numberOfSites. Enjoy!

Update May 19: Due to popular demand, we changed it so this on top by default, instead of the bottom. If you want it on bottom instead, we added a new preference to put it down there, on your user prefs page. You can also turn it off entirely, as you could before, if you don't want it at all.

10 comments |   May 18 2011 3:17pm

Next Page »




Copyright © 2017, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS