Microsoft's spam bot has been blocked

We've been getting more and more complaints of how annoying Microsoft's crawler for is. In case you haven't seen it yourself, what it does is pretend that it's a person and spams your sites with tons of hits and fake searches. Check out the screenshot below to see the kind of traffic we see in our own stats every single day.

But no more. Anytime we see a referrer from, we check if the IP is in the range of known IP addresses for their crawler, and if so, we just ignore it.

Microsoft ought to be ashamed of themselves for this type of behavior. This has been going on for a long time (just search Google about it) but it seems to have gotten a lot worse recently. I don't know if they're trying to falsely inflate their market share when people view their stats or what, but whatever the reason, one thing is certain - I'll never use for anything, ever.

17 comments |   Apr 09 2009 12:27pm

Mega spy out of beta + new features for free accounts

Mega spy has come out of beta. It is now just "Spy", and the old Spy is now called "Spy basic". This also means the new Spy is now only available to Pro members, as was originally announced.

We didn't have time to make as many improvements as we wanted but don't worry, we will be visiting this again in the near future. We did manage to squeeze in the top 2 most requested features though - linking to external searches, and showing the visitor's location in a tool tip when you hover over their flag in the right hand side.

For consistency we've been trying to make it so ALL external links in your stats are accessed via that little grey arrow you all know so well, and all other links (in your stats) are generally for navigation or to apply visitor filters. Spy's new layout was too cramped we felt to do this all with, so now all external links in the action stream (to your own pages, or to referrers/searches) just link directly to those external pages.

New features for free users

We've been wanting to offer some more features to our free users for a while now. We really wanted to offer the basic version of Spy to all of you - but it just sucks up a ridiculous amount of resources supporting that, so we can't do it. The load averages on our tracking serveres were averaging more than 4x their typical load during the last 2 weeks when all sites had access to both versions of Spy. So we can't give you that, I'm sorry.

But we did give you 2 more features: Bounce rates, and IP tags and filters (but you are limited to 10 max). Also "most active visitors" was for some reason a pay-only feature, which I don't ever remember deciding on, and it's not like it's the most amazing feature in the world that will get everyone to sign up for paid service. So that, along with bounces and ip tags/filters, are now available to free members.
17 comments |   Apr 01 2009 6:06pm

Mega Spy now in public beta

Mega Spy is now in public beta, which means all users have access to it.

Go to your Spy page, then click the "test the mega spy beta" link to check it out. It still has some issues and is not yet feature complete, but we've drastically increased the performance since the original beta was released on Friday. It now runs very fast on all browsers, including MSIE. This will also really help you out if you have a slow computer.

By this coming weekend we plan to add in support for goals and campaigns, and maybe organizations and hostnames, if the performance hit on the server end isn't too bad. We'll also be trying to add a nice highlight effect on the popular data view, so you can more easily tell which data is actually changing when it refreshes.

Free accounts also have access to both versions of Spy during the beta period. Once you get a taste, we think you'll have a hard time resisting that upgrade. Mega Spy will be a Pro-only feature when the final version comes out. Blogger accounts will only have access to the old Spy. Sorry, but all the new stuff that the new Spy is doing is quite intensive on the server end. If we could offer Clicky for free to everyone - we would!

Please leave all feedback as a comment on this post. Enjoy!

28 comments |   Mar 16 2009 11:53pm

Mega Spy preview + private beta

Mega Spy has taken longer than planned to develop but I promise the wait will be worth it. We're still not quite done, but we know how eager you are, so we're releasing a beta version to get feedback before we finalize the 1.0 release.

Mega Spy will be a Pro-only feature, but during beta it is open to all users. To be eligible for the private beta (we'll open it up to everyone else in the next 3-5 days), all you need is an existing Twitter account with at least 10 followers. To apply, simply tweet the following message: is the best web analytics service on the planet!

Or, make up your own tweet, and tag it with #megaspy. For example: - live web analytics FTW! #megaspy

We will be monitoring for these tweets via Twitter's search engine, and will send a direct message (tonight or tomorrow) to the first 50 eligible participants with instructions for accessing the beta.

UPDATE - private beta is now closed, invites have been sent out to everyone.

Mega Spy still has some bugs, and it needs some speed optimizations, so keep that in mind! It has been tested with Firefox 3, IE7, Opera 9.5, Safari 4, and Chrome 1.0, and works in all of them. Any lesser version may or may not work. A lot more things are going on in the background compared to the old Spy, so a browser with stellar Javascript performance will enrich the experience. MSIE is pretty bad, Firefox and Opera are decent, Safari and Chrome fly. Once Firefox 3.1 comes out though it should fly as well.

Here's a couple screenies to whet thy appetite. The second screenshot shows my absolute favorite feature of Mega Spy - seeing what pages on your web site people are looking at RIGHT NOW. As people move around your site, the values update on the fly. It's a blast to watch. You'll also notice that you can filter the data you want to see by many different criteria. When filters are applied, the entire view is relative to those filters. You can also filter down to a single visitor to watch just them navigate your site, but that's not shown here.

31 comments |   Mar 12 2009 2:14pm

Filter visitors by page, better segmentation - and more!

We just released an update that's been two months in the making. On the surface, there are only a couple of major differences you'll notice, but they are very cool ones for sure. What took so long was designing and implementing an entirely new system for how we store page URLs, and converting all of the existing data into that new format (many hundreds of millions of database rows). This fixed a couple of annoying problems we've had with the system from the beginning, and let us build some awesome new features on top. Let's get to it.

Filter visitors by page!

You can now filter your visitors by a page they have viewed. This has been a big request. Works in the API too, just use the parameter "href" with the "visitors-list" type.. You should only be sending the path and query (e.g. everything AFTER ""), unless the page you want to filter by is on a mirror or sub-domain, or it's an outbound link. For example here's how we'd grab all visitors who saw our /users/ page, and all visitors who clicked our company's web site in our footer (

New visitor segmentation!

When you filter your visitors by anything, the only type of summary data you used to see was basically everything from "the basics" dashboard module, but applied to these visitors. But now you can see anything you want about them - top searches, top countries, etc. This is particularly useful when filtering by page. Now you can see the top searches hitting any page on your site for example. We plan to add on to this quite a bit this coming week, so for now it's slightly rudimentary, but it performs its purpose well. Click on the red "View these visitors..." link (shown below) to choose what type of data you want to see. This feature is for Pro members only.

Links so hot you'll want to eat them!

Links to external content now hide the protocol by default, which saves precious space (particularly on the dashboard). We've also changed it so it highlights the domain name and the rest is slightly faded. This is inspired by the Google Chrome address bar - I really like how easy it is to pick out the domain when looking through links now.

Also, when viewing your visitors list, there is now an icon next to the referrer to indicate which type of referrer it is (search, social media, etc).

Links to external content have changed!

Used to be when you viewed your "links" or your "content", clicking on an item would take you directly to the external page. That's useful but inconsistent with how the rest of the site works, where clicking on anything else (search, country, etc) would instead filter your visitors by that item. Links and content have been changed to match this behavior in all areas of the site. Now, anytime there is an external link, there will be this arrow next to it. Click that arrow to be taken to the external link. You can see this in the screenshot above.

Much better support for sub-domains and mirrors!

If you had multiple domains pointing to the same site, or had the same code on a bunch of sub-domains for one site, you couuld still track that fine, but the "content" we tracked was never linked to the right domain that it was actually viewed on. Now, we'll store the domain it was viewed on, if it's anything other than what you have in your "hostname" field in your site prefs. Example:

No more dupes!

The new pages system gets rid of the dupe problem for your content. Unique pages used to be determines by a combination of the URL and the title. This resulted in a lot of duplicate entries for the same page, because of reasons like the visitor having Javascript disabled (no title), the visitor using an old browser sending us the title formatted slightly differently, etc. This will happen no more, although it will only affect your stats from this point forward.

In conclusion!

That covers all the new features that are obvious on the surfage. We hope you love them like we do, but as always leave your feedback, good or bad, below.

Also, there are a few gotchas that might catch you off guard. Let me cover that. First, pages will not have any titles until they get a page view under the new system. This is related to getting rid of the dupes problem. Give it a few days and almost all of your content will have titles again. Obviously your popular content should already.

Second, filtering by page may not work right away for all of your pages, because there were so many duplicate IDs. If "your" ID for a URL was not the "first" ID for that URL in our system (on your database server), then they don't match up right in the database. So filtering by a page URL may only work for some of your pages from "this point forward" (as I seem so fond of saying). I do have an idea of how to fix this so filtering will work for all pages, for any date, but it'll have to wait until tomorrow so I can test it.

Last, the new segmentation drop down menu doesn't work right in MSIE. I could not figure out the problem before release, but I will try my hardest to get it fixed on Monday.

Ok, now go enjoy the new features!
25 comments |   Feb 22 2009 7:46pm

Next Page »

Copyright © 2018, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS