Jigsaw company API integration is dead and we're looking for alternatives

About two weeks ago, our integration with Jigsaw broke. Jigsaw is (was) a crowdsourced database of information on businesses, mainly US ones but some in Europe and other places too. It let us provide you with information like this.

We looked into it and discovered that our API key got revoked. We did some digging and realized that Jigsaw had been bought out by SalesForce last year, and Salesforce has decided to kill free access to this API, which apparently became effective around September 1.

The API is still available, if you're willing to pay. We are definitely willing to pay for this kind of data, but the price is $25,000/year. That works out to just over $2,000/month, which would make it our biggest monthly expense, other than payroll. Sorry, can't justify that.

For now, we have modified the links that pulled in this data to instead just open up a Google search page with the organization name pre-filled. In some ways this is actually better because Google will almost always find the company in question, regardless of physical location, whereas with Jigsaw, when I wanted to look up info on a company it was only maybe 50% success rate. With Google you'll have to do a bit of work on your end to find the details that we were previously providing, but at least it's something.


If you know of an alternative service that is reasonably priced and includes an API, please let us know. We haven't found anything worthwhile as of yet. A lot of these services seem more geared towards providing you with "leads" at these companies, rather than just information about the companies themselves, which is not what we're really interested in at the moment.
11 comments |   Sep 17 2012 2:27pm

We're moving!

This Saturday, August 4, from approximately 2pm to 5pm PST (GMT -0700), our web site will be offline while we move our servers to a new data center in downtown Portland.

We have carefully planned this over the last 3 weeks to ensure that tracking will still be online during this time (no data will be lost and there will be no impact on your site's performance), and that the move itself will be as fast as humanly possible. The new data center is already pre-railed and pre-wired with power and ethernet, so de-racking and re-racking will be extremely fast. So why will it take ~3 hours? Well, the old data center is about 90 miles away... :(

New machines are already setup at the new data center to support tracking during the move. When the database servers get plugged in here, they will automatically start parsing the ~3 hour back log of traffic they will each have waiting for them on the tracking servers. It will take a good 3-6 hours from that point for all servers to catch back up with real time again.

This is something we've wanted to do for a while but as we grew to over 50 physical servers, it became unfeasible. However, thanks to our full virtualization that was completed in June after many months of work, we are down to just 11 physicals! Suddenly this dream became a real possibility so we jumped at the opportunity to make it happen before we needed to add any more hardware to the rack.

To say we're excited would be the understatement of the year. We've been with the same host as we've grown to enormous bandwidth over 6 years, so they've had to grow with us, which has been the cause of most of our major outages. The new data center is enterprise class with internet connectivity across 8 unique providers, so problems with internet connectivity should be near zero. Portland is much more major hub than where we were before, so connectivity should also be significantly faster, especially for those of you outside the US. Last, being 15 minutes away from our data center instead of 90 will be a very welcome change when we need to take a trip there.

A lot of time has been spent over the last 4 months on backend/sysadmin work like this, which has interrupted our regular flow of feature releases. We'll be back to that very soon, don't worry.
30 comments |   Jul 30 2012 12:44pm

New custom data report

If you log custom data with Clicky, you're probably going to like this new set of features. If you're not logging custom data, you should - it's one of our best features.

Up until yesterday, for our own reports on getclicky.com we have only been logging usernames of those of you logged in to our site. This adds a lot of personality to the visitor reports. But a lot of you log a lot more types of data such as shopping cart information, account status, things like that. I've been wanting to add summary reports for this custom data for quite a while, not only for you, but also because there were other types of data we were interested in seeing about who is using our service on a day to day basis. There wasn't much point though since there was no way to see a summary of it. But now there is!

If you log custom data, you will see a new item in the main tabs when viewing your site. And if you don't log custom data, this item doesn't show up. Here's what the main report looks like:

As you can see, when someone is logged in to Clicky, we're now also tracking what type of account they have and how long they've been a member. And we added support for attaching goals to this data, so we can see what goals the different types of accounts are completing, as well as their revenue (hidden here).

What I really love is that the sub-tabs for this Custom report are dynamically generated based on the different types of custom data you've been logging to your site. So you can click on any of those sub-tabs to see a report for just that family of data, or, you can just click the 'more...' link at the bottom of any family, just like other family style reports (browsers etc - speaking of which, some of those were broken because of horrible code, and have now been fixed as well as optimized to generate faster). (Note: Internally, data types that have "parents" and "chidlren" are called "families", in case you are confused).

You can click any of the items in this report to immediately see all visitors with that custom data attached to them, and you can also click any "parent", for example "account type", to see all visitors who have any "account type" data attached to them, no matter what its value is.

Not done yet!

The new goal report we released about 4 months ago has been a big hit. We thought it would be pretty great to see custom data in this report too, so we added it: (Screenshot has been slightly modified to make it smaller)

We also created a dashboard module for custom data, and the sub-tabs are dynamically generated just like they are for the main custom report:

Last but not least, all of these items are graphable too. Just click the trend percentage next to any custom data in any report (main report, dashboard, goal report) to see its history over time. Of course, we've only been logging this data for our stats for about 24 hours so it's not terribly exciting yet:

If you use custom data with Clicky, we think this will be a nice addition. And if you're not using it yet, you really ought to look into it. Full documentation is here.
10 comments |   Jun 21 2012 1:35pm

Cookies and you: An update

The EU cookie law goes into effect May 26. This is a great law because finally, all privacy issues will be permanently eradicated from the internet.

If there was ever an example of politicians really "getting" the internet, it would be this. The law requires that you list on your web site all of the cookies that get set by your web site, and the purpose of each cookie. Not even session cookies, or Javascript cookies that are never sent anywhere but simply used as boolean "flags", are exempt; and this makes sense, because cookies of this nature are infamous in hacker circles for their unparalleled ability to steal your credit card, read your email, and sleep with your wife. The law also requires you to get "opt in" permission from a visitor before any cookies can be set. Redirecting every visitor to a page with this information and the ability to opt in to cookies is a great solution, because every additional step between a visitor and a conversion increases revenue by 10x, according to a study from AreYouFreakingKiddingMe, LLC.

We just updated our privacy policy to list all of the cookies that may get set by your site if you have Clicky installed on it, in case you need it. If this law applies to your web site, we're sorry, but you can either explain all of this to your visitors and let them opt in, or follow the instructions here to disable cookies.

We've had the option to disable cookies in tracking for a while, however that was created back when we only had one cookie ("the" tracking cookie). Since then we have added several more, whose purpose is to make the code more efficient and save resources on our end from deactivated/non-paying site (these being the session/boolean cookies mentioned above). Up until now, disabling cookies had no effect on these extra cookies. Because they were never sent to us, we didn't think it would matter. And it shouldn't. But now it does. Our tracking code has been updated so all cookies will be disabled now when this option is set (except one, but that will only be set for sites that are no longer using Clicky but still have the code installed, which is against our terms of service).

UPDATE: A user pointed out that Cloudflare still sets a cookie. We use them for our CDN (static.getclicky.com). Their wiki says that cookies cannot be disabled. We have reached out to them for their plans regarding this law and will update you when we know more.

UPDATE: Interestingly, the BBC, one of the biggest UK web sites, specifically states on their new cookie settings page that they embed third party items such as Youtube and Flickr that set cookies and that these are beyond their control so those cookies will still be set no matter how they decide about cookies served directly by BBC. Does this mean that sites embedding third party content or services, such as Clicky, are exempt from having to worry about these kinds of cookies? Who knows, but it's interesting.
17 comments |   May 22 2012 3:47pm

Database server migrations over the next couple of weeks

We're migrating to a new server infrastructure over the next couple of weeks, which means all of our database servers will be finding new homes. While any given database server is being migrated, traffic processing will be halted, but the server will still be online so you can view existing data. Once the files have been copied, it will need to be taken offline for 10-20 minutes while some rsync'ing magic finishes everything up, then it will be brought back online with its new hardware. At this point it will be anywhere from 30 - 120 minutes behind real time, and traffic processing will then resume.

Anything that interrupts traffic processing is something we'd always prefer to do on the weekend. However, we can only do 2-4 servers per day and we have almost 50 database servers, so that would severely slow down the process. So, we are going to be doing it on weekdays too, and expect it to take 2-3 weeks total. But it will be worth it!
4 comments |   Mar 26 2012 5:09pm

Next Page »

Copyright © 2018, Roxr Software Ltd     Blog home   |   Clicky home   |   RSS