BY indefiniteloop

It’s all good that I’ve started playing with the UI of this blog. But, I did like to know what works, and what doesn’t. Google Analytics gives me metrics that let me know what’s being read. Inspectlet gives me how this blog/site is used, what do my readers do, while on this blog. So, the question now becomes… How do I measure the effectiveness of the UI changes I make to this blog? What are the metrics?

To answer those two questions, I would have to dive in, and think about what do I consider as goals? What do I want my readers to do on this blog? Then, I need to find the best UI changes that make attaining those goals possible.

To do that, I’d have to start with A/B, & Multivariate testing. And so, this post is exactly about that!

What’s A/B & Multivariate Testing?

A/B testing, or split testing is essentially comparing two versions of a web page, or in the case of this blog - a post, to see which of the versions retain readers for a longer time on this blog. Provided the fact that retaining the readers for a longer duration, on this blog, is a goal.

Multivariate testing is essentially the same as A/B testing but it’s more granular. Which allows you to test, and compare more variables, thus revealing more information about how the variables on a particular A/B test page/post add to achieving the above mentioned goals.

There are many tools out there that let you do this. The best, and most simple ones are paid, and are used daily by a lot of businesses to fine tune their conversion rates, sign-up rates, etc. A/B, and Multivariate testing is also used on mobile applications to test features, and funnels.

This blog on the other hand, does not have much in terms of conversation rates. The UI changes I am working on are more of an experiment than anything else, meant to teach me more about UI, UX, design, and A/B testing in general.

With that being said, I don’t want to spend on a monthly subscription for carrying out experiments with this blog (yea, I am a cheap-skate that way). In that regards I set out to hunt for free A/B, and Multivariate testing tools that could work with a Jekyll blog. Below is the list of what I found, and below the list of tools/frameworks there’s another list of resources, and articles that I thought worth a mention here.

Another thing to note here would be that this blog is statically generated with Jekyll. This limits my options further, but surprisingly there are plenty of A/B, and Multivariate testing tools to choose from. Some really old, others more recent.

8 Free A/B Testing Tools That Will Work With Jekyll

  1. PlanOut by Facebook: It’s feature packed to say the least. Has great documentation, and is used by Facebook, for Facebook. Currently there are only two ports ready for production use, viz. Ports for Java, and PHP. The JavaScript, Go, and Ruby ports are still under development. Read how Facebook uses PlanOut. While the framework looks awesomely promising (I’d never think of saying that for something that Facebook developed), I’d have to wait ‘til the JavaScript port, or the Ruby port to be released so that I could work with it, using Jekyll.
  2. SixPack: One of the good ones out there. This A/B testing framework is language-agnostic. You can use it with any language. What I like about SixPack is that it includes a very nicely designed dashboard for viewing the results, and making decisions. To be able to use SixPack across multiple web services, written in any language, you would have to first install the SixPack framework which is the Sixpack-server. The Sixpack-server receives requests from your web services or sixpack-client pages (multiple client libraries are provided). There’s a third component - Sixpack-web, which is the dashboard included in the framework. It’s a full-fletched framework, and it comes awesomely packed with a web interface. It’s updated frequently. What’s holding me back from using SixPack is the fact that the setup will take it’s own sweet time. If I was testing conversions for a e-commerce website, or sign-ups for a web-app in production, etc. I wouldn’t blink twice before using SixPack. But since it’s a static blog we’re talking about, I am gonna give SixPack a pass.
  3. Cohorts: A simple, pure JavaScript based multivariate testing framework. Cohorts works hand in hand with Google Analytics. It has a simple API to work with, and you have an option of delinking Google Analytics, if you don’t want use GA. Thus enabling you to use it with other datastores too. The downside is that it hasn’t been updated in the last 5 years (at the time of writing this post). Which doesn’t mean that it doesn’t work. It does. But since the last 5 years, time has passed, and there are better options available out there - just look at point 1, and 2 above. I will not be using Cohorts.
  4. Clutch.io: Clutch.io is from Twitter. Or rather was. It’s old, and hasn’t been updated in a long time. None the less it’s still out there, and it can primarily be used for native A/B testing your iOS or Android Apps. There are two parts to clutch.io; a) Native A/B testing for iOS and Android apps. b) Toolkit for testing hybrid native/HTML apps for iOS. And since this blog is not an app, we shall not use Clutch.io for that reason. I was under the impression that Clutch.io had a component for web app testing too, but alas I made a mistake. Thus technically, it’s seven free A/B testing tools!
  5. Abba: Abba is a straight forward, easy to use, and easy to setup (maybe) A/B, and multivariate testing tool by Stripe. Its client-side API is JavaScript based. Abba is a self-hosted framework. It does come with a dashboard that lets you filter by date, and browser. Setup requires Ruby 1.9.3 and Mongo - that’s the time consuming part, and this is why I mention that it “maybe” easy to setup. Maybe because it depends on how you want to set this one up. If I were to use Abba, I would be using their documented Heroku 10 seconds setup process, that’s why easy to setup. The JavaScript API is simple to use, and test with. What I like about Abba is that it’s really simple to use, understand and you can associate “weightage” with a particular test variant, thus making it possible for some variants to be used more often than others. Abba is one of the frameworks I am giving a serious thought to. I haven’t decided on it yet because the last commit was about two years ago. Not that I will hold that against it. But it’s good to have options. I am comparing Abba with another framework mentioned below (see pt 8).
  6. Genetify: Another JavaScript based A/B testing tool. This one’s abandoned too. And it’s the oldest one in this list. Last major commit was around 7 years ago (at the time of writing this post). The more recent commit of about 2 years ago, is for removal of .DS_Store files. Genetify allows developers to do any number of tests, be it CSS, JavaScript or HTML based. The framework, over time improves it’s own decisions of serving up variants based on a Genetic Algorithm backend. Genetify is composed of two components - a) The JavaScript client library (frontend), and the Genetify backend. Too much remains unknown for this framework. The idea was very sound, using a genetic algorithm to decide which variants to use while serving-up pages. So much so that Genetify got a mention by John Resig. So maybe you reading this, yes you! You can fork it, and start tinkering with it again! I won’t be using this framework anytime soon, because the documentation is broken, links are broken, etc. I mention it here because of the idea that the framework is based upon.
  7. AlephBet: AlephBet is a pure JavaScript based framework. It offers pluggable backends (defaults to Google Analytics, and localStorage), is cache-friendly, and is less than 5kb minified with no dependencies. AlephBet was inspired by Cohorts, and Optimizely. The JavaScript API is easy to understand, and use. AlephBet is for developers, and as such doesn’t offer any dashboard, or some such. You make decisions based on looking at raw data after running tests. It’s in active development, with the last, most recent commit being 7 days old at the time of writing this post.
  8. easyAB.js: Developed by Romain Strock, this is the first jQuery plugin for A/B, and multivariate testing that I’ve came across. The most simplest one to use out there! If you’re already using jQuery, and would like something small, and fast to work with then this should be your choice. It’s really simple to use, and the API is quite easy to understand. It uses Google Analytics as the backend, like a couple of frameworks mentioned above. easyAB.js uses cookies for the delivery of the variants. Thus it will default to the original page if cookies are disabled. This is what I am comparing Abba with, because of the frameworks simplicity, ease of use, and availability.

Lessons Learned from 100,000 A/B Tests

Here’s a not-so-long video about the importance of A/B testing on the TNW channel on youtube. Dan Siroker is the presenter at the TNW Conf. Europe 2015.



A/B Testing Resources, Articles, & Posts Worth A Mention

Expect more posts on A/B testing, UI, UX, and the design of this blog. For now I am still torn between Abba, and easyAB.js.

Hope this helps someone!

Prev
Next
Newsletter Series

About The Author:

Home Full Bio

© All content reserved. indefiniteloop.com