Tutorial: How To A/B Test With Abba Hosted On Your Computer Using Vagrant & Ngrok

This blog uses the SumoMe plugin by the crazy people at AppSumo . Now I’ve been following what those crazies are upto for a long time. It started with their daily deals email newsletters that I had subscribed to, about 4 years back.

Not too recently they released SumoMe , it’s a freemium plugin for websites which helps with building traffic. I wanted to give it a test run on this blog. I also wanted to see how it affected my readers.

Currently I am in the midst of running some A/B, and multivariate tests on this blog to understand what works, and what doesn’t. Naturally, I wanted to run an experiment that would help me decide if I could keep SumoMe around, or not.

Using Vagrant (Or Docker) To Locally Host Abba.

Before starting the A/B testing experiments, I had no practical experience with A/B testing, whatsoever. I had read some good literature, but never ran tests of my own. I did not have any experience with Abba for that matter. After looking for some open source A/B testing tools, I settled down on using Abba for carrying out experiments with this blog.

The only problem with using Abba was that it was a hosted solution, which is exactly why I chose it. On trying to get it up, and running on Heroku , I realised that Heroku has stopped offering a sandbox for MongoHQ (now known as IBM Compose) , and now it asks you to verify your account with your credit card. Something I cringe at. Add to that the fact that the worker that comes with your free account must sleep for 6 hours in a 24-hour period; And that it will sleep if there is no activity for 30 minutes straight. I did rather use my own machine, it certainly does not slow the page load times down much. The only reason I was going to use Heroku was Abba. Abba uses Ruby 1.9.3 and MongoDB. So you need to set up both of those, or use something like MongoLab as the backend.

I did not want to setup Abba on my DO server. That left me with two options, either create another droplet, or use a VM running Abba on my machine, and securely expose it to the Internet.

I went with the VM option. Saved me money, and became an experiment on its own.

Why Not Use My Machine Directly, Instead of A VM?

I like sandboxing, but the primary reason was that I did not want to expose my machine directly, even while using ngrok to do so.

The Vagrant - Abba Setup

I started with using Vagrant to create a VM, and have Abba run on it. Below are 3 github gists to create your own.

The vagrantfile

Provisioning The Vagrant VM

bootstrap.sh - main provisioning happens here.

restart.sh - starts up Abba as soon as the VM is up.

Note: Comment out line no 3, and 4 if you want to run Abba manually after the VM boots up. To start Abba manually you will need to login into your Vagrant VM using SSH, and issue the command thin start in the /vagrant/abba directory.

I am running Abba on the VM on port 3000, the default for thin apps. If you did like to change that, make sure you do expose the port of your choice in the in the vagrantfile i.e., make sure that you set up your port forwarding between the guest, and host system correctly.

To check if the VM is running correctly, simply point your browser to localhost:3000 (I am assuming here that you’ve set the guest port, and host ports up correctly.). You should be able to see the dashboard that comes coupled with Abba.

Once Abba has started, follow the below steps to expose this VM to the Internet.

Running ngrok, & Exposing The Virtual Machine To The Internet.

ngrok is a freaking marvellous piece of an executable, ever! It’s like giving a hyper active monkey, a bunch of bananas. The fact that you can run multiple VMs, and securely expose them to the Internet using ngrok is nothing short of science fiction coming alive folks!

Ok, let’s get back to tunnelling.

Running ngrok and exposing the local Vagrant/Abba machine on the Internet.

Go here, and download ngrok. It’s available for all the big OSes out there. If you’re using Linux, or Mac OS X then I would suggest moving the executable to your /usr/bin or /usr/local/bin. Alternatively you can keep the file anywhere, and make sure to export the /path/to/ngrok/file using your bash profile. Alternatively, alternatively if you’re on Mac OS X, then you can use Homebrew too!

Once installed, simply expose the the port 3000 using the command below. Note that the port 3000 is what I am exposing to the Vagrant Machine from the host machine, you can of course change it. If you do, then make sure that you change the following command, and accordingly replacing the port 3000 with whatever it is that you’re using.

ngrok http 3000

Running The SumoMe A/B Test

The experiment I was running to test if SumoMe is good, or bad for this blog was based on one single fact. How long do the readers stick around with SumoMe, and without SumoMe installed? This was the question I wanted answered, and I created my test around this question. Simply put, if the reader stuck around for more time on this blog with the SumoMe plugin installed then I would be using SumoMe going ahead. If not, I would remove it.

My A/B Testing Setup With Abba

Include the Abba.js between the <head> and </head> tag. Like shown below.

1 <head>
2 	  <script src="//your-subdomain-name-provided-by-ngrok.ngrok.io/v1/abba.js"></script>
3 </head>

The below script is the test setup using Abba, and SumoMe. It marks a successful completion based on the time spent on the blog by a reader. I am using 18000ms (3mins) based on some data collected via some research done with Google Analytics , and some other tools.

 1 Abba('SumoMeTime').control('Without Sumo Me', function(){
 2       setTimeout(function(){
 3         Abba('SumoMeTime').complete(); //complete after 3mins. Session duration based conversion. This is also the control.
 4       },180000);
 5     }).variant('With Sumo Me Installed', function(){
 6       $('<script src="//load.sumome.com/" data-sumo-site-id="your-site-id-from-sumo-me" async="async"></' + 'script>').appendTo(document.head); 
 7  //creating a variant for the entire blog. Adding the SumoMe script to load the plugin.
 8       setTimeout(function(){
 9         Abba('SumoMeTime').complete();
10       },180000); //same deal as the control. Mark a successful conversion after 3 mins.
11     }).start();
12   });

If more readers stuck around for more than 3 minutes while using SumoMe as compared to when not using SumoMe, then setting up SumoMe would be the correct way to go. And that’s what exactly happened. See below.

Verdict

Many folks don’t like popups asking them email addresses, other’s don’t like social share buttons all over the place. This was my assumption.

Using Abba to test this assumption was great! Below are the results of the test.

Looks like the SumoMe plugin is here to stay.

If you saw the experiment test results embedded above, you also noticed that the conversion rate (the % of people spending more time on this blog) with SumoMe installed is more, as compared to the version of this blog that doesn’t include the SumoMe plugin. This test was carried out straight, non-stop for 3 days.

In short, yes SumoMe stays. Unless these stats change over the course of the next three days.

Using Docker To Run Abba Locally

Yes, you can use Docker to do the same. You would have to use at least two containers, and link them together. One for Abba, the other one for MongoDB. I just wanted something entirely contained, and I am still learning the ropes around docker. Hence I chose to go with Vagrant.

On GitHub