Sifter: an API for website testing and optimization

I’m happy to finally give Sifter a name and a home. It’s a multi-armed bandit (MAB) API for testing web pages, optimizing which version (“arm”) of the test gets displayed. These arms could be article titles, ad positions, logos, colors, etc. Here’s the gist of how how a MAB test works:

• A user visits a page that is under test.
• The page makes a request to sifter asking for an arm to display.
• Sifter makes a calculated decision about which arm to display and returns the result to the page as a value between 0 and (# arms - 1).
• The page renders the selected version (specific colors, logos, layouts, etc).
• The user interacts with the page in one way or another. Maybe they sign up, maybe they buy something, maybe they don’t.
• When the interaction is complete, send the reward earned (if any) back to Sifter. This could be something binary like click/no click, or some other value such as the amount of money spent.
• Sifter updates the bandit algorithm, influencing the arm that is selected the next time the page is rendered.

There is an endless number of uses for something like this. Here are a few off the top of my head:

Run a news website?

• Run a few different title options for an article on your front page.
• Figure out how to get users to read more of a paginated article:
• Test the number of words shown on each page (say 500 or 1000)
• Each time the user clicks “next page”, update the default test result value to the user’s progress through the article. (See the /select_arm route in the docs for more on default values)

Run a website with promoted content?

• Allow advertisers to choose multiple promotions to run at the same time, and optimize which one gets shown based on clicks and/or user feedback.
• Test the amount of labeling around the fact that this content is promoted vs organically created by users.

Test the audience demographic response.

• Set up a standard A/B test.
• Display the same content to every user.
• Report back the results of the test where the “arm” represents a specific user demographic, rather than an alteration to the web page.
• Your test results will show which demographic responds best to your content.

There are a lot of features that I think are pretty clever, like setting a default test result value and a TTL for the test, so that the default value gets reported when the test “expires,” and the ability to update this default value and TTL by sending a “heartbeat.” And there’s a lot on my To Do list, such as confidence intervals, multivariate testing, bucketing/aggregating test results for high-traffic websites, and more.

Right now the project is in a limited beta mode. I’m running it on a few pages that don’t get much traffic and I would love some help working out whatever bugs may come up. If you’re interested, please get in touch!

An API for website testing/optimization

This is something that I’ve been working on very heavily for the last 6 weeks or so, and I’m excited to start showing it off. It’s currently being hosted at http://banditapi.herokuapp.com.

In a normal A/B test on a website, you randomly present users with one version of a feature or another. This could be the placement of your logo, for example. You let the test run for some amount of time, then measure the relative performance of the two versions (“arms”) and go with the better one from that point on.

Multi-Armed Bandit tests are different. They are designed to learn which arm is preferred, and move towards that version of the website as quickly as possible. There are many nuances to these types of tests, and I highly recommend John Myles White’s book on the subject (there’s also a video of him talking about this subject at tumblr recently).

The API that I built provides a couple of different Multi-Armed Bandit algorithms, which are exposed as simple routes to request an arm to play, and to update the algorithm with the test results. There’s also a little admin view to see how your test is performing:

I’m at the point now where I need some help testing. This thing is not at the point where you should deploy it on a production site, but if you have something to test out that doesn’t take a lot of traffic (and you don’t mind a little latency), it should be reliable. There’s a also a little client library on github if you just want to fuss with it. Either email me at myfirstname.mylastname@gmail.com, or send me a message through tumblr to get an account.

New year, new projects.

Happy new year! 2012 was a pretty crazy year for me. The biggest thing was probably moving into my great new apartment with Lana, but I also worked on a bunch of great projects, including:

• Led a team at the DataKind hackathon, which had great results.
• Speaking on a panel at the Data Gotham conference.
• Contributing a chapter to Bad Data Handbook.

The framework is an API for A/B and various bandit algorithm tests, where a web app can request a version of the page to display, and then report the results of the test back, thus influencing the version shown to the next user. The idea for this was inspired by John Myles White’s excellent book on the subject (watch his talk about it here).

It’s been fun to have a big project to work on again. Let’s hope 2013 turns out as well as 2012.