Adam Laiacano

Baseball by the (jersey) numbers

Tue, 30 Apr 2013 09:03:00 -0400

The other day I saw this video of a 14 year old high school basketball sensation. He’s billed as “The Next LeBron” who was in turn billed as “The Next Michael Jordan.” What caught my eye is that they all wear number 23 on their jersey (until LeBron moved to The Heat, at least).

When I was a kid I played lots of sports, and the best basketball player on my team was always #23 and the best football player was always #34 (Bo Jackson, Walter Payton, etc). So that got me to wondering what the “desirable” numbers are in other sports. I asked about other sports on twitter and #10 is clearly the best number to wear in soccer, and maybe #9 for hockey (Richard, Hull, Howe).

I wanted to figure out which jersey numbers have been the best historically, and baseball is the obvious sport to turn to for this, since data is so reliable and readily available. So I downloaded the career stats for a little over 17,000 players (which I think is every baseball player ever) and decided to see which are the best jersey numbers over all.

For each player, I got their career batting average and their jersey number, which proved to be harder than I expected. For example, Johnny Damon wore #18 from 2002-2009 (during his best seasons with the Red Sox and Yankees), but has also worn 51, 8, 22, and 33. So to make it a little easy on myself, I just got the number that they wore on the most teams. Johnny Damon has had 7 numbers on 6 teams, but wore #18 on 4 teams, so I’m associating him with #18.

For scoring the jersey number, I’m taking the mean of career batting averages for all players who wore that number, weighted by the number of plate appearances per batter. I perform this weighting so that someone like Roberto Clemente who hit .317 over 10,211 plate appearances would have more influence than someone like Buster Posey, who hit .311 with only 1,324 plate appearances. That’s a bit of a mouthful so here’s some math that might make more sense:

\[ S_j = \sum_{i=1}^{N_j} \frac{p_{j,i} b_{j,i}}{\sum_{k=1}^{N_j}p_{j,k}} \]

Where $ S_j $ is the score for jersey number $ j $, $ b_{j,i} $ and $ p_{j,i} $ are respectively the batting average and number of plate appearances for the $ i^{th} $ player who wears jersey number $ j $, and $ N_j $ is the total number of players who wore jersey number $ j $. I was also sure to limit the data set to $ p_{j,i} \geq 500 $ and $ N_j \geq 5 $. There were also a lot of older (~100 years ago) players whose number I couldn’t gather, so I dropped those as well. This narrowed the data set down to about 3,500 players.

The next thing to consider is that since I’m using lifetime batting average and number of plate appearances to rank players, what I’m really doing is ranking hitters. Many pitchers in the National League end up with over 500 at bats, but their batting average is just going to hurt the overall ranking of their jersey number. The following graph makes this crystal clear. The y-axis shows the jersey number, and the x-axis is the batting average for each player.

So once we remove the pitchers, here’s where each number ranks in terms of weighted batting average:

There are some clear winners here. The number 51 is a bit of an outlier because there are only 10 batters with that number who meet the minumum plate appearance requirement, but the list includes Ichiro Suzuki (.321) and Bernie Williams (.297) among others with high batting averages. Here are the top players who wore #4, a more “classic” number:

            name plate_appearances batting_average
  Rogers Hornsby              9480           0.358
      Lou Gehrig              9663           0.340
Riggs Stephenson              5134           0.336
  Dale Alexander              2736           0.331
     Babe Herman              6228           0.324
    Luke Appling             10254           0.310
     Hack Wilson              5556           0.307
    Paul Molitor             12167           0.306
    Smead Jolley              1815           0.305
         Mel Ott             11348           0.304

Not a bad group to be in. There are also some clear loser numbers, but they’re mostly higher numbers which I think are often worn by pitchers.

Here are the the 10 best and worst lifetime batting averages:

            name number plate_appearances batting_average
  Rogers Hornsby      4              9480           0.358          0
    Ted Williams      9              9788           0.344          0
      Bill Terry      3              7108           0.341          0
      Lou Gehrig      4              9663           0.340          0
      Tony Gwynn     19             10232           0.338          0
Riggs Stephenson      4              5134           0.336          0
      Al Simmons      7              9518           0.334          0
      Paul Waner     24             10766           0.333          0
  Dale Alexander      4              2736           0.331          0
     Stan Musial      6             12717           0.331          0
...
    Corky Miller     37               575           0.188          0
   J.R. Phillips     17               545           0.188          0
    Bill Plummer      8              1007           0.188          0
         Gus Gil     18               538           0.186          0
    Brandon Wood     32               751           0.186          0
     Drew Butera     41               531           0.183          0
      Kevin Cash     17               714           0.183          0
      Tommy Dean      3               594           0.180          0
       Ray Oyler      1              1445           0.175          0
   John Vukovich     16               607           0.161          0

If you’re a baseball fan, you can see that the bottom is all populated by pitchers, because no .890 batter would ever get 950 at bats in the majors. In fact, baseball’s obvious bias towards keeping good hitters and cutting poor hitters is obvious when you plot lifetime batting average against total plate appearances:

So what number would I wear if I were a pro baseball player? I can’t NOT choose #9, even though it’s retired by the Red Sox.

Also, all of the code and results for this blog post are available on my github page.

The blogroll « Statistical Modeling, Causal Inference, and Social Science

Mon, 29 Apr 2013 09:27:00 -0400

The blogroll « Statistical Modeling, Causal Inference, and Social Science:

A great list of blogs on statistics, visualization, politics and more. Curated by Andrew Gelman. I’m not sure how Simply Statistics didn’t make the list.

SADDLE: Scala Data Library

Tue, 02 Apr 2013 13:44:41 -0400

SADDLE: Scala Data Library:

I sent out a tweet last night asking if there are any good data libraries for Scala. The next morning, Saddle gets released. I’ve heard it described as “Pandas for Scala,” which is great because nothing beats having data frames ported to a new language.

"All models are wrong, but some are useful."

Sun, 31 Mar 2013 09:49:00 -0400

“All models are wrong, but some are useful.”

- George E P Box (1919-2013)

How to approach a problem: self-indulgent music recommendations.

Sun, 24 Mar 2013 10:00:00 -0400

I’ve been thinking a lot about music recommendations lately, and I realized that I’m usually a little bearish about listening to recommended bands that I’ve never heard of before. Maybe it’s just because I listen to a pretty broad variety of music, but I love re-discovering a band that I know but haven’t thought of in a while. So with that, let’s build a 100% self-centered music recommender. The goal is to remind myself of some bands that I might like to play next based on what I’m listening to right now.

Fortunately for me, I’ve used last.fm to record the last 135,000+ tracks that I’ve listened to over the course of 7 years (my “now playing” is listed at the top of this page). And even more fortunately, they let you grab your entire history via their API. I was actually able to get 127,873 of them, which is more than plenty to work with. So let’s check out which artists I’ve listened to the most and see how well it matches up with my last.fm profile:

Artist                 Plays
----------------------------
Tom Waits              3155
Justin Townes Earle    2613
Iron & Wine            2053
M. Ward                2005
Lucero                 1832
Old 97's               1761
The Black Keys         1755
Beach House            1624
Death Cab for Cutie    1592
Ryan Adams             1527

The first thing that should become clear is that I listen to a lot of Sad Bastard music. At least both Dillinger Four and Samiam are in the top 20.

Approach

When designing this recommender, I’m going to try to answer the following question: Given the artist I’m currently listening to, what have I generally listened to next?

Now since this is specific to me, I’m going add a few constraints. The first of which is this: I prefer listening to full albums rather than individual tracks. I’m not going to recommend songs, I’m going to recommend artists. Because of that, the only important attributes I need for each track are the time that I listened to it and the artist.

First attempt

Let’s look at the song-to-song transitions. That is, given that I’m listening to a song by The Antlers, which band am I likely to listen to next? This table shows the number of times I transition from listening to an Antlers song to each artist on the list, as well as the probability of the transition.

                  artist  transitions  transition_prob
             The Antlers          854         0.870540
             Beach House           22         0.022426
             Arcade Fire            5         0.005097
          Patrick Watson            4         0.004077
The Tallest Man on Earth            3         0.003058
          Okkervil River            3         0.003058
      The Avett Brothers            3         0.003058
             Carla Bruni            3         0.003058
                 Pinback            2         0.002039
         North Highlands            2         0.002039

This simply confirms what I stated earlier: when I listen to music, I listen to full albums. 87% of the time that I listen to an Antlers song, I listen to another one of their songs next. That’s not helpful for recommendations, so I’ll add another constraint: I’m only interested in transitions where the artists are not the same. Now the above list looks like this:

                  artist  transitions  transition_prob
             Beach House           22         0.173228
             Arcade Fire            5         0.039370
          Patrick Watson            4         0.031496
The Tallest Man on Earth            3         0.023622
      The Avett Brothers            3         0.023622
          Okkervil River            3         0.023622
             Carla Bruni            3         0.023622
                 Pinback            2         0.002039
         North Highlands            2         0.002039
           Fleetwood Mac            2         0.015748

The order is the same as before, but the transition probabilities are much higher. This is a reasonable list of artists to recommend to someone who listens to The Antlers. Even last.fm has Beach House and Okkervil River in the top related artists.

Modifications

We’re doing well so far, but let’s see if we can make it a little better with just a bit more work. Beach House is the top recommendation, but I listen to them a lot . Of all of the tracks I’ve recorded, 1.27% of them are Beach House tracks. Considering there are a total of 1,342 unique artists in my data set, that means I’m $\frac{0.0127}{1 / 1342} = 17 $ times more likely to listen to Beach House than the “average” band.

So let’s use this information by dividing each transition probability by the unconditional probability of listening to a given artist. The unconditional probability is simply the total plays for each artist divided by the total number of plays (1624/127873 = 0.0127 for Beach House). The equation for the ranking has now become the following, where $ Pr(artist | Antlers) $ means the probability of listening to a given artist immediately after listening to The Antlers:

\[ \frac{Pr(artist | Antlers)}{Pr(artist)} \]

When I divide by the unconditional probability, I will give weight to artists that I listen to less often overall, making the results a little more exciting. If I multiply by this probability, however, I’ll give extra weight to artists I listen to more often, making the results more familiar. It’s probably important to point out that this is a bit of a hack that I came up with while typing up this blog post, and shouldn’t be confused with Bayes’ Theorem even though it looks sort of related. Anyway, let’s see what these rankings look like:

                original             less familiar               more familiar
------------------------------------------------------------------------------
             Beach House               April March                 Beach House
             Arcade Fire              Broken Bells                   Tom Waits
          Patrick Watson               Beach House                 Arcade Fire
The Tallest Man on Earth             Army of Ponch              Patrick Watson
      The Avett Brothers                   Mineral    The Tallest Man on Earth
          Okkervil River               Carla Bruni              Okkervil River
             Carla Bruni               Arcade Fire                    Bon Iver
                 Pinback           North Highlands          The Avett Brothers
         North Highlands    The Murder City Devils              Dillinger Four
           Fleetwood Mac                   Pinback              Band of Horses

You can see that Tom Waits moved up on the chart on the right because I’ve listen to him more than anybody else. For the middle list, however, there’s lots of stuff that didn’t even make the original cut. A band like Army of Ponch might not seem like the best recommendation to someone currently listening to The Antlers, but I’ve made that transition twice and might want to again.

While we’re at it, here’s the list of recommendations for Bruce Springsteen:

           original               less familiar          more familiar
----------------------------------------------------------------------
        Chuck Ragan                 Buddy Holly              Tom Waits
     Built to Spill            Buckingham Nicks            Chuck Ragan
     Camera Obscura                   Sam Cooke         Built to Spill
          Tom Waits                The Jayhawks                  Wilco
              Wilco                 Chuck Ragan         Camera Obscura
     Okkervil River           Bridge and Tunnel         Okkervil River
         Mean Creek              Built to Spill    Death Cab for Cutie
Death Cab for Cutie                    Mastodon               Old 97's
       Dan Auerbach              Camera Obscura             Ryan Adams
  Bridge and Tunnel    Iron & Wine and Calexico                  Spoon

Just reading that list reminds me that Mean Creek has a new record out that I’m going to listen to right now.

Performance

So which list is best? How does it compare to standard information retrieval techniques? Well, that’s probably different for each person and the only way to find out is to test it. I could (and might eventually) put together a little app that recommends me some artists from my listening history based on what I’m listening to right now. With a simple A/B test, I could see which of the three recommendation algorithms I follow most often and stick with that one in the future. To do that, I would have to record

The artist that is currently playing
Which artists recommendations are displayed
Which recommended artist (if any) was played next

The recommendation algorithm that provides the highest play / display ratio is the one I’d like to go with in the future. This seems like an obvious place to plug sifter for performing A/B and other types of testing in scenarios like this.

Conclusion

The point of this blog post is more about the thought process than the technical parts of the recommender. There are lots of things that I could have done “right,” like using properties of Markov Chains (which is essentially what I built) to improve the system, or account for the fact that Buddy Holly follows Bruce Springsteen in my music library, so maybe that isn’t a true transition.

I think the main takeaway is really in the constraints that I put on the system. The idea for this one-day project followed the following course:

Build a music recommendation system
Build a music recommendation system that only uses my last.fm data
Build a music recommendation system that only recommends music I already know
Build an artist recommendation system, not songs
Only recommend artists that I’ve listened to immediately after the given artist
Come up with a few simple variations and test them for performance

Identifying the problem correctly let me build something in just a few hours. Is it the best recommender system the world has ever seen? Actually, it might be, because I’ve never seen any recommender that only suggests content that you are already familiar with and that’s what I wanted. But we’d have to test it against the likes of Last.fm, Spotify, and Pandora to find out.

As I said in the beginning, I’ve been thinking about this stuff a lot lately, but that’s not to say I’ve put this method into production anywhere. The code for all of this was done using python/pandas, and breaks pretty much every rule that I laid out in my previous blog post, so I’ll clean that up and get it posted soon.

"So, yes, lasso was a great idea, and I didn’t get the point. I’m still amazed in retrospect that as..."

Wed, 20 Mar 2013 09:19:00 -0400

“So, yes, lasso was a great idea, and I didn’t get the point. I’m still amazed in retrospect that as late as 2003, I was fitting uncontrolled regressions with lots of predictors and not knowing what to do. Or, I should say, as late as 2013, considering I still haven’t fully integrated these ideas into my work. I do use bayesglm() routinely, but that has very weak priors.”

- It’s always impressive to see a leader in any field admit when they were “way way wrong” about a certain technique or idea. This post by Andrew Gelman is a great summary of what Lasso (or regularization in general) is, and how his opinion on it has changed over time.

Tibshirani announces new research result: A significance test for the lasso « Statistical Modeling, Causal Inference, and Social Science

Keeping tabs on your data analysis workflow

Thu, 14 Mar 2013 14:43:00 -0400

I’ve been thinking a lot lately about data analysis workflows, from raw materials to a finished product. Here at Tumblr and at my previous job at Columbia Business School, I’ve had to create a ton of one-off reports that would “probably never be used again.” The problem is that an awful lot of those “one time” requests have come back to show their head a month, or even a year later, and suddenly you have to read through old code (that was written in a hurry) in order to figure out what exactly was in “counts2.csv” and how it was populated.

To avoid this problem, you should always make sure that you can go from raw/messy data to a finished result with one command. I generally have, at the bare minimum, a bash script to execute a series of commands to re-generate my results. The script will do any or all of the following things, in order:

Delete all of the previous temporary or final result files that have anything to do with this project. You need a clean slate.
Gather the data, whether it’s from an API, database, Hadoop job, or any other source, and munge it into the right format (usually .csv).
Build any models you may need, or summary statistics, or anything else.
Generate the report with a script.
If you send the results to anybody, put them in a .zip file with all code that you need to re-create it.

If you did it right, you should be able to clone your repository (you are using version control, right?) onto a new computer that’s set up with the right tools and libraries, execute your build.sh script, and get your final results back with no errors along the way. Trust me, your future self will appreciate the extra effort.

Here are some tips for making this process easier:

Avoid the command line

I love ipython and the R console as much as the next nerd, but you have to be careful in there. If you have a line like

In[342]: results.to_csv('final_output.csv')

then you’re going to have a hell of a time when you need to run those 342 commands again. If you’re working on the command line, make sure you’re copy/pasting the right code into a script while you’re working. Scrolling through a terminal history to see what worked and what didn’t is never fun. RStudio makes it really easy to do this. Just type your code in the editor and hit control+enter to execute any highlighted lines, rather than typing them into the console window.

Use Literate Programming

ipython notebook and knitr (for R) make it easy to describe what you’re doing while you’re doing it. You can easily combine descriptions with code, results, and graphs. They’re often used to present final results (we use knitr to generate some daily email reports), but this is as good for your own code documentation / commentary as it is for producing the report that you send off to someone else.

Broader workflows

The actual report generation is usually the tip of the iceberg. There’s a lot more that happens before that, and if you’re like me it’s usually spread out among multiple scripts, unix tools, and programming languages. For example, we at Tumblr use scribe to log events, and then store the archived data in HDFS. So if I want to run an analysis that includes some of today’s actions, I need to get data from two different places (the scribe server and HDFS), and probably treat it slightly differently. Here’s an example of gathering data from two places, combining it together, and generating a report.

https://gist.github.com/alaiacano/5163726

There are some more tools out there to help with this, such as ProjectTemplate for R projects, which executes a series of R scripts in an organized manner to gather, munge, model, and export data. There’s also age old GNU make, which Mike Bostock recently endorsed for data projects. This provides the ability to list dependencies for specific scripts and data files. In the above example, generate_report.R relies on combined_data.csv, which relies on both clean_data_${TODAY_DATE}.csv and archive_data.csv. With make, you can specify all of these dependencies and if something throws an error along the way, the whole project will hault.

Have any other tips or tools for this? I’d love to hear them in the comments.

In celebration of what I have come to call “Hey, math!...

Thu, 14 Mar 2013 11:21:00 -0400

In celebration of what I have come to call “Hey, math! Amirite? Day”, go read about how to estimate the value of $ \pi $ using Monte Carlo simulation.

(via Happy Pi Day, Now Go Estimate It! « Zero Intelligence Agents)

"Any data scientist worth their salary will tell you that you should start with a question, NOT the..."

Thu, 07 Mar 2013 15:37:10 -0500

“Any data scientist worth their salary will tell you that you should start with a question, NOT the data. Unfortunately, data hackathons often lack clear problem definitions. Most companies think that if you can just get hackers, pizza, and data together in a room, magic will happen. This is the same as if Habitat for Humanity gathered its volunteers around a pile of wood and said, “Have at it!” By the end of the day you’d be left with a half of a sunroom with 14 outlets in it.”

- Jake Porway on how You Can’t Just Hack Your Way to Social Change

Some more shameless self promotion

Wed, 06 Mar 2013 10:55:38 -0500

I’m going to be giving two talks in the next few weeks:

An overview of our data stack at the NYC Data Science Meetup at Foursquare HQ on March 28.
A keynote talk at the Big Data Techchon in Boston April 8-10

I hope to see a bunch of you there. Feel free to say hello if we haven’t met.

Using entropy to route web traffic

Fri, 01 Mar 2013 10:02:00 -0500

Earlier this week, Blake asked me for some help with a problem he’s working on. He has a couple of hash functions that are being used to route web traffic to a number of different servers. A hash function takes an input, such as a blog’s url, and outputs a number between 0 and 2^32. Say we have 1000 servers, that means that each one will handle about 430 million points in the hash-space.

The data looked something like this (with fake blog names, of course):

##       blog.name        H1        H2        H3        H4
## 1 23.tumblr.com 3.137e+09 1.866e+09 6.972e+08 5.792e+08
## 2 19.tumblr.com 1.875e+09 2.545e+08 2.606e+09 1.312e+09
## 3 34.tumblr.com 1.366e+09 2.236e+09 1.106e+09 3.640e+09
## 4 43.tumblr.com 2.639e+09 1.098e+09 8.755e+08 1.507e+09
## 5 90.tumblr.com 6.564e+08 5.397e+07 3.084e+09 2.961e+09
## 6 29.tumblr.com 2.476e+09 4.532e+08 2.787e+08 4.894e+08

One important thing to point out before we get started is that this has to be a representative sample of the request data. Despite the wild popularity of my personal blog, it doesn’t get a sliver of the traffic that Beyonce gets. That fact needs to be represented in the sample data, meaning that her blog should appear in more rows of the sample data than mine.

Plot the data

The first thing I ever do is plot data to get a sense of what I’m working with and what I’m trying to accomplish. The density plots below show the distribution of values in the hash space for each algorithm. If you’re not familiar with kernel density plots, you can imagine this to be a smoothed (and prettier) version of a histogram. For the electrical engineers out there, it’s the sum of the convolution of a kernel function (usually a gaussian), with an impulse function at each of the points on the x-axis (represented here by dots).

By comparison, here are the density plots of a “near-ideal” example (1000 pulls from a uniform distribution) and a bad example (all assigned to the value 2e+09). The worst case example here shows the shape of the kernel function.

Calculate the entropy

In information theory, entropy is the minimum number of bits required (on average) to identify an encoded symbol (stay with me here…). If we’re trying to transmit a bunch of text digitally, we would encode the alphabet where each “symbol” is a letter that will be represented in 1’s and 0’s. In order to transmit our message quickly, we want to use as few bits as possible. Since the letter “e” appears more frequently than the letter “q”, we want the symbol for “e” to have fewer bits than “q”. Make sense?

Huffman Coding is one encoding algorithm. There’s an example implementation here, which assigns the code 100 to “e” and 1111110010 to “q”. The more “uneven” your symbol distribution is, the fewer bits it will take, on average, to transmit your message (meaning you’ll have a lower entropy). The entropy value is a lower bound for the actual weighted average of the symbol lengths. There are special cases where some encoding algorithms get closer to the entropy value than others, but none will ever surpass it.

The actual entropy formula is:

\[ H(x)=-\sum_{i=0}^{N-1} p_i log_2(p_i) \]

Where $ H(x) $ is the entropy, and $ p_i $ is the probability of that symbol $ i $ will appear. In the example I linked to above, $ p_e = 0.124 $ and $ p_q=0.0009 $, so it makes sense that e’s symbol is so much shorter. In the example, the average number of bits per symbol is $ \sum S_i p_i = 4.173 \frac{bits}{symbol} $, where $ S_i $ is the number of bits in the symbol. The entropy, from the above equation, is $ 4.142 \frac{bits}{symbol} $.

The example problem of web traffic distribution is a little different. We’re not actually encoding anything, but rather trying to make the theoretical lower bound for average number of bits/signal as high as possible.

We can consider each server to be a symbol, and the amount of traffic that it recieves is decided by the hash function that we’re trying to choose. If one of our servers is the equivalent of the letter “e”, it’s going to be totally overloaded while the “q” isn’t going to be handling much traffic at all. We want each symbol (server) to appear (receive traffic) equally often.

So to calculate the entropy, we’ll take a histogram of the hash values with 20 buckets (representing the 20 servers). That will give us the number of requests that go to each server. Dividng that by the total number of requests gives us each server’s probability of handing the next incoming request. These are the $ p_i $ values that we need in order to calculate the entropy. In code, it looks like this:

calc.entropy <- function(hash.value) {
    h = hist(hash.value, plot = FALSE, breaks = seq(0, 2^32, length.out = 21))
    probs = h$counts/sum(h$counts)
    print(probs)
    entropy = -sum(probs * log2(probs))
}

The entropy values for our four hash functions are:

##   hash.function entropy
## 1            H1   4.203
## 2            H2   4.226
## 3            H3   4.254
## 4            H4   4.180

And while we’re at it, here’s the entropy of our best/worst case example that we plotted earlier.

##    hash.function entropy
## 1     near.ideal   4.309
## 2 worst.possible   0.000

Why is the worst case value 0? Because if all traffic is going to one server, we wouldn’t need any bits at all to tell us which server the request is going to. The theoretical limit for a histogram with 20 buckets is: $ -20\frac{1}{20}log_2{\frac{1}{20}} = 4.32 $, which we’re close to but can never exceed.

All of our hash functions appear to be working pretty well, especially for such a small sample size that I’m using for this blog post. It looks like our winner is H3.

To summarize here’s what we did to find the optimal hashing function:

Get a representative sample of your web traffic
Run each request through the hashing function
Take a histogram of the resulting values with N bins, where N is the number of servers you have available
Divide the bin counts by the total number of requests in your sample to get the probability of handing a request for each server
Calculate the entropy, $ H(x)=-\sum_{i=0}^{N-1} p_i log_2(p_i) $, for each hash function

There are more considerations to take, like setting an upper bound on the value of $ p_i $ to ensure that no single server ever gets so busy that it can’t handle its load.

If you want to read up more on information theory, Elements of Information Theory by Thomas Cover and Joy Thomas is an excellent book that is reasonably priced (used) on Amazon.

There is also, of course, Claude Shannon’s landmark paper from 1948 “A Mathematical Theory of Communication”“, in which he essentially defines the entire field.

Sifter: an API for website testing and optimization

Thu, 21 Feb 2013 10:08:00 -0500

I’m happy to finally give Sifter a name and a home. It’s a multi-armed bandit (MAB) API for testing web pages, optimizing which version (“arm”) of the test gets displayed. These arms could be article titles, ad positions, logos, colors, etc. Here’s the gist of how how a MAB test works:

A user visits a page that is under test.
The page makes a request to sifter asking for an arm to display.
Sifter makes a calculated decision about which arm to display and returns the result to the page as a value between 0 and (# arms - 1).
The page renders the selected version (specific colors, logos, layouts, etc).
The user interacts with the page in one way or another. Maybe they sign up, maybe they buy something, maybe they don’t.
When the interaction is complete, send the reward earned (if any) back to Sifter. This could be something binary like click/no click, or some other value such as the amount of money spent.
Sifter updates the bandit algorithm, influencing the arm that is selected the next time the page is rendered.

There is an endless number of uses for something like this. Here are a few off the top of my head:

Run a news website?

Run a few different title options for an article on your front page.
Figure out how to get users to read more of a paginated article:
Test the number of words shown on each page (say 500 or 1000)
Each time the user clicks “next page”, update the default test result value to the user’s progress through the article. (See the /select_arm route in the docs for more on default values)

Run a website with promoted content?

Allow advertisers to choose multiple promotions to run at the same time, and optimize which one gets shown based on clicks and/or user feedback.
Test the amount of labeling around the fact that this content is promoted vs organically created by users.

Test the audience demographic response.

Set up a standard A/B test.
Display the same content to every user.
Report back the results of the test where the “arm” represents a specific user demographic, rather than an alteration to the web page.
Your test results will show which demographic responds best to your content.

There are a lot of features that I think are pretty clever, like setting a default test result value and a TTL for the test, so that the default value gets reported when the test “expires,” and the ability to update this default value and TTL by sending a “heartbeat.” And there’s a lot on my To Do list, such as confidence intervals, multivariate testing, bucketing/aggregating test results for high-traffic websites, and more.

Right now the project is in a limited beta mode. I’m running it on a few pages that don’t get much traffic and I would love some help working out whatever bugs may come up. If you’re interested, please get in touch!

The Lakers are 6-22 when Kobe has more than 19 field goal...

Thu, 21 Feb 2013 08:42:00 -0500

The Lakers are 6-22 when Kobe has more than 19 field goal attempts and 12-3 in the rest of the games.

(via Data supports claim that if Kobe stops ball hogging the Lakers will win more | Simply Statistics)

I was going to spend today working on a new project, but...

Mon, 18 Feb 2013 14:00:00 -0500

I was going to spend today working on a new project, but realized that the amount of work required just to get the boilerplate register/log-in/log-out code written is way too repetitive. So I stripped some code out of my multi-armed bandit API and called it bootstrap-bootstrap. It’s super basic, built with mongo/tornado/bootstrap, easy to deploy to heroku, and now up on github.

radioon: WNYC did a “How Single & How Happy Are You”...

Thu, 14 Feb 2013 15:52:37 -0500

radioon:

WNYC did a “How Single & How Happy Are You” survey, which I annotated for them.

(h/t DataNews for the link)

Huge turnout for the first NYC Data Science meetup! Probably...

Mon, 11 Feb 2013 21:50:21 -0500

Huge turnout for the first NYC Data Science meetup! Probably about 300 people on a gross January night.

"Are you going to be optimizing the solution to one problem, or solving a wide variety of problems?..."

Mon, 04 Feb 2013 15:17:00 -0500

“Are you going to be optimizing the solution to one problem, or solving a wide variety of problems? If it’s just one problem, is it something that you can imagine being happy about working on a year from now?”

- Hilary Mason is on point, as usual. Read Finding a Great Data Science Job for more excellent considerations.

An API for website testing/optimization

Sat, 02 Feb 2013 14:40:00 -0500

This is something that I’ve been working on very heavily for the last 6 weeks or so, and I’m excited to start showing it off. It’s currently being hosted at http://banditapi.herokuapp.com.

In a normal A/B test on a website, you randomly present users with one version of a feature or another. This could be the placement of your logo, for example. You let the test run for some amount of time, then measure the relative performance of the two versions (“arms”) and go with the better one from that point on.

Multi-Armed Bandit tests are different. They are designed to learn which arm is preferred, and move towards that version of the website as quickly as possible. There are many nuances to these types of tests, and I highly recommend John Myles White’s book on the subject (there’s also a video of him talking about this subject at tumblr recently).

The API that I built provides a couple of different Multi-Armed Bandit algorithms, which are exposed as simple routes to request an arm to play, and to update the algorithm with the test results. There’s also a little admin view to see how your test is performing:

I’m at the point now where I need some help testing. This thing is not at the point where you should deploy it on a production site, but if you have something to test out that doesn’t take a lot of traffic (and you don’t mind a little latency), it should be reliable. There’s a also a little client library on github if you just want to fuss with it. Either email me at myfirstname.mylastname@gmail.com, or send me a message through tumblr to get an account.

BigData TechCon 2013

Wed, 23 Jan 2013 15:48:08 -0500

BigData TechCon 2013:

I’m honored to be giving a keynote talk at the 2013 BigData TechCon conference in Boston (April 8-10). There are a lot of excellent speakers at the event that I’m looking forward to seeing, including Claudia Perlich (Media6Degrees), Jonathan Seidman (Cloudera, formerly Orbitz), and Oscar Boykin (twitter).

Registration is open here. Here’s a short abstract for my talk:

Social networks are able to collect large amounts of activity data from their user and customer base. As Big Data professionals, we conduct experiments on custom data sets to measure the effectiveness of our products or advertising methodologies. Since a social network is effectively useless without an active community, our companies owe it to their users to create new and better products based on this information. Learn how our data analysis and predictive analytics must take a different approach than Big Data in fields like finance, medicine, and defense.

I’m extra excited that the conference is in my home town. It’s been a while since I’ve spent a few days in Boston and I look forward to seeing all of my friends up there. I can already taste the Silhouette popcorn.

"Significance testing in general has been a greatly overworked procedure, and in many cases where..."

Tue, 22 Jan 2013 21:22:33 -0500

“Significance testing in general has been a greatly overworked procedure, and in many cases where significance statements have been made it would have better to provide an interval within which the value of the parameter would be expected to lie.”

- George Box “Statistics For Experimenters,” 1978.