I’m always amazed by the amount of work that gets done at every DataKind event. In just 14 (consecutive) hours, we were able to understand a very complex data problem, scope the project, and start laying the groundwork for a number of tools to help Amnesty International save lives.
Thanks to the folks from AI and the 30 or so people who helped out with our project.
I gave a talk about Digital Signal Processing in Hadoop at this month’s NYC Machine Learning meetup. Here’s the abstract:
In this talk I’m going to introduce the concepts of digital signals, filters, and their interpretation in both the time and frequency domain, and work through a few simple examples of low-pass filter design and application. It’s much more application focused than theoretical, and there is no assumed prior knowledge of signal processing. I’ll show how they can be used either in a real-time stream or in batch-mode in Hadoop (with Scalding) and give a demo on how to detect trendy meme-ish blogs on Tumblr.
The guys at yhat released a port of ggplot2 for python!
(via yhat/ggplot)
This should be interesting: an online unconference, put on by the great people behind Simply Statistics. The listed speakers include Sinan Aral, Hilary Mason, Hadley Wickham, and others.
My [Sean’s] (five minute) talk from Ignite Foo Camp on how we can build a reputation system to identify the people who are best at making predictions.
This is a great post showing how to build and work with random variables in scala. It starts with something as simple as drawing from a uniform distribution and moves to distribution transforms, adding random variables, conditional probability, and more. The code examples are great if you’re a scala programmer who wants to learn more about probability, or if you already know the probability and want to learn some scala by example.
Here’s a quick taste of how to build a generic Distribution trait and create uniform and Bernoulli distributions out of it.
trait Distribution[A] {
def get: A
def sample(n: Int): List[A] = {
List.fill(n)(this.get)
}
def map[B](f: A => B): Distribution[B] = new Distribution[B] {
override def get = f(self.get)
}
}
val uniform = new Distribution[Double] {
private val rand = new java.util.Random()
override def get = rand.nextDouble()
}
def bernoulli(p: Double): Distribution[Boolean] = {
uniform.map(_ < p)
}
This is an excellent post about why raw numbers are not always a good representation of the overall data that you are analyzing. Turns out it’s really easy to smooth the data and find the real trends and outliers in your sample (which is often what you’re looking for as an analyst).
Thanks to Anna for the link
People put a lot of effort into predicting the sentiment around a certain article, tweet, photo, or any other piece of information on the internet. There’s huge value in knowing who is consuming your content, how they feel about it, and what kinds of things they feel similar about. For example, if I regularly share my love for Pepsi products and disdain for Coca-Cola products, then Pepsi can consider me a loyal customer and target their advertising dollars elsewhere.
Measuring sentiment is not always easy. The cononical approach is to build large list of “positive” and “negative” words (either manually or via labeled training data), and then count how many of each group appear in the text that you want to classify. You can add some weights and filters to the words, but it really all comes down to the same sort of thing. This works OK in some contexts, but will never be exact and gets much more difficult if you try to figure out the sentiment of specific users and not an aggregate “mood.”
A few months ago I spoke at a conference and focused my talk on building “data generating products.” By that, I mean building features that enhance a user’s experience, and simultaneously let you collect useful data that you would otherwise have to predict in order to build new products on top of the new information. The example that I used was tumblr’s typed post system.
With the seven post types, we can give users tools that make it easier to share specific types of media. Sharing a song? Search for it on Spotify or SoundCloud. Sharing a photo? Drag-and-drop the images, or take one with your webcam. It’s easy for us to determine the type of media that you share or consume. If your blog is focused on sharing songs that you enjoy, we can recommend it to people who are using tumblr to discover new music.
Today bitly released a bitly for feelings bookmarklet (above). It lets users bookmark and/or share articles and websites that they come across, and uses a cute short-url domains like oppos.es or wtfthis.me, based on how you feel about the article. I’m not sure what their long-term plans are for this product (it’s still in beta), but I’d love to be able to log into my bitly account and see all of the funny links that I’ve saved (lolthis.me), or all of the products I’d like to buy someday (iwantth.is).
It lets me, the user, organize content and makes me more likely to use bitly as a bookmarking service. It also tells them explicitly how I feel about a photo, article, or product and gives them an idea about why I am bookmarking and/or sharing it. There’s no need to scrape my twitter account and analyze the words I use to describe the link that I’m sharing.
There are other products that have been well designed to collect rich user data, such as LinkedIn Endorsements or when Facebook switched from having text lists of bands/movies/books that I like to subscribing to individual ‘like’ pages for each item. These are fine ideas that create clear signals, but I’m no more likely to use LinkedIn because I can verify that my friend knows how to program in Ruby. Bitly added a simple interface to enhance their simple service, and are getting a wealth of valuable data out of it.
Now that they’re collecting this great information, I can’t wait to see what they build with it. Hopefully they’ll work it into their real time search engine.
A great analysis of Zipcar’s twitter followers. Via Who are brands really talking to on social media?
The other day I saw this video of a 14 year old high school basketball sensation. He’s billed as “The Next LeBron” who was in turn billed as “The Next Michael Jordan.” What caught my eye is that they all wear number 23 on their jersey (until LeBron moved to The Heat, at least).
When I was a kid I played lots of sports, and the best basketball player on my team was always #23 and the best football player was always #34 (Bo Jackson, Walter Payton, etc). So that got me to wondering what the “desirable” numbers are in other sports. I asked about other sports on twitter and #10 is clearly the best number to wear in soccer, and maybe #9 for hockey (Richard, Hull, Howe).
I wanted to figure out which jersey numbers have been the best historically, and baseball is the obvious sport to turn to for this, since data is so reliable and readily available. So I downloaded the career stats for a little over 17,000 players (which I think is every baseball player ever) and decided to see which are the best jersey numbers over all.
For each player, I got their career batting average and their jersey number, which proved to be harder than I expected. For example, Johnny Damon wore #18 from 2002-2009 (during his best seasons with the Red Sox and Yankees), but has also worn 51, 8, 22, and 33. So to make it a little easy on myself, I just got the number that they wore on the most teams. Johnny Damon has had 7 numbers on 6 teams, but wore #18 on 4 teams, so I’m associating him with #18.
For scoring the jersey number, I’m taking the mean of career batting averages for all players who wore that number, weighted by the number of plate appearances per batter. I perform this weighting so that someone like Roberto Clemente who hit .317 over 10,211 plate appearances would have more influence than someone like Buster Posey, who hit .311 with only 1,324 plate appearances. That’s a bit of a mouthful so here’s some math that might make more sense:
\[ S_j = \sum_{i=1}^{N_j} \frac{p_{j,i} b_{j,i}}{\sum_{k=1}^{N_j}p_{j,k}} \]
Where \( S_j \) is the score for jersey number \( j \), \( b_{j,i} \) and \( p_{j,i} \) are respectively the batting average and number of plate appearances for the \( i^{th} \) player who wears jersey number \( j \), and \( N_j \) is the total number of players who wore jersey number \( j \). I was also sure to limit the data set to \( p_{j,i} \geq 500 \) and \( N_j \geq 5 \). There were also a lot of older (~100 years ago) players whose number I couldn’t gather, so I dropped those as well. This narrowed the data set down to about 3,500 players.
The next thing to consider is that since I’m using lifetime batting average and number of plate appearances to rank players, what I’m really doing is ranking hitters. Many pitchers in the National League end up with over 500 at bats, but their batting average is just going to hurt the overall ranking of their jersey number. The following graph makes this crystal clear. The y-axis shows the jersey number, and the x-axis is the batting average for each player.
So once we remove the pitchers, here’s where each number ranks in terms of weighted batting average:
There are some clear winners here. The number 51 is a bit of an outlier because there are only 10 batters with that number who meet the minumum plate appearance requirement, but the list includes Ichiro Suzuki (.321) and Bernie Williams (.297) among others with high batting averages. Here are the top players who wore #4, a more “classic” number:
name plate_appearances batting_average
Rogers Hornsby 9480 0.358
Lou Gehrig 9663 0.340
Riggs Stephenson 5134 0.336
Dale Alexander 2736 0.331
Babe Herman 6228 0.324
Luke Appling 10254 0.310
Hack Wilson 5556 0.307
Paul Molitor 12167 0.306
Smead Jolley 1815 0.305
Mel Ott 11348 0.304
Not a bad group to be in. There are also some clear loser numbers, but they’re mostly higher numbers which I think are often worn by pitchers.
Here are the the 10 best and worst lifetime batting averages:
name number plate_appearances batting_average
Rogers Hornsby 4 9480 0.358 0
Ted Williams 9 9788 0.344 0
Bill Terry 3 7108 0.341 0
Lou Gehrig 4 9663 0.340 0
Tony Gwynn 19 10232 0.338 0
Riggs Stephenson 4 5134 0.336 0
Al Simmons 7 9518 0.334 0
Paul Waner 24 10766 0.333 0
Dale Alexander 4 2736 0.331 0
Stan Musial 6 12717 0.331 0
...
Corky Miller 37 575 0.188 0
J.R. Phillips 17 545 0.188 0
Bill Plummer 8 1007 0.188 0
Gus Gil 18 538 0.186 0
Brandon Wood 32 751 0.186 0
Drew Butera 41 531 0.183 0
Kevin Cash 17 714 0.183 0
Tommy Dean 3 594 0.180 0
Ray Oyler 1 1445 0.175 0
John Vukovich 16 607 0.161 0
If you’re a baseball fan, you can see that the bottom is all populated by pitchers, because no .890 batter would ever get 950 at bats in the majors. In fact, baseball’s obvious bias towards keeping good hitters and cutting poor hitters is obvious when you plot lifetime batting average against total plate appearances:
So what number would I wear if I were a pro baseball player? I can’t NOT choose #9, even though it’s retired by the Red Sox.
Also, all of the code and results for this blog post are available on my github page.
