Skip to content

Mastodon, Leviathan, Moby Dick, etc.

Here’s what Google Trends says about the search volume for the four Mastodon albums. You can see that Crack the Skye came out in the end of March, people listened to it, realized it sounds like crappy nu-metal produced by the guy that did all of the Stone Temple Pilots records, and then remembered how great Leviathan is.

The peak for “leviathan” in July was because Clash of the Titans was released that day.

I’m also not sure why “Moby Dick” is becoming such a popular term. Maybe because of that Blackberry commercial?

Lastly, check out this amazing comic that somebody made (click for full size). Or just listen to Blood and Thunder.

2010 12 19 adam Data
music Comments (0) Permalink

How excel corrupts your data

I was just emailing back and forth with a professor about errors in some data that I had prepared for him, speciffically for unique ID’s for a bunch of different firms. These ID’s are supposed to be 8 alpha-numeric characters, and he was saying that the ID’s I provided him are not all 8 characters and therefore can’t be used to merge with more data.

This is is a clear example of why you should never ever open flat files in Excel. Here’s what he probably saw when loading the data:

And here’s what I saw using R:

           DATE                 COMNAM    CUSIP TICKER PERMNO
39844  20080102                  3M CO 88579Y10    MMM  22592
217087 20080102              A B B LTD 00037520    ABB  88953
185515 20080102 A B N AMRO HOLDING N V 00093710    ABN  84766
144754 20080102             A E S CORP 00130H10    AES  76712

Have you ever gotten the “Do you want to save your changes?” dialog box when exiting Excel, even though you didn’t modify the data at all? That’s because they took the liberty of doing so for you.

Yes, Excel will let you use quotations or some other character as a text qualifier, but you can’t assume that they will always be there, especially when the data is from an unknown source. So always be cautious of opening data in Excel.

Or better yet, don’t use Excel at all. This makes me wonder if there are any good lightweight spreadsheet-type programs out there that let you view some data, maybe make some text changes, can handle large datasets, handles a variety of text delimiters and quotation characters, but does nothing to the formatting, etc. Anybody know of one?

2010 11 16 adam Data Comments (2) Permalink

Can twitter spam response rates possibly be this high?

My father just forwarded me an email from his company warning about an increase in spam email for the holiday season. The chart they included is pretty crazy. It looks like about a 70% increase in spam in the month of October.

So if 98% of all email gets blocked by spam filters, and let’s say that 1% of the mail that actually makes it through is spam (I’m sure it’s much lower than this), that means 1 of every 5,000 emails sent to this company is a spam message that makes it through to a person’s inbox. That seems incredibly high to me, but maybe they don’t get as much spam emails as an @aol.com or @hotmail.com address would.

I’ve also been thinking about twitter spam lately. I got this message from some job hiring website:

So I checked out the statistics from the bit.ly link, and it looks like they actually do get a fair number of clicks (click to enlarge).

I thought about writing a little script to calculate the response rates to this particular ad as a function of time, but it turns out it would take more than 5 minutes of effort. However, if you assume that @coolnewjobs is the only person sending this link around, they have an amazingly high average response rate. The account has generated 34,150 tweets, and this link has 5,914 clicks. If every single tweet included the link (they don’t), that’s a 17% response rate. The average response rate is actually probably somewhere over 20%. Amazing. No wonder so Columbia Business School had a conference about using “new” web technologies last week.

I think it would be interesting to aggregate a handful of job posting twitter spam accounts and look at how the response rate correlates to unemployment, political polls, etc. Maybe I’ll get back to it some time.

2010 11 16 adam Social Networking
Statistics Comments (0) Permalink

Steinbeck.

After seeing this post, I came across a copy of East of Eden and picked it up. Everyone in the world should read Travels with Charley

via the impossible cool.

2010 09 06 adam Uncategorized Comments (0) Permalink

Martha’s Vineyard.

I grew up in a touristy New England beach town, so my family never really went to the Cape, Vineyard, etc.  I went for the first time this weekend with my friend Kate, hoping to meet up with our old friend Barry O. I didn’t find him, but I did get to enjoy some Barack-o-Guaco and a brutal 10-mile bike ride (rolling hills + fixed gear = sore legs).

2010 08 23 adam Uncategorized Comments (0) Permalink

Looking at WikiLeaks Data

The Guardian posted a bunch of data about IED explosions in Afghanistan from the WikiLeaks data that just came out. Drew Conway took a look at it and it looked pretty similar to the plots that I made the other day.

So here’s what I wanted to see: what percentage of the deaths/injuries reported were from friendly fire? It looks like we started off by killing ourselves a whole lot, and then the balance came back down a bit.

Of course, that could be either from less friendly fire or more enemy fire. It seems, unfortunately, to be the later. Here’s a breakdown of the friendly/enemy-caused deaths and injuries by date (each bar is 30 days) and unit type.

2010 07 30 adam Statistics Comments (0) Permalink

Hockey: Goals and Missed Shots

A coworker is doing some kind of sports research and needed some NHL data. I’m going to see what I can do with it as well.

Black = missed shots
Red = goals
Ice = imaginary

2010 07 28 adam Uncategorized Comments (0) Permalink

Keeping cool

The other day my cat Olive (seen here preparing a drink) was chasing a fly around for about 1/2 hour, and when he finally caught it, he was panting like crazy.  As I sit here now sweating in my living room I was wondering, naturally, if monkeys and other primates sweat or pant to cool themselves off.  Being so closely related, I figure they must sweat, but that must be less effective since they’re covered in hair.

Anyway, I looked it up, and found this cool Infrared Zoo page at Cal Tech that says:

To stay cool, warm-blooded animals sweat or pant to loose heat by water evaporation. They can also cool off by moving into a shaded area or by getting wet. Only mammals can sweat. Primates, such as humans, apes and monkey, have sweat glands all over their bodies. Dogs and cats have sweat glands only on their feet. Whales are mammals who have no sweat glands, but then since they live in the water, they don’t really need them. Large mammals can have difficulty cooling down if they get overheated. This is why elephants, for example, have large, thin ears which loose heat quickly.

So I guess they do sweat. Now you know.

More interesting, they have tons of IR pictures of warm and cold blooded animals.

2010 07 19 adam Uncategorized Comments (0) Permalink

Warren Buffet’s philanthropic pledge

This man is truly incredible. I saw him talk recently and he was sharp, charming and modest.

Some material things make my life more enjoyable; many, however, would not. I like having an
expensive private plane, but owning a half-dozen homes would be a burden. Too often, a vast
collection of possessions ends up possessing its owner. The asset I most value, aside from health,
is interesting, diverse, and long-standing friends.

My wealth has come from a combination of living in America, some lucky genes, and compound
interest. Both my children and I won what I call the ovarian lottery. (For starters, the odds
against my 1930 birth taking place in the U.S. were at least 30 to 1. My being male and white
also removed huge obstacles that a majority of Americans then faced.)

My luck was accentuated by my living in a market system that sometimes produces distorted
results, though overall it serves our country well. I’ve worked in an economy that rewards
someone who saves the lives of others on a battlefield with a medal, rewards a great teacher with
thank-you notes from parents, but rewards those who can detect the mispricing of securities with
sums reaching into the billions. In short, fate’s distribution of long straws is wildly capricious.

The reaction of my family and me to our extraordinary good fortune is not guilt, but rather
gratitude. Were we to use more than 1% of my claim checks on ourselves, neither our happiness
nor our well-being would be enhanced. In contrast, that remaining 99% can have a huge effect on
the health and welfare of others. That reality sets an obvious course for me and my family: Keep
all we can conceivably need and distribute the rest to society, for its needs. My pledge starts us
down that course.

via http://givingpledge.org/

2010 07 14 adam Uncategorized Comments (0) Permalink

Happy 12th of July

I wish I had this bad boy last weekend.

2010 07 12 adam Uncategorized Comments (0) Permalink