I used this wonderful flowingdata guide to mapping connections in R and decided to look at the carrier delay time for each route. The plots below show the average delay from 0 (blue) to 100 (red) minutes for each carrier. For each of the plots, I averaged the delay (or percentage of delays) for each direction of each route.

First, the pretty graphs which I take no credit for designing. I made three graphs for each carrier:

1) Average delay time (these will be low, since most flights have no delay)
2) Percentage of delays of at least 5 minutes
3) Average delay time for flights with at least a 5 minute delay (these will be much larger)

So, here are three airlines. You can click on any of the pictures for slightly larger versions or view higher quality pdfs here (with other airlines as well, check out the Pinnacle ones).

American Airlines:

Delta:

JetBlue:

There were a lot of generalizations that I made for these. First off, I only looked at the carrier delay, which I assume to mean that it was the airlines fault. My next step is to look at how each airline handles weather delays, to see which airlines or airports recover best from a big storm, but the scope of this project is pure incompetence. I also didn’t consider how much of the time is made up for in the air, since a 30-minute delay on a cross-country flight could easily be accounted for by flying a little faster.

For example, there were 2041 American Airlines flights between MIA and LGA. 1166 were MIA->LGA (average delay: 20.3 minutes), and 875 were LGA->LGA (average delay: 15.3 minutes). So the average delay graph will show 18.15 minutes.

Next I looked at the percentage of flights that were delayed by at least five minutes, which I don’t think requires any explaining.

Lastly, I took a subset of the flights that were delayed by at least 5 minutes (who is REALLY bothered by a 5 minute delay?) and calculated the average of those flights. I really like the graph below that shows the percentage of flights that are still delayed vs. time from the intended take-off. In that plot, you see that Southwest has the most short (< 5 minutes) delays, but the fewest long (> 30 minutes) delays (click for larger).

All of the data that I used is available here in either R or .csv format. I’ll make the code less embarrassing and then get it up onto github. I mostly just hacked on the code from the flowingdata post above.

*yet

Most of what I read on the times are three blogs: Economix, Bits, Bitten. I also read the top stories on my iPhone and browse around on the article skimmer sometimes.

This graph is from google reader and shows the number of articles posted by a given blog (blue) and how many I’ve read (orange) per day for the last month:

After the paywall comes into effect, it will look something like this if I don’t want to subscribe:

If all of the regular readers start their 20 article period on the first of every month, it won’t be long until the blogs post articles with a pattern like the one below (though they obviously won’t go all the way to zero).

The actual paper would obviously still have stories every day, but the blogs will (or should, at least) post as many articles as possible in the first few days of the month while they’re still open to the public. In the full paper, the best journalists will try to get their articles posted on the first of the month, which would likely mean the quality of articles would follow a similar pattern.

But here’s what should save this from happening (where the “sort of” comes in). FAQ #12 says:

Can I still access NYTimes.com articles through Facebook, Twitter, Google or my blog?

Yes. We encourage links from Facebook, Twitter, search engines, blogs and social media. When you visit NYTimes.com through a link from one of these channels, that article (or video, slide show, etc.) will count toward your monthly limit of 20 free articles, but you will still be able to view it even if you’ve already read your 20 free articles.

When you visit NYTimes.com by clicking links in Google search results, you’ll enjoy up to five free articles per day.

That could actually increase readership from people like me. Once I pass my 20 free articles, which will probably happen very soon, I will still be able to read 5 articles per day (I’ll just have to google the titles or something), giving a possible total of 160 free articles in 30 days. Under this system, I’ll have the incentive to check the site every day instead of coming every so often and reading 40+ articles like I did on 3/10. The graph could potentially take this form:

Making “check the Times” part of my daily routine will do one of two things:

1) I’ll get tired of checking all the time and I’ll just go back to my current reading habit, but capped at 5 articles per day.
2) I’ll become more committed and perhaps actually start paying for a subscription.
3) I’ll just get my news elsewhere.

My guess is that it will be #1.

The first thing I put up is a simple webcrawler for downloading the outcome of all of the basketball games since 1970. I’ll play with the data some more, but for now here are the best and worst teams since 1970, and the average score difference.

By win/loss record:

team year wins losses   avg.win
 CHI 1996   72     10 11.940000
 CHI 1997   69     13  9.811881
 LAL 1972   69     13 10.876289
 BOS 1973   68     14  7.294737
 BOS 1986   67     15  9.580000
 CHI 1992   67     15  9.538462

team year wins losses    avg.win
 DAL 1994   13     69  -8.695122
 LAC 1987   12     70 -11.426829
 NJN 2010   12     70  -9.121951
 DAL 1993   11     71 -15.195122
 DEN 1998   11     71 -11.780488
 PHI 1973    9     73 -12.085366

And by the average score difference:

team year wins losses   avg.win
 MIL 1971   66     16 12.583333
 CHI 1996   72     10 11.940000
 LAL 1972   69     13 10.876289
 MIL 1972   63     19 10.602151
 CHI 1997   69     13  9.811881
 LAL 1987   65     17  9.680000

team year wins losses   avg.win
 LAC 1987   12     70 -11.42683
 LAC 2000   15     67 -11.52439
 HOU 1983   14     68 -11.59756
 DEN 1998   11     71 -11.78049
 PHI 1973    9     73 -12.08537
 DAL 1993   11     71 -15.19512

The next thing to do is figure out who these teams are who are scoring more points than they should be, but for now I’m off to dinner.

The image below shows which programs are used the most by CBS professors. There are some pretty strong biases to the plot:

1. This is only data from our computing cluster, so desktop use or use on other servers is not included. My work, for example, is not well represented on here (I’m the circle in the second graph).
2. All of our datasets on this server are stored in SAS format, so if you want to access them you’re going to have to run SAS at least once. I have a handy “get out of sas” script.
3. Some programs are more interactive and only need to be opened once to do all of your work. That’s probably why the number of program launches for Stata and Matlab are lower than SAS even though there are more unique users.
4. I had to exclude python, java, bash, etc. scripts because they are run so many times for non-computing purposes. I also excluded custom compiled programs (mainly in C++) because they have unique names and are only launched a few times each.

One thing I find very interesting and would love to have someone explain is the linear relationships between memory usage and CPU.  I talked to the “star” user in the plot below about this and looked at the dates, and he said that the lines correspond to two different projects. Anyone have any references on memory/processor tradeoffs and whether or not there’s an advantage to a steep or narrow slope in a plot like this?

And here it is with names. I don’t have time to fix the overlap right now.

h/t Flowing Data for the data.

[EDIT: I posted this in February, before Charlie Sheen became Chief Douchebag of the media, though probably well after he became an actual douchebag. I would certainly have a different title for this post today]

Let me describe the reasons I hate it:

1. It appears that all of the US presidents were in office the same amount of time. George HW Bush should be half the size and Obama should be 1/4 the size.
2. All of the dates are evenly spaced, but one of them is only a 4 year difference.
3. All of the presidential terms are off. It makes it look like George W. Bush’s term ran from 1993 to 2001, when in fact it ran from 2001 to 2008.
4. The timeline ends in (presumably the beginning of) 2009. Obama shouldn’t even be on there.
5. If the point of the graphic is to show that Hosni Mubarak was in power through 5 US presidential administrations, and that Japan has had many leaders, there should only be one picture of Mubarak.
6. I don’t know if a 30 year term is typical for Egypt or not. How long was the previous leader in office? (11 years, with his term ending in assassination)? How about the guy before that? (14 years). They should have put up at least one other guy on there.

Based on all of these disasters, I know absolutely nothing about what
was going on in Japan. Have they really had 5 leaders in the 2 (or is
it 8) years that Obama has been president?

</rant>

While Matlab does a good job of using multiple processor cores on some of its built-in functions, I wanted to get add that to some of the custom mex functions I’ve written. I’ve achieved some great speed increases when re-writing matlab functions in c. Especially with iterative loops (x(i) = x(i-1)+x(i+1)), or times where I can make enormous assumptions about the input to a function and avoid all kinds of time-wasting error detections.

Anyway, I spent a little time trying to figure out how to use threading within a matlab mex function and was able to get it working. The actual code is below, and I assume some familiarity with mex functions and POSIX threading (read the links above). Here’s a brief overview of how it works:

The code below simply fills a matrix with this very exciting pattern:

Within the mexFunction(), I figure out how many columns of this matrix will be filled by each thread (always try to process by column. See the bottom of this post for more). In my case, I used 7 threads and had a 1000×1000 matrix, so each of the first 6 threads process 142 columns, and the 7th thread processes the remaining 148. I realize it’s better to have 6 threads process 143 and the 7th process 142, but I didn’t really want to get into figuring that out right now.

You can compile and run the code with two simple lines. Make sure that when you compile it, you link against the pthread library. In this example I have each thread print out it’s ID and the number of columns it will be working on.

>> mex -lpthread fill_matrix_par.c
>> x=fill_matrix_par(1000);
In thread 0. Filling 142 columns
In thread 6. Filling 148 columns
In thread 4. Filling 142 columns
In thread 3. Filling 142 columns
In thread 2. Filling 142 columns
In thread 1. Filling 142 columns

#include "mex.h"
#include <math.h>
#include "pthread.h"
 
/**
 * prhs[0] : Nrow - dimension of the square matrix to be output
 *
 * plhs[0] : CS   - an Nrow by Nrow matrix output
 */
 
 
/* We need to define a structure to pass to each thread, since you can only
 * pass it one variable.  This will hold everything needed by the thread.
 */
typedef struct {
   int Ncol;           /* number of columns each specific thread will process */
   int Nrow;           /* number of rows in the matrix */
   int offset;         /* this is for use in filling the matrix */
   int i;              /* this is just and ID for the thread */
   double *col_start;  /* This is a pointer to a specific section of the matrix */ 
} info_T;
 
 
/* Define how many threads you want to use. 
 * This shouldn't exceed the number of cores on your computer. I'm using
 * an 8-core server, so I'll use 7 of them for now
 */
#define MAX_NUMBER_THREADS 7
 
/* Here's the definition of the function that will be called 
 * in each thread
 */
void *calc_col( void *ptr );
 
 
 
void mexFunction(int nlhs,
            mxArray* plhs[],
            int nrhs,
            const mxArray* prhs[])
{
 
    /* Grab the input from matlab */
    int Nrow = *mxGetPr(prhs[0]);
 
    /* Create the output DoubleMatrix */
    mxArray* CSarr  = mxCreateDoubleMatrix(Nrow, Nrow, mxREAL);
    double* CS = mxGetPr(CSarr);
 
    int i;
 
    /* An array of threads and structures to be passed to each thread */
    pthread_t pThreads[MAX_NUMBER_THREADS];
    info_T pass_struct[MAX_NUMBER_THREADS];
 
    /* Each thread will handle this many columns, except the last thread
     * will will handle the remainder as well.  I could speed this up by
     * evenly distributing the remaineder threads, but I don't.
     */
    int cols_per_thread = Nrow / MAX_NUMBER_THREADS;
 
    for (i = 0; i < MAX_NUMBER_THREADS; i++){
        /* fill the structure with info to be passed to the given thread */
        pass_struct[i].Ncol      = cols_per_thread;
        pass_struct[i].Nrow      = Nrow;
        pass_struct[i].col_start = CS + Nrow*i*cols_per_thread;
        pass_struct[i].i         = i;
        pass_struct[i].offset    = i*cols_per_thread;
 
        /* The last thread needs to get the remainder of the columns.
         */
        if(i == MAX_NUMBER_THREADS - 1){
            pass_struct[i].Ncol = Nrow - i*cols_per_thread;
        }
 
        /* spawn the thread */
        pthread_create(&pThreads[i], NULL, 
                calc_col, (void*) &pass_struct[i]);
    }
 
    /* wait for all of the threads to finish */
    for (i = 0; i < MAX_NUMBER_THREADS; i++){
        pthread_join( pThreads[i], NULL);
    }
 
    /* return CSarr to matlab */
    nlhs = 1;
    plhs[0] = CSarr;
 
}
 
 
 
void *calc_col( void *ptr ){
    /* Here's the function that calculates the column starting at the 
     * given point 
     */
 
    /* create a local structure to point to the input pointer */
    info_T *struct_ptr;
    struct_ptr = (info_T *) ptr;
 
    mexPrintf("In thread %i. Filling %i columns\n", 
            struct_ptr->i, struct_ptr->Ncol);
 
    /* perform whatever you need to do on the matrix here */
    int rows, cols;
    for(cols=0; cols<struct_ptr->Ncol; cols++){
        for(rows=0; rows<struct_ptr->Nrow; rows++){
            *(struct_ptr->col_start + rows + cols*struct_ptr->Nrow) = 
                    -rows+cols+struct_ptr->offset;
        }
    }
 
    /* exit the thread */
    pthread_exit(NULL);
}

So here’s the catch: Each of the threads are writing to the same matrix at the same time, each accessing a different section of the memory. struct_ptr->col_start points to column 0, 142, 244, … depending on the thread number.

This is PROBABLY a bad idea and I don’t think it’s normally considered to be thread-safe, but it works here. I most often write specific functions for specific tasks that fit inside specific Matlab scripts. I’d love for someone to correct me on this, but for now I just figured it out on my own.

With regards to processing by column instead of row, it’s always better to work down columns because of how the arrays are stored in memory. The image below sums it up pretty well:

So if you want to fill a matrix by row in c/c++, you would need to use two loops and do lots of multiplying:

for(r=0; r < Nrows; r++){
  for(c=0;c < Ncols; c++){
    *(array_loc1 + c*Nrows + r) = c+r*Nrows;
  }
}

and get this for an output:

     0     1     2     3     4
     5     6     7     8     9
    10    11    12    13    14
    15    16    17    18    19
    20    21    22    23    24

Whereas to fill it by column you would just loop over the total number of cells:

for(c=0; c < Nrows * Ncols; c++){
  *(array_loc2 + c) = c;
}

and you’ll get this:

     0     5    10    15    20
     1     6    11    16    21
     2     7    12    17    22
     3     8    13    18    23
     4     9    14    19    24