Data Diversions

Over the past few years, I’ve had a lot of fun writing on dfkoz.tumblr.com. Beginning in 2012, however, I started a new adventure with FirstMark, which left less and less time to write here.

While I intend to occasionally share a data-driven post here from time to time, I’ll be focusing a lot more of my writing time on sharing lessons from my work at FirstMark.

This post is a response to a question posed by Chris Whong on Twitter: "I am curious to know what the $ implications of adding a 15% button would be." Chris is the source of the raw data for my previous post, Analysis of NYC Taxi Tip Data: 44% of Passengers Hit the 20% Button.

By my math, a 15% tip button reduces total taxi tips by between $35MM and $60MM. This works out to a range of $2,600 to $4,600 for each of the 13,437 medallions active in 2014 and $700 to $1,200 for each of the roughly 50,000 drivers in the city.

I calculated the impact four different ways. On the low end, I assumed that only those who default to the cheapest button (currently the 20% button) migrate to the 15% button. This reduces total tips by $26MM, but seems like unlikely behavior in practice.

The second approach reduces all of the exact 20%, 25%, or 30% tips by 5%, with the assumption that all "non-standard" tipping behavior remains consistent. This leads to a $35MM reduction in tips.

The third calculation lowers all credit card tips by 5% across the board, for an aggregate impact of $60MM.

Thanks to Ben Wellington, the man behind I Quant NY, I discovered that Chris Whong posted a year's worth of NYC taxi fare data, which he obtained from the New York City Taxi and Limousine Commision by FOIA request. What follows is an analysis of 578,233 rides, or about 1/300th of the roughly 175 million yellow taxi rides taken in 2013. 

Tipping Behavior

45% of taxi rides were paid for with cash. Of these 264,154 transactions, only 29 (0.00931%) included a tip. As a result, the following graphs depict only credit card transactions, totaling 311,623 rides.

The average tip was $2.53 with a standard deviation of $2.43. Excluding riders who did not tip at all, the average tip was $2.61 with a standard deviation of $2.43. As a fraction of the total fare, the average tip was 18.9% for all rides, and 19.5% for all rides that included a tip. 3.16% of riders did not tip at all. 1.75% of riders tip more than 30% and 0.21% tip more than 50%.

Unsurprisingly, there are some extreme data points in the raw data. In this sample of 1/300th of the records, there were about 20 negative fares and tips that (supposedly) went as high as 4067%.

As NYC residents know, the in-cab credit card interface offers three default tip levels: 20%, 25%, and 30%, once described as the "The $144,146,165 Button." 44% of riders defaulted to the 20% button, 10% to the 25% button, and 4% to the 30% button.

Tipping Behavior by Time of Day

Riders apparently feel most generous shortly before lunch and through the end of the work day. That, or - more likely - employers are picking up tab, resulting in higher average tips during the day.

Here is how rides are concentrated across an average day, an average weekday (defined as 6 PM Sunday to 6 PM Friday), and an average weekend (6 PM Friday to 6 PM Sunday). One hour that stands out is 4-5 PM, which - as any resident will tell you - is when New York cab drivers change shifts, leading to fewer available cabs.

Tipping Behavior by Day of Week

There is also a relationship between day of week and tipping behavior: Monday (the slowest taxi day of the week) is the best day for tips, and Friday (the busiest day) is the worst.

Again, my theory is that this has more to do with the type of trip (professional vs. personal) than anything else. Both Sunday and Monday are lower-volume days, but the tipping behavior is different.

Tipping Behavior by Month

There also appears to be a relationship between time of year and tipping behavior. December is the most generous month for tips:

Distribution of Total Fares

The average taxi fare in 2013 was $14.80, with a standard deviation of $12.13. About 30% of all taxi fares are less than $10, and nearly 80% are less than $20. 3.79% of fares were greater than $50 and 0.15% were greater than $100. The most expensive trip recorded in my sample was a whopping $544, paid in cash. Here's what the distribution of total fares, cash included, looks like:

Zoomed in a bit:

And the distribution of total fares:

That's a wrap for me. But if you're hungry for more taxi data visualization, you can check out Chris Whong's NYC Taxis: A Day in the Life or I Quant NY's analysis of Green Taxi tipping behavior by neighborhood (which is evidently the kickoff to a series of posts.)

There are currently 319,151,000 people in the United States. As of October 31, 2014, 150,000,000 of them were employed in full-time positions. At this wonderful time of year, most or all of them are presumably wishing colleagues, friends, and family a “Happy Thanksgiving”, whether they’re doing it from their PCs or phones. What would happen, however, if we collectively agreed to wish each other a “Happy Turkey Day”?

If the average adult types at 40 WPM and average word length is 5 characters, then an adult produces 200 characters in a minute, or one character every 0.3 seconds. Assuming that an average employee produces $67.32 of GDP per hour worked, one extra character -- repeated across, say, 25 holiday greetings -- amounts to $0.14025 of GDP. With 150,000,000 workers, that one-character difference costs a whopping total of $21,037,500 of GDP.

So if we all wished each other a “Happy Turkey Day,” we would reclaim two characters and $42,075,000 of GDP. “Happy T Day,” which eliminates seven characters, represents a savings of $147,262,500. Taken to the extreme, we would all simply write “HT,” saving 15 characters and $315,562,500 of GDP.

Founded in 1990, IMDB is a treasure trove of data with "...2,882,569 titles (includes episodes) and 5,909,020 personalities in its database." Using a simple scraper written in Ruby, I was able to grab data on the >90,000 TV shows in the IMDB database. This is what I found.

  There has been significant growth in the number of new television series created each year:

  Here is the same chart from 2000 forward:

It is important to note that our collective record-keeping has improved since IMDB's launch in 1990, and with the rise of internet usage generally (and IMDB specifically) over the past decade. IMDB's Google Trends chart: 

War and history series are the highest-rated genres on average. Among common genres, documentaries, animated series, and dramas are the highest-rated. Game shows and reality TV are rated significantly lower than other genres:

Certain genres have grown or faded significantly since 1950:

For example, westerns have almost completely disappeared:

The number of reality TV shows has had an astonishing rise starting in the late 90s, but has thankfully flattened out:

The prevalence of fantasy shows has been relatively steady over time, but has made a comeback in the past few years:

The overall quality of TV peaked in the 1960s, declined through 2004, and has since steadily improved (although I would take this chart with several grains of salt):

Here is the same chart from 1990 through today:

The top ten highest-rated television shows of all time are as follows:

1. Breaking Bad  (2008): 9.6 2. Game of Thrones  (2011): 9.5 3. Planet Earth  (2006): 9.5 4. Cosmos: A SpaceTime Odyssey  (2014): 9.5 5. The World at War  (1973): 9.5 6. The Wire  (2002): 9.4 7. True Detective  (2014): 9.4 8. Sherlock  (2010): 9.3 9. The Sopranos  (1999): 9.3 10. Cosmos  (1980): 9.3

The top TV series that have debuted since 2010 (note that I did *not* filter out shows with a small number of ratings):

1. Game of Thrones  (2011): 9.5 2. Cosmos: A SpaceTime Odyssey  (2014): 9.5 3. True Detective  (2014): 9.4 4. Sherlock  (2010): 9.3 5. Rick and Morty  (2013): 9.3 6. Africa  (2013): 9.3 7. Kardes Payi  (2014): 9.3 8. Leyla and Mecnun  (2011): 9.2 9. Fargo  (2014): 9.2 10. 24: Live Another Day  (2014): 9.2

Best of the 2000s:

1. Breaking Bad  (2008): 9.6 2. Planet Earth  (2006): 9.5 3. The Wire  (2002): 9.4 4. The Life of Mammals  (2002): 9.3 5. Al-Taghriba Al-Filistinia  (2004): 9.2 6. Arrested Development  (2003): 9.2 7. Firefly  (2002): 9.2 8. Life in the Undergrowth  (2005): 9.2 9. Life in Cold Blood  (2008): 9.2 10. Avatar: The Last Airbender  (2005): 9.1

Best of the 90s:

1. The Sopranos  (1999): 9.3 2. The Private Life of Plants  (1995): 9.2 3. The Trials of Life  (1990): 9.2 4. The Life of Birds  (1998): 9.1 5. Friends  (1994): 9.0 6. Twin Peaks  (1990): 9.0 7. Freaks and Geeks  (1999): 9.0 8. Batman: The Animated Series  (1992): 9.0 9. Cowboy Bebop  (1998): 9.0 10. Ice TV  (1996): 9.0

Best of the 80s:

1. Cosmos  (1980): 9.3 2. The Living Planet  (1984): 9.2 3. The Simpsons  (1989): 9.0 4. Seinfeld  (1989): 9.0 5. Dragon Ball Z  (1989): 8.9 6. Only Fools and Horses....  (1981): 8.9 7. Blackadder Goes Forth  (1989): 8.9 8. Fauji  (1988): 8.9 9. Solunski patrdii  (1985): 8.9 10. The Morning Program  (1987): 8.9

Best of the 70s:

1. The World at War  (1973): 9.5 2. Life on Earth  (1979): 9.2 3. I, Claudius  (1976): 9.1 4. La zulianita  (1977): 9.0 5. Doña Bárbara (1975): 9.0 6. Fawlty Towers  (1975): 8.9 7. Matador  (1978): 8.9 8. El chavo del ocho  (1972): 8.8 9. The Man and the City  (1971): 8.8 10. The Julie Andrews Hour  (1972): 8.8

Best of the 60s:

1. Monty Python's Flying Circus  (1969): 9.0 2. Fujimaru of the Wind  (1964): 9.0 3. The Prisoner  (1967): 8.8 4. Nu, pogodi!  (1969): 8.8 5. Ferien in Lipizza  (1966): 8.8 6. The Great Adventure  (1963): 8.8 7. Alle Hunde lieben Theobald  (1969): 8.8 8. Hafenpolizei  (1963): 8.8 9. Landarzt Dr. Brock  (1967): 8.8 10. Die Gäste des Felix Hechinger (1964): 8.8

Best of the 50s:

1. Twilight Zone  (1959): 8.9 2. Walt Disney Presents: Annette  (1958): 8.8 3. Corky and White Shadow  (1956): 8.8 4. Mayor of the Town  (1954): 8.8 5. The Spade Cooley Show  (1958): 8.8 6. Alfred Hitchcock Presents  (1955): 8.7 7. Mr. Squiggle and Friends  (1959): 8.7 8. Studio 57  (1954): 8.7 9. J.P. Patches  (1958): 8.7 10. The Honeymooners  (1955): 8.6

Best of the 40s:

Remember the Flesch Kincaid Grade Level? If you ever ran Microsoft Word's spell check in in the 90's, you probably do:

The Flesch and Flesch-Kincaid tests are "readability tests designed to indicate comprehension difficulty when reading a passage of contemporary academic English." The scores incorporate two primary factors: the average number of words per sentence, and the average number of syllables per word. A higher Flesch-Kincaid Grade Level is supposed to reflect more difficult text, with longer sentences and more multi-syllabic words.

The Ruby gem Lingua makes it quite easy to perform readability analysis against text. And with more than 950 scripts available on the Internet Movie Script Database, I had the raw ingredients necessary to run a fun and entirely inane analysis of the Flesch-Kincaid scores of movies. Here are the results.

Movies with the lowest Flesch-Kincaid grade level (simpler):

Glengarry Glen Gross: 0.11

Win Win: 0.13

Last Tango in Paris: 0.25

Silver Linings Playbook: 0.45 [Oscar winner, Best Adapted Screenplay]

Gran Torino: 0.54

Next Friday: 0.60

Hannibal: 0.64

Sling Blade: 0.65

Frozen: 0.66

Boogie Nights: 0.68 [Oscar nominee, Best Original Screenplay]

Movies with the highest Flesch-Kincaid grade level (more complex):

Fletch: 5.94

The Adventures of Buckaroo Banzai Across the Eighth Dimension: 5.48

The Ninth Gate: 5.07

L'Avventura (The Adventure):  4.74

Mulan: 4.31

Omega Man: 4.18

The Last Samurai: 4.17

Gattaca: 3.99

The Saint: 3.76

American Psycho: 3.76

If you're interested in seeing the full list, click here. This list includes two additional scores: the Flesch Reading Ease score, which uses slightly different weights (with a lower score indicating more readable text), and the Gunning fog index, which incorporates average sentence length and the density of words with three or more syllables.

A few caveats: first, this list is far from complete. In fact, there were ~100 scripts on the IMSDb that have been taken down (e.g., most of the Batman and all of the Harry Potter movies).

Second, the analysis is unfairly favorable to scripts with detailed stage directions, which tend to be more complex than dialogue. Here is the first passage from the "most sophisticated" movie, Fletch:

Seagulls squawk, and the waves pound, but we’re not talking about Malibu Colony, here. This is a fairly rundown beach area, catering to lower-echelon surfers, vagrants, and strung out druggies of all ages, several of whom stand or sit on their haunches by a dilapidated old hamburger stand. Over the stand is a faded sign: "FAT SAM’S HAMBURGERS".

In any case, enjoy. In the immortal words of John McClane:

I discovered the Stack Overflow Data Explorer this morning, which allows you to query the entire catalogue of Stack Overflow posts. After playing with it, I decided to figure out which Wikipedia articles have been linked to most often by Stack users over the past two years.

Over a two-year period, there were 10,831,467 total Stack Overflow posts. 93,290 (0.86%) posts contained the text 'en.wikipedia.org', representing 115,591 links. These links pointed to about 25,000 unique articles. 14,600 articles (58.25%) appeared just once. 91.76% of the articles appeared fewer than ten times, accounting for 38.31% of all links.

The most common link, SQL Injection, was cited over 1,000 times. Another 42* links appeared more than 200 times, and 113 total links appeared more than 100 times. The top 10 most linked-to articles are below. You can see the full list here. 

en.wikipedia.org/wiki/SQL_Injection (1001 links)

en.wikipedia.org/wiki/Same_origin_policy (909)

en.wikipedia.org/wiki/JSONP (501)

en.wikipedia.org/wiki/Cross-origin_resource_sharing (492)

en.wikipedia.org/wiki/Singleton_pattern (468)

en.wikipedia.org/wiki/Undefined_behavior (420)

en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller (406)

en.wikipedia.org/wiki/Single_responsibility_principle (402)

en.wikipedia.org/wiki/Don%27t_repeat_yourself (398)

en.wikipedia.org/wiki/Database_normalization (358)

Over at FirstMark, we're running an internship program called FirstMark Elite that connects the best undergraduate and graduate computer science students with summer internships at some of the most exciting startups in the world.

We have received an incredible response from colleges all over the country. I know because I receive an email each and every time a student applies for the program. Over the past few months, I have noticed a distinct pattern in the days and time that we receive applications. So, as part of the "data exhaust" of running FirstMark Elite, here is an overview of "when students apply for jobs". Presumably, these are also graphs of "when students aren't eating or having fun."

Here is a real-time FirstMark temperature monitor, created with a SmartThings starter kit, Highcharts.js, a little bit of PHP, and MySQL. (Doesn't look great on mobile yet.)

Now we'll always know whether it's LDL or RH temperature! This is how crazy simple the SmartThings SmartApp code is:

Using the yes.com API, I was able to load playlists for over 1,000 different radio stations. There is a lot of data to analyze here, but one of the first questions I sought to answer was "What are the most popular songs and artists on classic rock radio?"

Originally, I categorized any station that played Aerosmith and Led Zeppelin at least once per day as a classic rock station. An astute friend pointed out that I was "sampling on the dependent variable." To address this, I filtered for stations that played (i) at least one song by Pink Floyd, Led Zeppelin, The Beatles, Fleetwood Mac, the Rolling Stones, and Aerosmith, and (ii) played at least one song by all of those artists. The source data do not have reliable genre tags, so this is the most appropriate test I could think of; I am open to alternative suggestions.

Of the 1,000 radio stations, 91 are considered 'classic rock.' Across 1 week, 111,000 songs, 5,421 unique songs, and 1,448 unique artists, here are the twenty most popular songs:

   Sweet Home Alabama, Lynyrd Skynyrd (0.55 plays/day)

Sweet Emotion, Aerosmith (0.53 plays/day)

Carry On Wayward Son, Kansas (0.51 plays/day)

More Than A Feeling, Boston (0.49 plays/day)

Dream On, Aerosmith (0.47 plays/day)

Another Brick In The Wall (Pt II), Pink Floyd (0.47 plays/day)

The Joker, Steve Miller (0.46 plays/day)

Don't Stop Believin', Journey (0.45 plays/day)

We Will Rock You, Queen (0.45 plays/day)

Another One Bites The Dust, Queen (0.45 plays/day)

You Shook Me All Night Long, AC/DC (0.45 plays/day)

Renegade, Styx (0.45 plays/day)

Wanted Dead Or Alive, Bon Jovi (0.44 plays/day)

Magic Man, Heart (0.44 plays/day)

I Want You To Want Me (Live), Cheap Trick (0.44 plays/day)

We Are The Champions, Queen (0.43 plays/day)

Barracuda, Heart (0.43 plays/day)

Centerfold, J. Geils Band (0.43 plays/day)

I Love Rock 'N Roll, Joan Jett (0.43 plays/day)

Free Fallin', Tom Petty (0.42 plays/day)

  These are the twenty most popular classic rock artists:

Led Zeppelin (5.30 plays/day)

Van Halen (4.36 plays/day)

Rolling Stones (4.18 plays/day)

Pink Floyd (3.92 plays/day)

Aerosmith (3.63 plays/day)

Queen (3.59 plays/day)

Tom Petty (3.55 plays/day)

AC/DC (3.14 plays/day)

Journey (3.09 plays/day)

Eagles (2.98 plays/day)

Boston (2.82 plays/day)

ZZ Top (2.74 plays/day)

Steve Miller (2.60 plays/day)

Foreigner (2.59 plays/day)

Def Leppard (2.57 plays/day)

Who (2.42 plays/day)

Beatles (2.42 plays/day)

U2 (2.26 plays/day)

Police (2.16 plays/day)

I was backing up some photos the other day and spent 20 wonderful minutes flipping through pictures from our honeymoon. I thought it was a shame that we rarely took time to enjoy old photos. We snapped 1,700 on a month-long honeymoon in Spain, and have thousands more from other trips. Then, inspiration struck: why not get a regular email with a random photo from our personal archives? Thus was born Honeymailer, a completely passive and spontaneous way to enjoy some old memories.

Now, my wife and I get gems once a day, like this photo from Ainsa, Spain:

Here is the code for Honeymailer. It runs as a Google Apps Script. If you have any trouble setting this up, don't hesitate to leave a comment.

Based on a sample of over 2,000 tweets with the text "Chris Bosh looks like":

The Rock and Roll Hall of Fame completed its 26th induction ceremony about a week ago, honoring Rush, Heart, Randy Newman, Donna Summer, Albert King, and Public Enemy. This year’s crop of inductees struck me as more appropriate for a “Popular Music Hall of Fame” than one dedicated specifically to rock. In fact, if I were a betting man, I would put money down on a name change in the next 10 years.

The first hip hop artist to be inducted into the Hall was Grandmaster Flash and The Furious Five in 2007. Since then, Run D.M.C. (2009) and the Beastie Boys (2012) have also been inducted. Artists are eligible for election beginning 25 years after their first record, so a large cohort of popular hip hop artists will become eligible over the next decade; the Grammy for Best Rap Album was first awarded (to Naughty by Nature) in 1996. 

There are also 11 artists that have been inducted into both the Country Music Hall of Fame and the Rock and Roll Hall of Fame, including Elvis and Johnny Cash. Of course, there are at least three reasons this is not surprising: genres are fluid, artists’ styles evolve, and—perhaps most importantly—rock was born, in part, from country music.

Even Miles Davis, who “flirted with rock from the late 1960's on” was inducted as a performer, which the New York Times described as “provocative for a second, and then a little meaningless.” In any case, this made me wonder about the breadth of genres represented across the Hall’s 279 inductees, 179 of whom are performers.

I wrote a short script to crawl Wikipedia pages for each inducted performer and tabulate the genres mentioned. Neil Young’s page, for example, lists “rock” in the lead left position followed by “folk rock, country rock, experimental rock, hard rock”. This surfaced 144 unique genres, 50% of which are attached to only one artist, including baroque pop (Bee Gees), bubblegum pop (The Jackson 5), drone (The Velvet Underground), glam metal (Alice Cooper), jangle pop (R.E.M.), krautrock (David Bowie), spoken word (Leonard Cohen), and zydeco (Dr. John).

There is nothing extraordinarily surprising about the overall distribution of genres. The Rock and Roll Hall of Fame has, for the most part, been true to its name -- although we will see how much longer that remains true.

"Lead left" genre, all genres with at least 5 inductees

% of inductees tagged with most common genres

The following graph shows the total percentage of startups listed in Crunchbase that include the word "social" in either the company's description or tags, by year founded.

In response to a comment regarding percentages versus count of startups, I cooked up the following graph. This shows the total number of "social" startups founded in a given year (blue) and the total number of startups founded according (blue + gray).