I Quant NY

With New York City and State in a State of Emergency and the State on PAUSE, it’s no secret that the city has seen a drastic decrease in subway ridership - down about 93% year over year at its lowest point. 

The drop has largely been sustained, as you can see by this graph of daily year over year ridership changes since early March: 

The decline is striking, and subway ridership, as of this past Friday April 24th- the last day data is available - is still very much down.  You might notice two bumps on this chart, but those are caused by unusually low ridership on some weekend days in 2019 (probably due to construction), and so the relative drop in 2020 appears less extreme on those days. 

So lets instead look at this same data with just pure weekly ridership numbers, and not worry about what happened last year.  Here I plot the average daily ridership over the last seven days measured on any given day: 

The bumps are gone.  But there is something else striking.  It’s hard to see here, but the chart actually bottomed out on April 17th.  After that, ridership seems to be climbing through April 24th - the last day we have data.  The tick upward may seem small when you look at it compared to normal ridership, but by plotting this same data as a week over week percent change in average daily ridership, we can see how things are changing more easily:

Grey bars mean that less people rode the subway on an average day over the last 7 days than the previous 7 days before that.  Red means the opposite.   And where do we top out - the last day for which we have data - Friday April 24th.  The climb is actually quite striking if you think about it in terms of rides taken.  During the 7 days ending on Friday April 17th, over 2.7 million rides were taken.  Over the following 7 days ending on April 24th, the number climbed to 3.2 million rides.  That is roughly a 17% increase in ridership over the last week compared to the previous one, making up almost 500,000 additional rides over that time.  When it comes to reducing the spread of a virus, 500,000 rides is not trivial. 

So why might the ridership be creeping up without any regulatory changes made by the government? Part of it may be that people are starting to get stir crazy and the weather is slowly getting nicer.  But it’s scary to think that a nearly 20% ridership change can happen in a week without any official reopening. Given that, imagine what even a limited reopen might bring in terms of subway usage.    

Another possibility for the uptick is that on Saturday, April 18th - the very first day of a week over week increase (the first red bar) -  Governor Cuomo said in his daily virus briefing: “If you look at the past three days you could argue that we are past the plateau and are starting to descend, which would be very good news.”  The New York Times headline for the article covering that day’s briefing read: “New York Governor Sounds Optimistic Note as Coronavirus Numbers Improve”.  Many New Yorkers are tracking the virus carefully, and positive sounding statements from politicians could likely lead more people to feel comfortable heading outside.   

I should point out that it’s not just those in New York City that are starting to venture out more.  New York State Thruway Toll data, filtered for cars, shows an uptick over roughly the same period of time, first turning positive on April 16th.

Possibly sensing the shift in behavior, during Saturday’s briefing, Cuomo pointed out his daughter said "I can't do this anymore,” and made the point that those who are sacrificing are doing that for a noble reason, saving over a hundred thousand lives.  Even so, in the same briefing he said it was encouraging that we are “back to where we were 21 days ago” in terms of hospitalizations, likely without realizing that he could have also been referring to subway ridership.  This plot shows the last 21 days of that: 

Let’s just hope that trend on the right side does not continue ahead of any regulatory changes, since if increased ridership leads to more illness - and I don’t see how it wouldn't - then it could slow down the pace at which we could reopen the city.

Notes: 

After much delay, the New York City Department of Health recently released data on the number of tests given and the number of positive COVID-tests in each ZIP Code.  And what followed was a flurry of maps and analysis by news organizations hoping to pinpoint which neighborhoods were hardest hit. 

I recently pointed out that subway ridership had fallen much slower in low income neighborhoods than in higher income ones, leading me to believe that the effect of COVID may be harsher in those neighborhoods as more people continued to venture out as things worsened (likely due to the nature of their work).

A quick glance at this new dataset would at first suggest otherwise.  This map shows the number of positive tests per 1K residents in each neighborhood: 

It shows some wealthy areas of Manhattan having elevated levels.  And when I ran the actual numbers, there was no correlation between the median income of a ZIP Code and the proportion of a population that had tested positive.   

That being said, there is a problem with this metric - access to testing is quite varied.  If you are wealthy, have health insurance and a primary care doctor, your ability to get tested is going to far outpace those in lower income brackets that lack the same healthcare access.  Those who lack insurance might rely solely on a public hospital where they would only be tested if symptoms were severe and required hospitalization.  An urgent care facility might test with lesser symptoms.  Therefore, you might have to be less sick to be tested if you are wealthy than not. 

 If that were the case, then the proportion of tests in higher income neighborhoods that come back positive should be lower.  And indeed, that is very much the case (r=-0.50):

A map of the proportion of positive cases looks like this: 

This likely tells us that positive testing rates, given the disparities in testing, is likely not a good proxy for the rate of infection when comparing across income lines. 

Since we can’t get the information we want from positive tests, I turned to the symptom surveillance website that shares symptom counts from those visiting Emergency Rooms.  

A look at the overall number of visits to ER with respiratory symptoms over time shows a climb start on March 8th:

The same goes for Influenza like symptoms. As of April 1st, the number of visits per day with respiratory or influenza-like symptoms was 3,873.  The good news is that this is trending down from Mar 25th.  (Note that having less people walk in does not mean less people are in our hospitals, as many stay for a long time.  So the number of people in our hospitals continues to grow at an alarming rate).

The tool allows you to see the ZIP Codes of a subset of these emergency room visitors and so I plotted the per-capita rate of visits per ZIP Code below on a map.

There are many caveats to this data of course.  First, these are ER visits, not confirmed cases.  So some flu and other respiratory issue visits are in there.  Second, the city gives ZIP Codes for only a subset of visits, and it’s not clear if there is bias in that sample.  For example, if only a few hospitals reported ZIP Codes to the city, it could cause this to over sample some areas.  So the rates I provide are likely higher in each neighborhood, though the relative rankings should be reasonable.  Third, lower income New Yorkers are more likely to use ER visits for non life-threatening conditions because they might lack access to other healthcare options.  So this could end up over-reporting lower income communities as having more cases.

With all of those caveats aside, here is the map of ER visits with symptoms of influenza or respiratory issues per capita in each ZIP Code Mar 8-Apr 1:

A quick glance seems to show that lower income communities might have higher ER visit rates for respiratory symptoms over the last 3 weeks than higher income ones.  As noted before, this could likely be due to the ER playing a different role in healthcare in lower income communities.  A quick analysis seems to confirm that there is a relationship between median income in a Zip Code and the ER visit rate for respiratory symptoms (R=-0.51):

Each dot represents one Zip Code.  The scatterplot shows a relationship between income and the rate of visits, where no higher income communities have high rates.  The plot shows a few outliers, so I am listing them below: 

And here is the striking thing - it turns out there is no mathematical correlation between hospitalization rates and positive test rates, adding the evidence that positive test rates is not a great proxy for understanding the spread.  

And to bring things full circle, let’s take a look at hospitalization rates compared to the reduction in subway ridership in each ZIP Code. Again, the theory is that neighborhoods that kept taking the subway in higher numbers might be at higher risk.  The scatterplot below shows that to be the case (R=0.44): 

As the severity of the COVID-19 crisis in NY accelerates, the subway is still running.  There have been asks by both Mayor De Blasio and Governor Cuomo that people should not crowd into subway trains - they should instead wait for the next one with more space.  If you have ever taken a subway at rush hour, you know that ask is a ridiculous one.   Nonetheless, it might be reasonable advice as the number of riders starts to drop on New York City’s subways.  And that really started to happen as of Friday (Mar 13th), which saw a citywide drop of about 30% when compared with the Friday the week before (Mar 6).  

Even though it was a 30% drop overall between those two, each station saw it’s own unique drop in ridership, and I figured that those drops would not be evenly distributed. Though many companies moved to telecommuting last week, companies like retail operations and restaurants still need their employees to come if they are going to remain open.  So, those employees can’t simply call in.  Given that, I wondered if the drop in subway ridership was going to be larger in more affluent neighborhoods that have a higher proportion of employees that can telecommute?  

To find out, I made a map of NYC subway stations, where small circles indicate large decreases in the number entrances at that station, and larger ones indicate small decreases.  This is a bit odd, but the idea was to make it clear where there was relatively MORE sustained ridership when compared to other stations. Clicking on a station will give you the name and the % change for that station.

It becomes abundantly clear that as you go farther out - away from Manhattan - the drop in subway ridership is far less extreme.  Those generally are also less affluent neighborhoods.  To test that theory, I made a quick scatterplot which reveals a -77% correlation between the income of the census tract of the station, and the change in ridership.

What’s troubling is that this may mean an additional health burden on those who may have less access to quality healthcare and are less likely to have paid sick time (or a security blanket for lost income).  The health risks of keeping the subway open are not evenly distributed among income groups.

Data used: 

Subway Turnstile Data (via MTA)

Median Income by Census Tract (via esri)

I’ve lived in New York for over fifteen years, and I’ve never seen the City as paralyzed by snow as it was on Thursday. I’ve heard story after story of stranded friends and colleagues. Major thoroughfares became clogged with stuck cars.  It was almost like - dare I say it - Washington DC when it snows.

So what happened?  

NYC Mayor De Blasio at first told NY1: 

“When it finally became clear we were going to get fast and intense snow, rush hour was already beginning. Everything was having a multiplier effect in a bad way.”

He later seemed to admit he had gotten the revised forecast that morning well before rush hour:

“I think the crux of this matter was on Wednesday night the National Weather Service was saying 1 to 2 inches. Around Thursday morning, it jumped up to 3 to 5 inches,” 

Interestingly, WPIX actually reported as early as Wednesday morning -- the day before the storm --  that one of the two main models used for prediction was predicting 5 inches for New York City, which is close to what we got: 

Total accumulations for the city vary between the various forecast models.

The "global" model is indicating mainly a rain event for the city with little to no snow accumulation, whereas the "North American" model is forecasting close to 5 inches for the city.

So we should not have been completely blindsided, though it sounds like the National Weather service did not give the right warnings until the morning of. Let’s give the city the benefit of the doubt here and say we had no idea what was happening until Thursday morning, a few hours before the snow started coming down.   One question is as soon as we got the revised totals, we must have pushed to bring more plows out when things started to look bad, right?  

To find out, I downloaded raw data snapshots from PlowNYC (which had only been viewed 41 times before apparently), and decided to see if I could get a read on the city’s snow plow activity.

The data is stored in an odd format - every 15 minutes it tells you the last time every street was serviced.   We can’t see the number of trucks out which I was curious about, as that is the main knob the city likely has to turn.  But we can see the rate that streets are plowed, which is going to be very correlated with that.

To investigate this storm, I took snapshots from 3PM (about 1.5 hours after the snow began to come down), 5PM  and 7PM.   If the city decided to change strategy that morning, and push to get more and more plows out ASAP, we should see the number of streets recently plowed start to go up as we get later in the day and more trucks become mobilized.  By 5PM, it was pure chaos out there, so we must have pushed hard for a change, right?

To measure this, I made a chart of the number of street segments plowed in the last N minutes, where N is the x-axis of the chart.   As N increases, more and more street segments are included as you are looking farther back in time.

The data shows that about 20,000 street segments were being touched in the previous 30 minute period at 3PM (in orange).  What is odd is that you can see what looks like a very similar rate at 5PM in grey and then again at 7PM in yellow.   That means that even while the commute was turning into complete hell, the number of plows on the street do not seem to have increased in any meaningful way.  Puzzling given what was happening around the city.  I realize you can’t mobilize plows immediately, but I was still surprised to see this given the updated weather received in the morning, and how bad things were going.

Another question is what percent of our plows did we have out for the storm?  Is 20,000 street segments in 30 minutes close to maximizing our plow capacity? To find out, I compared this snow storm to a storm last season, marked in Blue.  That storm, on March 21st, dropped 8 inches or so, but was expected to drop much more.  So we probably fully mobilized for that one.

The chart shows this historical storm at rush hour in blue. You can see the rate of plowing is almost double, (about 40,000 street segments per 30 minutes of plowing).  This helps us quantify the different in equipment deployed between a storm we take seriously and one we don’t.    

So in the end, the data shows us two things- first we seemed to have rolled out last week at about half our capacity and that cost us. Second, we did not ramp up our deployment even as things got worse and fell apart.

I was also curious what the state of the world looked like at 5PM, so  I made a map which shows how long it had been since each road had been plowed when rush hour came. Street segments in black had not yet been serviced at all.  The others you can roll your mouse over to get a sense of how many minutes it has been as of rush hour, green being the last number of minutes and red being the most: 

Open Data creates a new layer of transparency in government, and here we’ve used that transparency to compare what seemed to have happened against what we are told. Of course you can't just make plows appear quickly, and I get that. But it's interesting that even in one of the most dier snow situations I've ever seen in NYC streets, we as a city, were unable to reacllocate resources in any thing close to real time. It's probably something our ellected officials should thing about for the future.  

Raw Data Used: 

PlowNYC Data from NYC Open Data portal

The headline in a recent NYC press release caught my eye: “MAYOR AND CITY COUNCIL LAUNCH SEARCHABLE OPEN BUDGET FOR NEW YORK CITY”.  I was pretty excited.  As mentioned in my talk on Ted, NYC has entombed this data in PDFs for years, making it basically impossible to analyze and understand what is going on.   But for the first time, we can actually do things like look at the top spending for each agency.   This is a big deal.  

The bad news in all of this is that the City only put this year’s budget (FY2017) on the Open Data portal and left all prior years off. This makes it impossible to look for trends and year-to-year changes, which is exactly what is most interesting in a budget - so that was a real disappointment from a progressive administration. The omission is especially silly given that many years of prior budgets are in the exact same format as this year’s budget, making export to the Open Data portal equally easy.  

Now to the data.  After a quick look, it was clear that the most granular (and thus most interesting) open budget data available is the data associated with the Supporting Schedule.  It categorizes expenses all the way down to the “Object Code” Name.  Object Codes include everything from “Full Time Position” costs and  “Overtime”  to “Postage” and “Telephone Services.”  Each Object Code is assigned to an Agency, so you can track the expenses from each agency.   Not before the Open Data release, if you wanted to, for example, understand the NYPD’s biggest expenses, you would have to troll through hundreds of pages of PDFs.  Now, one just has to know how to use Excel or program a bit. Major steps forward.

And it turns out that  it was this exact question, “what are the NYPD’s biggest expenses,” that led me to a discovery in the very first place I looked in the FY2017 budget. Let’s have a look at the largest 20 Budget Codes for the NYPD:

Does anything stand out?  Just a little?  Protection of Foreign Missions leads the way here.  And not just leads, but conquers.  In fact, according to this year’s adopted budget, we are going to spend more on protecting foreign missions than School Safety, Transit, Housing and Narcotics combined! This amounts to about 1% of NYC’s entire budget and 15% of the NYPD’s entire budget.     

Seems unreal, right? Might this be a data problem with the new Open Data release?  Nope. The same thing is right there in the Budget’s Supporting Schedule PDF as well, split between pages 1125 and 1126.  

So our city voted on and adopted this budget, putting that much money to protecting foreign missions?  Yup.   But before you get all upset about this crazy distribution of police funding, I’m going to go out on a limb and say that this is, in fact, a $800 million typo that, for some reason, no one noticed in the budget.  

Here’s the proof:

In a 2012 statement, Ray Kelly pegged the amount budgeted for the protection of foreign missions at $27 million.  A 30 times increase since then would seem nearly impossible.

More convincingly, last year’s budget did not even include the protection of foreign missions as a line item, but the same number of staff resources and a similarly large budget size was given to “Chief of Department” instead of “Protection of Foreign Missions”. (See pg. 624 here.)

So unless the Chief of Department staff is now exclusively doing protection of foreign missions, this is a typo.  (If I am wrong, then I have much bigger questions!)  What’s so troubling to me is that if anyone within government had used the data of the budget to even shallowly analyze NYPD spending at a high level, this would have been caught.  After all, it was in the proposed Executive Budget in April and stayed in there throughout negotiations with the Council that ended in June.  And then it was voted on and adopted.  

The fact that this amount of money made it through the budget process presumably mislabeled makes the case for Open Data even stronger.  Open Data is not just about “transparency”.  Our government officials are only human and our agencies have limited resources. Budgets are far too large and complex to be understood end-to-end by our legislators.  As more data gets out to the public, we’ll start to see that our citizens can help improve the way government operates - even if it’s sometimes proof reading!  

Though I wish they put up more history, credit should be given to OMB for getting this data out at all.  It is a NYC first.  And with that, I’ll leave you with this last piece of data:  Number of times “Open Data” appears in the NYC budget?  0.

----

Owning property in NYC is a privilege- albeit one that comes with a hefty tax bill each year.  Property taxes are essential to NYC’s bottom line - in 2015, for example, they made up 27% of the city’s $70 billion in revenue.  And yet data on the specific sources of these property taxes is fairly hard to come by.  That’s because New York City has done limited work to create access to property tax information on the open data platform.  However, you can freely access each individual bill one-at-a-time online.  When John Krauss reached out to the Department of Finance with a FOIL request for this very data, he was told it would cost “approximately 1,000 hours at a cost to you of $50.36 per hour” to get it. That’s right, 6 months of employee time and $50,000 for some lines in a table that were already online.  Oy. 

Not being one to take no for an answer, John wrote a PDF scraper and downloaded the PDF tax bill for each and every lot in NYC in 2015, and I can assure you, he did not spend 6 months on it!  Then he worked with Chris Whong to convert these PDFs into tables and the results are at taxbills.nyc.  And there you have it, a new dataset was born.   As long as government keeps putting out PDFs without access to underlying data, we will have to go to the trouble to create datasets in this roundabout way.

Whong’s first look at the data showed that the sum of exemptions and abatements totaling $12.9 billion, a staggering number considering the total tax due is about $21.6 billion.  But many of these tax abatements go to public agencies.  Properties like parks ($1.2 billion in exemptions), and those run by the Department of Education ($926 million in exemptions), as well as the Port Authority ($913 million in exemptions) top the list.  But sitting at number 6 on the list of the 156 different abatements and exemptions are Houses of Worship, totalling $476 Million.  A closer look shows that religious exemptions add up to over $650 Million a year - about 1% of the City’s budget.  To put that in perspective, it’s more than the exemptions that NYCHA received in 2015 ($427 million), which suggests that religious institutions have more valuable real property than our public housing stock.  

With underlying data, we can take a look at these religious property tax exemptions with far more granularity.  Here’s what I found: 

The Types of Religious Property Tax Exemptions Available

Looking through the list of exemptions, I identified eight categories that seemed to be religious in nature: “House of Worship,” “Religious Dormitory,” “Clergy,” “Parsonage,” “Religious Mission,” “Bible,” and “Salvation Army.”  Of those, House of Worship and Religious School make up the vast majority of exemptions.

Where Are These Exemptions Given?

In short, everywhere.  Mapping each of these lots looks like a view up towards the heavens, but with a lot more money involved.  Check out the religious tax exemptions in your neighborhood:

Clicking on any building will give you the exemption type, amount and the name of the religious institution.

There is a “Bible” exemption taken by just two properties in NYC...including one owned by Jews for Jesus

Though I see no “Torah” exemption, there is in fact a Bible Exemption!  Plus, one of the two properties in NYC that is taking it could apply to either.  It’s “Jews for Jesus” and the tax bill shows that they save $133,249 in taxes a year with the Bible Exemption.  The other is held by the Faith Bible Seminary in Flushing. 

Where does this bible exemption come from you may ask?   Well State law offers the exemption if the property is used for “bible.”  What does it mean to be used for “bible” with no verb?  We’ll have to ask these two property owners, since all other religious institutions used other exemptions.  

The Salvation Army has it’s very own Exemption!

In what seems to be a unique situation, the Department of Finance issues an exemption called the “Salvation Army” exemption to- wait you guessed it -- The Salvation Army.    You can see it right there on the tax bill.   44 Properties in NYC take that exemption, totaling 2.3 million dollars a year.   

South Jamaica Has the Most Houses of Worship per Capita

Tax bills can also teach us about the composition of our communities and neighborhood land uses.  South Jamaica, the neighborhood with the most Houses of Worship per thousand residents, has five times more per resident than NYC as a whole. They have one house of worship for every 271 residents, compared to one for every 1400 or so residents in NYC as a whole. Darker areas on the map below have more houses of worship for capita. 

  The top 20 neighborhoods are below: 

Clergy May be Priced out of Manhattan, but there are Plenty Living in Ocean Parkway South

Though there are no shortage of Houses of Worship in Manhattan, there does seem to be a shortage of home-owning clergy members.  A map exploring clergy exemptions per 1,000 residents of each neighborhood shows this acutely: 

Astoundingly, only 5 clergy exemptions are taken in all of Manhattan, compared with 667 in Brooklyn and 404 in Queens. Ocean Parkway South has 2.5 Clergy Exemptions per thousand residents, a rate 20 times higher than the city average.

Religious Exemptions add up to $76 dollars per NYC resident 

$650 million dollars in exemptions is nothing to sneeze at, but it adds up to about $76 dollars per resident of NYC.  That means a family of four on average might pay $300 more a year in taxes to make up for the lost income.  So while many religious institutions don’t charge membership fees, the cost is being passed on indirectly to everyone. 

The neighborhoods receiving the largest amount of exemptions per resident are some of our wealthiest.

Wealthy neighborhoods have high land values.  High land values equals high property taxes.  So it might come as no surprise that the wealthiest neighborhood in NYC, the Upper East Side-Carnegie Hill, has the third largest amount of exemptions, at $610 per resident.  Midtown has a higher rate per resident, but that is largely because that neighborhood has relatively few people who live there, and in addition, the property values are high.   More religious areas, like Williamsburg and Borough Park also appear high on the list, due to the large number of exemptions.

Communities with Large Jewish Populations Have the Most Religious Schools per Capita

NYC has one religious school for every 14,000 residents or so, but Williamsburg has one for every 1200 (note this is the Neighborhood Tabulation Area (NTA) for  Williamsburg, which has a high concentration of observant Jews).

So we’ve used these tax bills to explore both some issues about our city’s revenue, and the religious fabric of our city.  Call me crazy, but I dream of a world where we can explore the fabric of our own city without having to write PDF scrapers. A world where it does not cost $50,000 for the City to write some data to a file. But until then, we’ll just have to rely on our passionate citizens to keep pushing the ball forward, making “public” data actually public because its the right thing to do.  That sounds almost religious... wait a minute, I’ll be right back.

---

New York City is a complex place to drive.  And when it comes to parking, there are plenty of rules and regulations to follow.  It’s no wonder that sometimes people get confused and end up getting their cars ticketed or towed. 

But in all of these rules, there is one thing that very few drivers seem to know. As of late 2008, in NYC you can park in front of a sidewalk pedestrian ramp, as long as it’s not connected to a crosswalk.  It’s all written up in the NYC Traffic Rules, and for more detail, take a look at this article. The local legislation making these parking spots legal was proposed by Council Member Gentile, and adopted by the Department of Transportation before it ever made it for a vote.  Though few people seem to know about the change.

Is it a problem that drivers don’t realize that there are some extra parking spots they are now allowed to park in?  Not so much.  But, I’ve got a pedestrian ramp leading to nowhere particular in the middle of my block in Brooklyn, and on occasion I have parked there.  Despite the fact that it is legal, I’ve been ticketed for parking there.  Though I get the tickets dismissed, it’s a waste of everybody’s time. And that got me wondering- How common is it for the police to give tickets to cars legally parked in front of pedestrian ramps?  It couldn’t be just me...

In the past, there was not much you could do to stop something like this. Complaining to your local precinct would at best only solve the problem locally.  But thanks to NYC’s Open Data portal, I was able to look at the most common parking spots in the City where cars were ticketed for blocking pedestrian ramps.   It’s worth taking a moment upfront here to praise the NYPD for offering this dataset to begin with.  Though we are behind on police crime data in the city, we are ahead in other ways and the parking ticket dataset is definitely one of them.  

What I found when I dove into the data surprised me.  To start, I found the top address where this ticket were given: in front of 575 Ocean Avenue in Brooklyn, where over $48,000 in parking fines were issued in the last 2.5 years. 

To my surprise, the spot, (or really spots since there are two ramps), are legal, since they are in the middle of the block, with no crosswalk.  $48,000 in tickets at a legally parked spot, and that is just the last 2.5 years.  If we had more historical data, I suspect we would see a similar story. 

The next top spots on the list had the same story to tell.

1705 Canton Avenue in Brooklyn, 273 Tickets, $45,045: Legal.

270-05 76 Avenue in Queens, 256 Tickets ($42,440) Legal.

143-49 Cherry Ave, Queens, 246 Tickets, ($40,590).  Legal.

Those were of course the most extreme cases that topped the list.  I started to skip down the list.  This spot in Battery Park, ranked #16 on my list and the top spot in Manhattan, had 116 tickets ($19,140) and turned out to be legal.  (Yellow paint has no meaning in NYC traffic enforcement, and the spot has no crosswalk)

I started to skip down the list faster and faster.  Take #1000 in my top list, at 1059 Virginia Avenue, where 8 tickets had been given ($1320).  It is a classic T intersection, meaning it’s legal.

See the pattern?  I did.  So I then selected 30 random spots that had received 5 or more tickets over the time period, and based on Google Maps found that all of them appeared to be legal parking spots!  (Randomly selecting spots with a single ticket in the database showed some illegal spots as well, so I chose 5 as a conservative cutoff.)

How many spots received 5 or more of these pedestrian ramp tickets in the last 2.5 years?  We are talking 1,966 spots that are generating about 1.7 million dollars a year in tickets at parking spots that are mostly legal.  Are all 1966 spots legal?  Surely not, but the majority sure are and many more that have fewer than five tickets are likely legal too.  

To see where the hotspots were for this sort of abuse of drivers, I mapped the top 1000 pedestrian ramps generating parking tickets below, so you can see if you have a spot in your neighborhood that the police have been wrongly ticketing.  Clicking on a spot tells you if the tickets are given in front of, or opposite the address, as well as how many tickets have been given.  

My challenge to you...  click on a spot and then look at pictures of it on Google Maps to determine if its legal.  You will most likely find it is. 

Other findings in the data:

Brooklyn’s 70th Precinct seems to have the most cars wrongly ticketed, bringing in over $100,000 in fines a year.  The 77th, also in Brooklyn, comes in second:

So what is going on here? Well, it seemed from my browsing that many police officers were systematically ignoring the 2009 rule change, all across the city.  

But that is not where the story ends. As mentioned, we live in a city progressive enough to release parking ticket data.  I’ve repeatedly said that putting data into the hand of citizens will make our city run better and more equitably.  So, I reached out to the NYPD via the Mayor’s Office of Data Analytics and Manhattan Borough President Gale Brewer’s Office (who is one of NYC’s earliest NYC’s Open Data champions)  to tell them about the findings.  They helped me get in touch with the appropriate people at the NYPD, and then I waited.

After a couple of weeks, I received the following quote from the NYPD:

"Mr. Wellington's analysis identified errors the department made in issuing parking summonses. It appears to be a misunderstanding by officers on patrol of a recent, abstruse change in the parking rules.  We appreciate Mr. Wellington bringing this anomaly to our attention.

The department’s internal analysis found that patrol officers who are unfamiliar with the change have observed vehicles parked in front of pedestrian ramps and issued a summons in error. When the rule changed in 2009 to allow for certain pedestrian ramps to be blocked by parked vehicles, the department focused training on traffic agents, who write the majority of summonses.

Yet, the majority of summonses written for this code violation were written by police officers. As a result, the department sent a training message to all officers clarifying the rule change and has communicated to commanders of precincts with the highest number of summonses, informing them of the issues within their command.

Thanks to this analysis and the availability of this open data, the department is also taking steps to digitally monitor these types of summonses to ensure that they are being issued correctly.”

I was speechless.  THIS is what the future of government could look like one day. THIS is what Open Data is all about.  THIS was coming from the NYPD, who is not generally celebrated for its transparency, and yet it’s the most open and honest response I have received from any New York City agency to date. Imagine a city where all agencies embrace this sort of analysis instead of deflect and hide from it.   (For comparison purposes, see NYC Department of Health response to earlier work as example of how NOT to respond to Open Data.) 

Democracies provide pathways for government to learn from their citizens. Open data makes those pathways so much more powerful.  In this case, the NYPD acknowledged the mistake, is retraining its officers and is putting in monitoring to limit this type of erroneous ticketing from happening in the future.  In doing so, they have shown that they are ready and willing to work with the people of the city.  And what better gift can we get from Open Data than that. 

------

See Previous Government Changes brought on by this blog below:  -DOT Repainting Streets due to confusing fire hydrant -TLC Reprogramming Cabs due to odd tipping algorithms -MTA Redoing subway vending machines due to change issues.

See all blog coverage on NYPD here.

------

As noted in my last post comparing NYC neighborhood murder rates to American cities, the City has finally released raw crime data for the first time in its history. But the more I look at data released by other cities, the more the data released by NYC seems simply inadequate.  I have found over a dozen cities which release raw crime data, but NYC is the only city I have found that has filtered the crime data down to a subset of seven major crimes, (a subset which just so happens to paint the rosier picture for the Administration.  Those numbers are down and not controversial.)

Though I continue to call for NYC to stop filtering most crime out of the data set, that won’t stop me from finding value in what has already been released to date.  And my question for today’s post: What times of day do major felonies happen?  

First, a look at the hourly crime rate for the Major Seven Felonies below:

It’s no surprise that crime dies down at 5 or 6 AM, when most everyone is asleep. (Note that 5AM on these charts indicate everything from 5:00-5:59AM) But some other patterns stand out in the chart.  First, there are some odd spikes in Grand Larceny (shown in green) at noon and midnight. I can’t be sure, but I’d bet those are police officers entering in some sort of default times when victims are unsure exactly of the time of a crime.  But two other spikes caught my eye - one at 8AM and one at 3PM.  The 8AM one is most seen in the dark blue Burglary line, but you can see traces of it in other crimes. The 3PM spike is most pronounced in the light blue Robbery line, but once again traces show up in other crimes.  (Note that Robbery involves force, while Burglary involves breaking and entering.)

Now what do 8AM and 3PM have in common?  They are more or less the bookends of the school day in most NYC high schools.   So that had me wondering something...   Could the end of the school day really lead to a noticeable spike in crime? 

To find out, I looked at only crime that happened during the school year (after the first day of school but before the last), and split crimes into three buckets- School Days, Vacation Weekdays and Weekends.  If my theory about school days was correct, then we should not see any spike in these crimes on Vacation Weekdays or Weekends at 3PM, but we should see a spike on School Days. Let’s first use Robberies as an example, since it had the largest 3PM spike:

This chart said a lot to me.  The 3PM spike (and the smaller 8AM spike) appear on School Days, but don’t appear on Weekends or on Weekdays when school is out on vacation.  

Vacation days only happen a few weeks a year, so to double check my numbers, and remove any seasonal issues that may arise from the fact that vacations are more frequent towards the end of the school year , I reran the analysis only looking at crimes in February or April, two months with week-long school vacations.  That’s a bit more of an apples to apples comparison.

Once again, the hypothesis that crime increases after school gets out seems to be true.  A spikes appears centered at 3PM and a smaller one appears at 8AM. Strikingly there are two times more robberies between 3 and 4PM on school weekdays than non-school weekdays in February and April.

Burglaries, on the other hand, see a morning spike on school days. They seem to be about 50% higher between 8AM and 9AM on school day mornings than non-school day weekdays:

Felony assault and Grand Larceny also seem to be a bit higher between 6AM and 5PM on school days compared to vacation days, though the increase is much more slight.  Both of these also spike at the wee hours of weekends.

Not all crimes show increases when kids go to school.  For example, car theft seems to be relatively unaffected, showing the same pattern on both types of weekdays.

Murder and Rape both have too few instances to draw this kind of comparison, so I’ll skip them for this analysis.

The final question for me was whether more crime is committed on these school days overall, or if the crimes just happen at different times.  A quick look at the daily crime rates tells us a bit about that: 

It turns out that School Days have 15% more Robberies than Vacation Weekdays, and School Days have about 8% more Burglaries, Felony Assaults and Grand Larcenies.  

All of the analysis I’ve shown above was over the last ten years, but how has this changed over time?  

Here’s a bright spot in all of this.  Comparing the red and blue lines shows the difference between 2006 and 2015 specifically, and we see a drop in robberies at all times of day.  But what is more impressive is that while robberies have dropped 29%, robberies on school days have dropped 32% and robberies committed between 3PM and 4PM on school days have dropped 44%!  While a large part of that 3PM decline has happened under the deBlasio administration in 2014 and 2015, it’s unclear from this analysis if increased after school programming has played a role in it.  (The biggest percent drop (~11%) from 2014 to 2015 once again appears at 3PM).  In the end, this chart makes me pretty optimistic about the future of our city and its youth.

Crime data tells us a lot about our city.  It can be used to understand where more resources might help improve the quality of life for all city residents. I’m happy to see this data being released to the NYC public for the first time.  But the insights will be much bigger when the city stops filtering out most types of crimes from the data before releasing it and gives all New Yorkers access to the data sets that make up the fabric of our city.  I look forward to that day, but in the meantime I’ll still be quanting!

----

There is a lot of talk about NYC being a global leader in Open Data. But when it comes to police data, New York City lags well behind its peers in transparency. Los Angeles, Chicago, Seattle, Denver and Boston all have released comprehensive incident level crime data, allowing residents to understand where and when many types of individual crimes happen.  

Up until very recently, the NYPD and the de Blasio administration has kept all of that data to themselves. But there is evidence that the City is starting to take transparency just a bit more seriously.  A few months ago, we saw our first signs of this- the release of nine months of historical crime data.  At the time, I pointed out that although this was a small step forward, we lacked the longer data history that other cities had.  But just a few weeks ago, the City quietly turned that nine months of data into ten years of history.  No fanfare, no press release, just crime data.

For Open Data advocates here in NYC, this is a really big deal.  And eventually it will be a big deal for all New Yorkers- once we get the appropriate tools in people's hands and novel insights start to come forward. Because for the first time, we can see what block each crime happens, as opposed to what precinct. 

So given this latest release, why is New York City still behind other cities on police data transparency?  Well, to date, New York refuses to release data on crimes beyond the “Major Seven” felonies.  And its the crime data currently being withheld by the City that is exactly the data that is most controversial in a “broken windows” oriented police department. We lack visibility on so many of the quality of life crimes  It’s disappointing that even with a progressive Mayor, there is no sign of any movement in that data.  

Fortunately, the missing data does not preclude us from doing something with what we have so far, and I plan to do just that in a series of posts over the next month or so.

My first finding?  Well, NYC is really safe.  I know, I know.  We always hear that.  But when I heard recently that our most dangerous neighborhoods were safer than some cities in the US, I had to find out if that was true.

To do so, I took nine years of murder data from the FBI and calculated the average homicide rate (annual # of homicides per 100K residents) for cities in the US with populations above 250,000.  There were 72 such cities in the FBI data.

The 20 cities with the highest murder rates are shown below: 

Despite its size, New York comes in in 54th place in this metric.

I did the same calculation (annual Murders/100K residents) for NYC’s neighborhoods from 2006-2015- something which was not possible before this data release.

The neighborhood with the highest murder rate turns out to be Ocean Hill, followed by neighboring Brownsville.  Though the murder rate in Ocean Hill is about five times higher than NYC’s overall rate, it is still lower than the entire cities of New Orleans, Detroit, St Louis, Baltimore and Newark!  Of course there are parts of each of these cities that are safer than Ocean Hill, and I am comparing neighborhoods with entire cities.  But still, I found this to shed a new light on just how safe New York City really is. By comparing neighborhoods to Cities, I am essentially taking the highest murder rate in New York City, and showing how it compares to the average in those cities. It's like saying the hottest day in City X is colder than the average day in City Y. I think this approach gives a much clearer impression of the seperation between the two cities than simply comparing the averages of both.

So what happens if we compare each neighborhood’s murder rate to one of these cities?  I made a map that does just that.   Each neighborhood in the map is labeled with the city on my list that has the closest murder rate above its own.

Notice that some of the higher crime parts of Brooklyn share crime rates with Newark, Oakland, Cleveland, Chicago and Sacaramento. Some higher crime areas in the Bronx share crime rates with Pittsburgh, Millwaukee and Houston. You may also notice a lot of comparisons to Honolulu in some of the more safe areas of the City. It turns out that Honolulu was the safest city in the data set, and so many low crime areas of the city end up getting compared to it.

A full list of NYC neighborhoods, their rankings, and their comparable city is below:

Data from nyccrimecompare.silk.co

In the end, these comparisons gave me a new way to contextualize some aspects of crime in NYC's neighborhoods. They also gave me a new reason to visit Honolulu - besides the weather.

FBI data found in Table 8 of each year here.

NYC Crime data found on Open Data Portal here.

Neighborhood (NTA) Shape files NTA found here.

Neighborhood (NTA) populations found here

In November, 2004, the Trump Palace on East 69th Street received a visit from NYC’s Department of Environmental Protection and ended up with a violation for “Unregistered Fuel Burning Equipment.” The building agreed to the violation in exchange for the right to pay a reduced fine- something called a “stipulation”. Though they paid that fine, it appears that they did not pay everything off in time, leading to a $105 additional fee. Fast forward 11 years, and that fee is still unpaid according to city records from the Open Data Portal.

In fact, those records, along with the City’s ECB violation website, show that Trump’s various luxury properties in Manhattan owe the City $27,536.56 in unpaid fines for 19 violations over the last 12 years.  (Payment is generally due 10 days after a hearing).

A full accounting of those violations with balances still due are below.  (Higher res version here.) 

The most common unpaid violation is for the “Failure to Submit an Annual Test Report for a Backflow Preventer”.  A Backflow Preventer prevents contaminated water or chemicals from flowing back into the drinking water supply if there is a sudden or unexpected change in water pressure.  Failure to send in a test report does not sound so bad, though the unpaid “Failed to Install a Backflow Preventer” at Trump Parc in 2008 is a bit more unnerving.  Even so, Trump Parc still owes $700 on that ticket more than seven years later.

While Trump properties have yet to pay the outstanding balances above, the same data shows they have paid for plenty of other violations. In fact, his Manhattan properties have been cited for violations with listed fines totaling $304,165, on 460 separate violations since the year 2000. This total only includes those violations which were not dismissed, which listed the word “Trump” somewhere as a recipient, and which happened at one of his properties. The 460 violations happened at 14 separate Trump properties: 

It seems like compliance is not getting any better overall. It turns out that 2015 had the highest sum of violations yet: 

What were the violations for?  The most common violation was the “Failure to Maintain an Elevator,” which Trump’s buildings were fined for 62 separate times. The next four most common issues were all from the FDNY for issues with fire systems, documents, signs or instructions.  There were also four additional elevator issues which were considered “hazardous”. 

A closer look at the elevator issues over the years reveals that the two biggest violating buildings were the two that Trump sold shortly after getting those violations -  the GM Building and the Empire State Building. (Note that the violations here are only ones accrued during Trump’s Ownership).

All in, most of the violation money went to 175 FDNY violations but eleven asbestos control issues came up as well.

A more thorough look at the asbestos violations show they all happened at Trump Parc at 502 Park Avenue on two distinct dates.  Seems like they may not have learned the first time around.

Though Trump properties offered defenses in most cases, only 6% of the all of the violations were dismissed: 

So if you catch Trump at a rally anytime soon in NYC, tell him the properties he owns or manages owe the people of New York a bit of money.  You can also mention that the White House has three elevators in it. Given his record, he may just not want to take that on. ---

I stumbled upon a glorious data set a few weeks ago, which gives information on all unpaid parking tickets in New York City back to 1998.  Sounds cool right? Whats not so cool is the fact that the City loaded many rows of the data in there twice accidentally.  That meant there were multiple rows with the same ticket number and conflicting outstanding debt amounts.  Though I understand that data errors happen, I don’t understand how the City can keep putting out data sets with no ownership and no effective way to send in fixes.  A city who cares about the usability of its Open Data can do better.    

Anyway, once I cleaned up the duplicate data, the first thing I wondered was which license plate owes the most money in unpaid tickets.  But when I started poking around to see which license plates had the largest outstanding fines, I noticed many of the worst offenders had a familiar pattern. License plates 001THD, 003THD, 118THD and 129THD were all towards the top of the list.   

After some research, I realized that these were diplomatic plates. From there, one thing led to another and I found myself calculating how much each countries diplomats currently owe NYC in outstanding parking tickets. A clickable map of the world showing just that is below:

All in, diplomats owe NYC $16,024,266 in tickets. The top 20 countries in terms of their diplomatic debt are listed below.  Egypt tops the list with about 2 million dollars currently owed.  Nigeria and Indonesia round out the top 3.  

This table begged the question- how did Egypt rack up so many tickets?  A quick look at the worst individual offenders list adds some color: 

The car with license plate 001THD racked up $109,165 in fines, which have now ballooned into $223,350 after penalties and interest! That is 1,985 unpaid tickets on a single car (including blocking hydrants 126 times). In fact, the top four Egyptian diplomatic plates owe over $724,000!  It’s also interesting to note that an Italian diplomatic plate still owes money for 747 tickets, the top of any European country.

The top two hydrant blockers were two Alabanian diplomatic cars with 001GPD (247 hydrant tickets) and 006GPD (172 hydrant tickets).

Digging around some more, I noticed that most of this debt is owed on tickets before 2002.  

What happened in 2002?  Mayor Bloomberg came up with a plan where the state department would refuse to reregister cars with large outstanding parking ticket debt. The press release from the city at the time stated:

The Parking Program, which finalizes an agreement reached in principle on August 9, requires Diplomatic and Consular officials to pay future parking tickets and a substantial portion of the parking debt that has accrued since 1997. ...

In addition, if Diplomatic and Consular officials do not pay their future parking tickets the State Department will suspend or refuse to renew their registrations and the City will reduce or eliminate the parking spaces assigned to each mission or consulate.

It seems that the program did a remarkable job at reducing unpaid fines, as unpaid tickets dropped by an astounding 95% in one year!  But the whole thing about the officials “paying a substantial portion of parking debt accrued since 1997?” Yeah- that clearly didn’t happen. 

Over the last few years, we’ve started to see the unpaid tickets start to tick back up.  That may be because it takes years to pay... who knows.  But I was wondering who the worst offenders are in the period after the new rules came into effect, post 2002.

It looks like Indonesia has taken over Egypt’s spot on top of the unpaid ticket list, followed by Italy.  Indonesian diplomats owe over $25,000 since those changes took effect.

And what about in 2015?  Who is the diplomatic king of parking tickets? 2015′s winner is Senegal by a long shot, whose diplomats currently owe $15,851 for 2015 tickets alone.  Over all, diplomats owe $185,400 for tickets in 2015.

The diplomats from Senegal racked up 230 tickets in 2015 including 14 at hydrants, and decided to pay 30% of them, leaving 161 unpaid, more than double any other country.  License plate 0083RDD topped the list for Senegal, with 36 tickets. Note that Senegal’s 30% payment rate is substantially higher than Rawanda, which paid 2% of tickets, and Gambia, which paid 0%.  

The data shows that Bloomberg’s Parking Plan worked wonders, but that there are still those who feel they can ignore the laws of the City they work in, simply because they have immunity.   I’m glad that the Open Data can put a dent in that by bringing to light when regulations are being thwarted by those who have immunity, making our city less safe as a result.

---

Today, the City Council will hold a hearing on pay raises for certain elected officials, including themselves.  They are also proposing some other bills that are meant to increase transparency and reduce conflicts of interest within the body.

One bill that caught my eye seeks to “require the financial disclosure forms of elected officials to be made available for public inspection on the conflict of interest board’s website.”   At first blush, this sounds great.  But I noticed something was missing...  The bill included no mention of the data being released in a machine readable format.  I see PDFs in our future!

I’m disappointed to see that the council routinely drafts bills without this important caveat.  Transparency without machine readability is not real.  Take, for example, the Citizen Complaint Review Board- an agency designed to provide oversight and transparency over police complaints.  Want to know the monthly number of complaints filed against the police each month for the last four years, or what percent of those ended in charges?  No problem just open 48 PDFs one at a time and scroll until you find the appropriate table, which will have the monthly value you seek.  Don’t have time to do that?  Either does anybody else.  

You see, PDFs are where information goes to die, rather than to be used. Despite this, New York City continues to use PDFs to release so much of its own data and the Council does not seem to include this important caveat in so many of its important information sharing bills.  

With the hearing coming, I set out to find more information on NYC salaries to understand the context of the proposed Council salary increase.  While doing research, it came as no surprise that I could only find the data in our city budget-- in PDF format.   Luckily, there are talented people out there that can extract the data, and in this case Chris Whong and other members of BetaNYC wrote a scraper and released a copy of the Salary portion of the City Budget. 

The budget shows average salaries for each job title in the City, as well as the number of positions needed with that position.  A list of the positions with the highest and lowest salaries can be seen below.

Note that Council Members are not on this list.  In fact, the budget shows 6,404 people with 316 unique job titles that have base salaries more than the Council Members, or 2.3% of all city workers.  If the salary increase is approved, that number would dramatically drop to just 816 city employees with higher salaries, putting Council Members in the top 0.3% of city employees. In short, the Council Members would be jumping ahead of the 87% of city employees who currently get paid more than them.  The jump would be from red bar to green bar in the histogram below.

The list of job titles with average pay higher than the Council Members is varied, and I’ve listed the most common titles below.  Those in red would still be paid more even after the raise while those in black would fall behind. 

While looking at Council pay, it was also interesting to see that Council Member’s staff (outside of the Speaker’s office) are not listed as salary items in the budget.  I found an alternative source for that data and learned that Council Member’s staff are paid significantly less than most NYC employees. The median salary for Council Member staff is around $35,000.  This is less than one fourth of what the Council Members are proposing as their new salary.  It’s also less than 87% of all New York City Employees and its about half the median income of all New York City Employees.

City Council Members play a vital role in our city, as the boots on the ground that interact with and represent all New Yorkers.  Given that, there is reason to believe that a raise may be justified for them (and one day for their staff). But what is not justified is the continued lack of machine readable language in transparency bills.  If we are going to have intelligent debates about the future of our city, we will need the data to back them up.  

Sources/Tools: -Analysis done in Beaker Notebook -Tables made in Excel -Raw Data for City Budget is here -Raw data for Council Staff Salaries scraped from Empire Center Website.

So someone nabbed your bike from your apartment building and you need to file a police report.  You might think that you should just head on down to the closest precinct house and fill out a police report.  But in 28% of the city, you would be wrong.  That’s because your closest precinct house is not necessarily the precinct that patrols your neighborhood.  I’ve created the map below that shows which parts of New York City are not covered by their closest precinct houses in red.  Current precinct boundaries are in black.  The boundaries in white would make each part of the city patrolled by its closest precinct house.   Clicking on any part of the map will tell you which precinct covers that area and which one is closer.  

Analyzing the data shows that Manhattan has the lowest percentage of closest coverage- less than two thirds of the island is covered by the closest precinct house.

I was surprised to learn that there were thirteen precincts where more than half the area they cover is closer to other precinct houses than their own.

For example, more than half of the 42nd precinct in the Bronx is closer to the 48th precinct house than it is to the 42nd precinct house.  (The 48th precinct has carved out a few blocks just to include its precinct house). 

The 108th precinct in Brooklyn, where only 31.4% of the land is closest to its own precinct house, includes an area that has parts closest to six other precinct houses: those of the 83rd, 94th, 104th, 110th, 114th and 115th.

Precinct houses like the 70th have been drawn into a specific precinct with some creative border drawing. As you can see, in this part of Brooklyn, its rare for a precinct house to be in the center of its district.

Of course, there are often good reasons for precinct boundaries to be drawn the way they are: traffic flow, land availability, waterways, roadways, population density, etc.  Still, I was surprised to learn just how much of New York City is patrolled by a precinct house that is not closest.  

Which brings us to the big question... do areas in red have more crime per capita than areas in blue on the map?  My quick analysis showed that there was not a meaningful difference.  I’d imagine that officers are rarely responding from the precinct house itself.  

So then why does this matter?  One hypothesis might be that a centrally located precinct house could help strengthen relationships between the police and the communities they serve.  The community events that precinct houses sometimes hold (haunted house anyone?) might draw a broader neighborhood audience.  Not to mention the fact that a more centrally located precinct is one that a neighborhood resident is more likely to walk by and internalize.  In an era where community policing is increasingly discussed, there always room to make our city just a bit “closer” to our police.

When’s the last time someone asked you where you lived or worked, and you answered by naming a police precinct?  I’m guessing never.  Why?  Because while police play a vital role in our communities, we tend to connect more to our neighborhood boundaries than our precinct numbers.  (How many of you can even name the number of your home precinct?)

And yet, for as long as modern crime statistics have been recorded here in our city, the police have been delivering those statistics to us, aggregated as counts by precinct number.  This type of aggregated data is useful to the NYPD, as it allows them to move resources around according to crime and assess progress, precinct by precinct.  But unfortunately, it’s not so useful for the rest of us. Single precincts can cover a wide range of neighborhoods, so even when you hear there is a change in crime levels in your precinct, you might not know if it affects a specific neighborhood.  You might want to split crime data up in other ways, like by street or council district, and that has not been possible.  Or you might want to know on what days or times of day crimes in a neighborhood occur most often, but with aggregate date, no luck.

But something big quietly happened last week; something that starts to put some of the issues I just outlined to bed.  For the first time in its history, the New York Police Department released incident-level reported crime data. That means for each individual reported felony (for the first three quarters of the year), the citizens of New York now have access to data showing where and when that felony was reported instead of a single number telling us the total number that were reported in a precinct over a period of time. There are countless insights to derive within, and with this post I am going to start to scratch the surface.

(Note that location data was only released for six out of seven of the major felony categories-  rape locations have been left off to protect the privacy of victims. So for the rest of this post, I’ll talk about the Six Major Felonies, which are the “Seven Major Felonies” that NYC reports, minus rape.) 

Before I dive in, it’s worth congratulating the NYPD and the de Blasio administration for making this happen.  It’s been a long time coming and I have no doubt that some innovative techniques and insights will arise from this release.  Those new ideas will eventually find their way back to our government and make our city even safer. But it’s also worth noting that many other cities (e.g. Los Angeles, Chicago, Seattle, Denver, Boston) provide more granular crime data, with longer histories (many years, instead of just three quarters of data), a broader set of reported crimes (not just six or seven major felonies), and with more detail (has an arrest been made, what are the victim demographics), so as exciting as this moment is, it’s good to remember that New York is still very far behind its peers on police data transparency.  

OK, enough context. On to what I learned:

Mapping and Ranking NYC’s Reported Felonies by Neighborhood

The data released is from the first three quarters of 2015, and gave location and time for six major felony categories: Robbery, Murder, Grand Larceny Auto, Grand Larceny of Motor Vehicle, Burglary and Assault.  We have had some view on this data in the past – a few years back the NYPD put out its Crime Map, but did not provide access to the underlying data, a BIG Open Data faux-pas.  One of the provided map views shows aggregate crime data by precinct, but I chose to redo this sort of analysis at the neighborhood level (well at least Neighborhood Tabulation Area level NYC’s census designated neighborhoods), which is much more granular than precincts.

For each neighborhood, I ranked it against the 188 neighborhoods with NTA’s (removing parks, cemeteries, airports and Riker’s Island).  I created a rank for each of the six felony types, where a rank of 1 means that the neighborhood leads for the most of that felony type per resident, and a rank of 188 means the neighborhood has the least of that reported crime per resident.  Lastly, I calculated which neighborhood had the highest average ranks versus the lowest.  This, in essence, gives each neighborhood an overall felony crime rank, giving equal weight to rankings from each type of crime.  (You could argue that weights should be calibrated differently, but we’ll leave that for a future post.)  With that, I labeled each neighborhood with its overall felony rank and made the map below:

Clicking on any neighborhood will show you its rank for each of the six felony types, as well as its overall rank (which is also what it is colored by).  

Interestingly, the safest seven neighborhoods by this measure are all contiguous in Staten Island and are all fairly low density.  It’s also interesting to see that Midtown-Midtown South is ranked 1st overall.  This is likely due to the low number of residents in that area compared to the number of people, since each crime rate here is normalized by the population size of the neighborhood.

The top 10 ranking neighborhoods for all felony types can be found in the table below:

While looking at the data, I also noted a few more interesting patterns...

More Robberies Happen in the three hours between 2-5PM, than 2-5AM

One great advantage of disaggregated data is that we can start to explore the time of day that different crimes happen.  I know many a New Yorker who is nervous about getting mugged coming home from a late night out.   The city certainly feels less safe late at night.  But I was interested to learn that the newly released data shows that robberies happen throughout the day, not mainly in the late evenings.

It turns out there are more robberies between 2-5PM then 2-5AM. This fact seems largely due to a large spike on weekdays that begins right after 2PM, seen in the table below, possibly coinciding with school release times, but obviously this warrants much deeper analysis.  (Continued investments in after school programs anyone?)  Of course there are more people out during the afternoon than early morning, so that does not mean any individual is more likely to be robbed when they are out and about.  But nonetheless, it’s an interesting finding which challenged an assumption I had made about New York City.  That being said, the most problematic single hour of the week (as opposed to hour of day) seems to be between 4AM and 5AM on Sunday mornings (e.g. late Saturday night), not surprisingly right after all NYC bars close.

Murder and Assault are at Least Four Times More Common in Our Least Affluent Neighborhoods as Compared to Our Most Affluent Ones.

I put each neighborhood into one of five buckets, 1 being the lowest 20% median income income quantiles (those neighborhoods with a median income of less than $34,000) and 5 being the highest (those neighborhoods with a median income of $68,950).  I then calculated the felony rates for the residents of each of the five buckets.    

For example, the first number in the burglary row is 1.16 meaning that in the least affluent 20% of neighborhoods, the number of burglaries per resident is 1.16 higher than the city average.  

Five out of six types of major felonies are more common in less affluent neighborhoods than more affluent ones, as seen by the general lowering of numbers from left to right in the table.  The exception is Grand Larceny, which is 1.75 times more likely in the most affluent 20% of neighborhoods.  The 1 to 5 Ratio numbers on the right of the table tell you how much higher the reported felony rate is in the least affluent 20% versus the most. The numbers that stand out are Murder and Felony Assault, which are more than 4 times more likely in less affluent neighborhoods.  Of course it’s no surprise that crime is higher in less affluent neighborhoods, but quantifying it this way shows just how much in a more digestible form.  This was not generally possible to understand before last week’s open data release, as census data is not usually collected by police precinct.

Conclusion

There is so much more to do with this data, but in order to do a meaningful analysis, the public needs access to more than one year of crime data, and other reported crimes besides major felonies. As noted, many other cities are providing just that.  Just today, the de Blasio administration proudly proclaimed that 2015 was "the safest year in the modern history in New York City”, and yet they are holding back the very data that can back up that claim with more specificity.  That is truly unfortunate, but the good news is we are seeing transparency for the first time in our history .  And that is a win for all New Yorkers, including those that work at the NYPD.  Thanks again to all that made this happen (with a special thanks to The Transparency Working Group,  The Mayor’s Office of Data Analytics and DoITT who worked behind the scenes).

-----

 There are few traditions in New York like the Thanksgiving Day Parade.  Since 1924, it has delighted spectators and today it draws 3.5 million people from around the world.

But did you know there is another New York tradition involved in the parade? Lobbying!   New York State has only released lobbying data back to 2009, so we don’t know how long it has truly been part of the tradition, but data from this time period shows $48,000 spent on lobbying about the parade by one group.

Who is doing the lobbying?  Well it seems that George Lence from Nicholas & Lence Communications has been the lobbyist and firm behind this since records started. Each year, the lobbying involves the “location and route” of the parade.   

Which brings me to the next question.  Why are people lobbying over the parade route?  Could it be safety concerns?  Traffic?  Doubtful. The client is always the Times Square Advertising Coalition, an organization that has a lot to gain given all the cameras on the parade route and all the billboards seen by those cameras.

And who exactly are they lobbying?  The lobbying is directed at government offices including the Department of Transportation, the City Council and the Manhattan Borough President.

Lastly, where does Macy’s sit in all of this?  Records show they seem to have more important things to lobby about.  Since 2009, they have spent $65,193 on lobbying over zoning changes at their flagship store with the firm Akerman.

Well, is this all a lot of money?  Here is a fun fact which puts these amounts into perspective.  Columbia University, the local university that has spent the most on lobbying, paid $5.2 million over that same period. 

So is the lobbying as big as those floats? No. But if you see the parade take a funny turn one day, you might just notice an extra billboard or two.

----

As part of a homework assignment, I ask my students at Pratt to go out and chart something of interest to them, and Katherine Savarese came back with a simple chart about farmers markets that I loved- it inspired this post.

You probably know that farmers markets are a staple across all five boroughs of New York City, but September happens to be a very special month. Why is that?  Well, it turns out September is the month with the most unique types of fresh produce- forty three to be exact.

To show the current offerings, we charted how many types of fruits and vegetables are available by month, and showed if they were fresh or from storage. 

If you miss the month of September, you might be behind the fruit curve, but there are plenty more months of vegetables left.  The chart shows that fresh vegetables are available 9 out of 12 months, but fruit is only available 5 out of 12. Now if only we could quantify how delicious the produce is... 

—–

It has been a big summer for transportation open data.  First, the TLC released a dataset with every taxi and green-cab trip taken since 2014 (a release which I had advocated for in my talk on TED).  This is a great step forward and will likely save the TLC much time in filling requests for their data.  I’d love to see other government agencies follow suit.

And another exciting development happened on the Open Data front, and it was not something the government did this time- it was FiveThirtyEight.  You see, the folks there ran a couple of pieces on Uber and Taxis based on some data they FOILed from the government. Usually, this is where newsrooms move on to the next project.  But, instead, FiveThirtyEeight posted the raw FOILed data on github for us all to analyze.  That is big news and I’d love to see more data journalists follow suit for two main reasons:

Releasing your data along with your work amplifies your messaging.  It allows others to follow up on what you do, and add on to it

More importantly, reproducibility is an important part of the scientific method.  If “data science” is truly a science, it should be meeting basic tenets of the scientific method.  Releasing data allows others to verify and follow up on your work. 

What happens when data journalists don’t release their data?  You get silly back and forths like the one between the Wall Street Journal and Bratton over the Times Square Pedestrian Plaza.  We, as citizens, have to sit on the sidelines as neither has backed up their positions with any data releases.  If the WSJ posted the data they used, we could cut through the rhetoric and see what is really going on. 

With all that being said, I made a few quick observations which I’ll put on the blog this week.  In the battle between Uber and Lyft, Uber is obviously dominant with about 7 times more rides being given.  But, it seems that Lyft has made some serious inroads on the late night scene:

Surprisingly, at 2AM the ratio is nearly evenly split between the two.  

In fact, a quick look at the hours that each service provides the most rides shows incredibly different shapes- Uber maxing out at the evening rush and Lyft maxing out at around Midnight.

The drastically different shapes point to a very different clientele - I always assumed that Lyft’s pink mustaches might appeal more to a younger crowd, but the data backs that up.  So if you are wondering what the difference is between driving for Uber and Lyft, its night and day.

This small insight was made possible by FiveThirtyEight’s release of FOILed data, and I can only imagine how many more data insights will flow if other journalists follow their lead. In New York City, FiveThirtyEight, The Upshot and WNYC seem to be setting the data-release trend.  Will your favorite news agency be next?  

-----