NYC OpenData

What/Who is HPD?

The New York City Department of Housing Preservation and Development (HPD) publishes nearly 40 open datasets on the city’s Open Data Portal. These datasets highlight the work we do every day to meet our agency mission: to promote the quality and affordability of the city’s housing and the strength and diversity of its many neighborhoods. HPD accomplishes its objectives through four key services:

•       Preserving affordable housing and protecting tenants

•       Developing new affordable housing

•       Enforcing the Housing Maintenance Code

•       Engaging neighborhoods in planning

While we publish Open Data related to a variety of our programs and services, this blog post will focus on the datasets related to the Mayor’s Housing New York plan.

What is Housing New York?

Chart based on Affordability Levels Table in Housing New York by the Numbers https://www1.nyc.gov/site/hpd/about/housing-new-york-by-the-numbers.page

HPD is the agency responsible for leading Housing New York (HNY), Mayor de Blasio’s plan to create or preserve 300,000 affordable homes by 2026. HNY is the most ambitious housing plan in the country; 300,000 units are enough homes to house the entire population of Seattle or Boston.

Over the course of this 12-year plan, the goal is to preserve 60% and create (or build) 40% of the total 300,000 affordable housing units. Each housing unit we create or preserve targets a specific income group. Per the graphic above, nearly 85% of our housing units serve extremely low, very low, and low-income households, which are defined as a family of 3 making $76,880 or less each year. 

In an effort to better enable the public to track progress towards Housing New York targets, HPD publishes two datasets with information on projects, buildings, and units counted towards the plan. Broadly, these datasets present information on projects that have been counted towards HNY at a building- and a project-level. You can find them by searching for “Housing New York” on the City’s Open Data Portal.

Housing New York Datasets

Housing New York data is divided into project-level and building-level data because some datapoints collected for the plan describe a whole project, and other datapoints describe attributes of buildings and units counted towards the plan.

As shown below, a project is one or more buildings financed and/or regulated as a group entity. A building is a physical structure containing one or more units within a project. And a unit is an apartment, house, or any single residential dwelling unit.

The building-level file displays project information along with its associated building datapoints such as house number, street name, borough-block-lot (BBL), and Building Identification Number (BIN) for each building within a project. Unit counts are provided by building.

The project-level file displays information by project and includes datapoints such as the presence of senior units. Unit counts are provided for each project, rather than by building.

How to use Housing New York Open Data

You might ask: How can I use these Housing New York open datasets to track progress towards the plan’s objectives? Both datasets contain diverse datapoints, but here’s a quick overview of some of the fields included in each file:

You can use the projects file to see what percentage of HNY units serve low-income households:

The buildings file can be used to look at HNY metrics by construction type. See the visualization of HNY units by construction type below.

And that’s not all! We recommend that you play around with the Open Data Portal’s built-in visualization features to explore our data even further. You can access the visualization tool by clicking the “Visualize” button on the top right of the Housing New York dataset pages.

Tips + Tricks for using HPD’s HNY Datasets

Now that we’ve covered the differences between the Housing New York datasets and some basic examples of what you can analyze with this information, we want to leave you with some final tips and tricks for working with this data.

•       HNY by Project dataset presents data at the project-level

•       HNY by Building dataset presents the data at the building-level

•       ‘Project ID’ links the two datasets together

•       HNY buildings are geocoded, except for confidential projects (confidential projects assist specific homeowners or other special populations, e.g.:domestic violence survivors)

•       Data are updated quarterly

This blogpost originally appeared here. 

Since the passage of the Clean Water Act of 1972, we’ve been hard at work building and upgrading wastewater treatment plants to ensure the city’s daily 1.3 billion gallons of wastewater is treated to the highest standards and that the cleanest, most biologically similar water is released back into New York Harbor. We’ve also taken that a step further by constructing miles of new sewers to increase capacity, planting acres of wetlands to serve as a natural filter and wildlife habitat, and installing thousands of green infrastructure assets across the city to curb the release of pollution into our waterways.

So, we know why New York Harbor is cleaner than it has been in more than a century, but, how do we know it’s so clean?

Whales!

Well, yes, whales are definitely an indication. Whales, and a number of aquatic species, are being drawn back to New York in numbers we haven’t seen in decades, thanks to a newly burgeoning food supply that is flourishing in cleaner waters. But we can also prove that our water is cleaner thanks to data gathered by DEP scientists who conduct the New York Harbor Survey, an ambient water quality monitoring program that has been tracking water quality data in New York Harbor since 1909. Begun under the former New York Metropolitan Sewerage Commission in response to public outcry over degraded water quality of the harbor, the survey included 12 monitoring stations around Manhattan using just five sampling parameters. Today, our scientists utilize 89 monitoring stations in both open water and tributaries, and study 27 parameters in all four major areas of the harbor.

Why is this data so important? Well…

This expanded and refined monitoring network has allowed us to demonstrate the steady and significant improvement in harbor water quality over the past 110 years and has played a pivotal role in helping us focus our efforts and resources on targeted areas that still need improvement.  The role of data is essential in fulfilling our mission to not only sustain, but also improve, New York Harbor’s water quality and surrounding ecosystems. We use data collected throughout this program, and beyond, to identify trends, correlations, and causation of water quality conditions. The data is also used by regulators, scientists, educators and citizens to assess impacts and trends and to forecast how environmental factors will affect water quality.

Many of you have kindly asked us in the past "How can I help with NYC Open Data efforts?" and we have an opportunity for you! 

The NYC Open Data Team is looking for 10-15 volunteers to become NYC Open Data Queens Borough Ambassadors from August - November 2019 as a part of a pilot programming initiative called Data Counts in partnership with the Queens Public Library and civic technology nonprofit BetaNYC. Apply by next Tuesday July 16th via this form.

The goals of this pilot as a part of the Data Counts Program are to: (1) pilot a train-the-trainer approach to scaling up efforts at teaching communities about NYC Open Data and (2) to raise awareness about Census 2020. Specifically, Borough Ambassadors will: 

Attend a training led by BetaNYC on 07/22 from 5:30-7:30pm to learn how to lead an Open Data 101 community training in Long Island City (Queens)

Lead a minimum of three 90-minute community trainings in Queens Public Library branches between August - November 2019

To be a successful Borough Ambassador, you will need to be able to attend the training on 07/22 mentioned above, pass BetaNYC's certification criteria for providing Open Data 101 community trainings and lead three 90-minute community trainings in Queens Public Library branches between August - November 2019. No prior experience with Open Data or teaching is required; though seasoned Open Data users and community leaders are encouraged to apply. 

You need not be a resident of Queens to apply - there is no residency requirement to be involved, though we do require you to be able to attend the BetaNYC training in Long Island City on 07/22 and the Queens community trainings you will lead. 

If you're interested, we encourage you to apply by next Tuesday July 16th via this form. If selected, you will be notified no later than Thursday July 18th and expected to join the training on July 22nd. Please email: [email protected] with any questions.

UPDATE as of 8:30 AM EST 07/09/19: The 311 Service Request and 311 Call Inquiry datasets are both being refreshed daily as of Sunday July 7th. The blog post below was updated at 8:30 AM EST 07/09/19 to reflect the most accurate updates to the 311 datasets on NYC Open Data. 

Many of you are familiar with NYC311, a New York City service that provides access to non-emergency City services and information about City government programs. NYC311 is available online and by phone (dial 311). 

The NYC311 system underwent a system upgrade over the June 29-30 weekend, which has resulted in changes to the following three 311 open data datasets: 

311 Service Requests from 2010 to Present

311 Call Center Inquiry

311 Web Content Services

These datasets paused all updates from June 27-July 4 2019, and were refreshed on Friday July 5th with another refresh on Sunday July 7th. These datasets will have a few changes as of the system upgrade which are outlined below. 

We realize these changes could have downstream effects on any applications or analyses connected to these datasets. If you have any questions or concerns, please contact us via the NYC Open Data Help Desk.

We will be exploring potential updates to these data sets in the coming months with the 311 Team. Additionally, we are organizing an event with a local civic technology nonprofit, BetaNYC, and the 311 Team, on the morning of August 14th to discuss these changes and hear your feedback. Sign up here to learn more about the event.

Summary of changes to 311 open data datasets: 

311 Service Requests from 2010 to Present 

The column DUE_DATE will contain null values beyond June 27, 2019 and the column ADDRESS_TYPE will contain a significant number of null values beyond June 27, 2019.

311 Call Center Inquiry

The columns AGENCY, AGENCY_NAME, and CALL_RESOLUTION will contain null values beyond June 27, 2019. 

311 Web Content Services

Engaging all five boroughs, thousands of New Yorkers

The NYC Open Data team, in collaboration with local civic tech nonprofit BetaNYC, held the third annual NYC Open Data Week from March 1-9, 2019. This was our largest NYC Open Data Week yet: more than 65 New Yorkers and organizations helped plan 48 events across all five boroughs, bringing out over 3,000 New Yorkers to learn about NYC Open Data and how they can get involved. Online, web traffic to nyc.gov/opendata surged 20% over an average week. According to a post-event survey, participants ranged from ages 18 to 65+ and came from every corner of New York City, as well as Long Island, upstate New York, New Jersey, Pittsburgh, and even London.

For the first time, NYC Open Data Week included events in every borough, allowing New Yorkers from all walks of life to see open data in action and inspiring them to create civic change in their own neighborhoods. Event highlights include School of Data, an annual day-long community conference produced by BetaNYC and attended by over 400 people; an event featuring the Speculation Watch List, developed by the NYC Department of Housing Preservation and Development to predict where tenants may be at risk of losing their apartments; a workshop teaching volunteers how they can use open data to support their local parks produced by NYC Parks; a data exhibit featuring sixteen artists curated by Data Through Design at the Brooklyn Navy Yard’s New Lab; and a webinar discussion between the City of Buenos Aires and City of New York about each other’s efforts to encourage residents to use open data. To ensure that Open Data Week represented diverse voices, the Mayor’s Office for People with Disabilities produced an event demonstrating how to make open data accessible to people with visual and hearing impairments, and NYC Service presented on how young people can use NYC Open Data to shape the city they want to live in.

While Open Data Week 2019 is now behind us, the NYC Open Data program runs year round. Here are some upcoming initiatives we are excited for this summer:

Our first Open Data Youth Leadership Council will share their ideas on how youth can make an impact using Open Data after working with our team for the last six months.

Our first Open Data Advisory Council will convene in June -- we look forward to the ideas and input from this new strategic advising body made up of New York City community and civic technology leaders from academia, non-profits, civic technology companies, and City agencies.

We have already begun the planning process for Open Data Week 2020! This work will continue through the summer months.

The Open Data Program will be undergoing a strategic planning process; we will share details on the process and results in our annual Open Data Compliance Report which will be published in September 2019.

Check the NYC Open Data Calendar for updates on events related to open data and civic analytics across the City. And don’t hesitate to get in touch  on “Contact Us” section on the NYC Open Data website. Lastly - check out the 2500+ free data set offerings available on www.nyc.gov/opendata!

NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. In an effort to provide even more transparency to New Yorkers, the NYC Administration for Children’s Services (ACS) website now contains a hyperlink to the Open Data site so that one can access our most recently published statistics and reports with just the click of a mouse.

 Data is an extremely important and powerful tool; at ACS, data allows us to make continuous improvements so that we deliver the highest quality services to our city’s children and families. From finding innovative ways to build outreach to the communities and families we serve, to examining and working towards ways to strengthen current processes, using data is critical. Data can even reveal stories and help us gain insight into what’s happening.

 We are proud that the Open Data site provides critical information about ACS and the children and families we serve throughout the City. For instance, through Open Data, New Yorkers can easily find the number of NYC children in each community district and borough served in prevention services. New Yorkers can also access the list of prevention services providers ACS works with throughout the City. Nearly 45,000 children per year receive prevention services and supports targeted to their family’s individual needs, such as substance abuse, mental health, and domestic violence, among others.

 It is our hope that by making this data more accessible, New Yorkers will better understand the work we do.

Local Law 8 of 2016 requires the Mayor’s Office of Data Analytics to examine three agencies annually to verify their compliance with the NYC Open Data Law. The law also requires MODA to “accept suggestions from the public as to possible public data sets within mayoral agencies that have not yet been disclosed.”

The Open Data Examination and Verification Survey is based on Local Law 8 of 2016 and is intended to help City agencies uncover datasets that may not already be on the Open Data Portal (ODP) or scheduled for future release in their Open Data Plans. In 2018, the Department of Small Business Services (SBS), Department of Transportation (DOT), and Business Integrity Commission (BIC) were required to complete this survey, which includes:

●     Agency Overview: High-level description of the agency, the types of data it collects, and technical systems

●     Current​ ​Inventory:​ Inventory and data quality questionnaire for datasets already on the Open Data Portal

●     MMR:​ Inventory of Mayor’s Management Report (MMR) indicators and underlying datasets

●     Reports:​ Inventory of legally mandated reports and underlying datasets

●     Shared:​ ​Inventory of datasets shared between agencies for operational purposes

●     FOIL:​ Inventory of datasets used to respond to freedom of information law (FOIL) requests

●     Websites: Inventory of datasets regularly updated on agency websites

●     Future Releases: Updates to agencies’ Open Data Plans for future dataset releases

 Agencies were required to report whether each dataset identified in these inventories is a “public dataset” according to the definition in the Open Data Law, and if not, a reason why. Agencies submitted these workbooks in July 2018 as part of the 2018 Open Data Plan.

 Request for Public Comment

The surveys for SBS, DOT, and BIC are available for review here: https://opendata.cityofnewyork.us/wp-content/uploads/2018/10/2018-EV-Draft-Surveys-for-Comment-1.zip 

The public is invited to submit feedback on agencies’ workbooks until 11:59pm on Wednesday October 17, 2018 in this webform. You may use this form to:

Request a dataset you believe to be maintained by the agency but not     already in the Open Data Plan

Identify what you believe to be an omission from any section of the     agency’s dataset inventory

Disagree with the agency’s reported rationale for why any dataset is or is     not public

Ask a question about something reported in the survey, including     questions on existing datasets on the Open Data Portal or datasets     scheduled for future release

In case you missed it, the winners of the Open Data citywide competition were announced! Talented New Yorkers from all over the city submitted projects based on one or more of the City’s 2,000+ Open Data sets.

This exciting Competition gives New Yorkers the opportunity to be featured on the new Open Data Project Gallery receive a Certificate of Recognition from the Mayor. First place winners will also be invited to present at an upcoming NYC Open Data Event.

The NYC Open Data team today announced four first place winners of the NYC Open Data Project Gallery Contest:

Data Science Award

For the most compelling data analysis using at least one dataset from NYC Open Data

WINNER: Plan(t)wise: planning a green canopy over the Big Apple

Submitted by Niki Athanasiadou.

This project shows which trees can grow the best in NYC based on growth history and survival.

Most Creative Award

For the most compelling visual analysis using at least one dataset from NYC Open Data

WINNER: An Interactive Visualization of Street Trees

Submitted by Allen Yee of cloudred.

This creative project shows the variety and quantity of street trees in all five boroughs. Ever wonder what the most/least common tree in NYC is? Well this project is the place to look!

Mayor’s Civics Award

For the project that best facilitates engaging in civil society and uses at least one dataset from NYC Open Data

WINNER: myPB.community

Submitted by Bitsy Bentley & Hadassah Damien of Participatory Budgeting Project’s Participation Lab: 

Open Data Award

For the project that had the highest overall rating and uses at least two datasets from NYC Open Data

WINNER: myPB.community

Submitted by Bitsy Bentley & Hadassah Damien of Participatory Budgeting Project’s Participation Lab: 

The Open Data award goes to myPB.community. The project uses a democratic process to help community members decide how to spend public money. The PBNYC completed its 7th cycle of voting in about 60% of the Districts across the city, spending about $30 million tax dollars in the process!

Second and third place winners in various award categories included:

BoardStat by BetaNYC in partnership with the Manhattan Borough President's Office, Manhattan Community Boards, Microsoft Cities & CUNY Service Corps.

311 Data & Life in NYC by Oleh Dubno & Danny Argov

RentLogic by Yale Fox

NYC Data Explorer by William P Karavites

Open Sewer Atlas NYC by Korin Tangtrakul of Open Sewer Atlas of NYC

After the Pride Parade by Wenfei Xu of CARTO

Five Communities of Williamsburg by Wenfei Xu of CARTO

NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. The platform hosts more than 2,000 data assets that are regularly used by entrepreneurs, students, nonprofits, journalists, city staff and others. Each week, more than 20,000 users visit the platform at nyc.gov/opendata. The NYC Open Data program started in 2010 and was strengthened by the NYC Open Data Law (Local Law 11 of 2012), and a series of additional amendments which mandate that City agencies publish all public data on www.nyc.gov/opendata. The Mayor’s Office of Data Analytics (MODA) and the Department of Information Technology and Telecommunications (DoITT) partner to form the Open Data team.

We just made it past the first day of Spring (March 21st!) just to land in another snowstorm…?!?

Speaking of snowstorms, we wanted to make sure you were all aware of events rescheduled from our Open Data Week’s snowstorm on March 7th. We hope you’ll be able to join us!

Wednesday March 28th

The Missing Product Manager: Panel discussion of different digital City products

12:30-2pm

Free event

Hosts: Department of City Planning & Civic Hall

Register here

 Open Data Showcase: Open Data Workshop using python (for beginners!)

4:30-6pm

Free event

Hosts: Hunter College

Register here

 Wednesday April 25th

Be in the Know: NYC Open Data for Youth: Open Data Workshop (for beginners!)

5-7pm

Free event

Hosts: NYC Service | Mayor’s Office

Register here

 Thursday April 5th

Talk Data To Me: Data Science Project Showcase

6:30-8:30pm

Free event

Hosts: General Assembly

Register here

 Wednesday April 25th

Using and Improving NYC Open Data Dictionaries: Workshop for librarians engaging with the NYC Open Data Team

1-5pm

Free event

Hosts: METRO, Tiny Panther

March 3 through March 10, more than 30 events for new yorkers to explore the use & power of NYC Open Data

NYC Open Data Week 2018 is a week-long celebration to raise awareness of the City’s public data. Through March 10, New Yorkers are invited to experience more than 30 events, exhibits, panels, and workshops across the city that explore how NYC Open Data is being leveraged by New Yorkers. Events include a data art exhibition, a demo of a new platform to identify risk to affordable housing in Brooklyn, a tour of a data exhibit at the Museum of the City of New York, workshops for student entrepreneurs on how to use open data to build their business, and more. The full schedule is available here. The City’s Open Data Portal, visited 75,000 times each month on average, allows New Yorkers to access nearly 2,000 free municipal datasets, ranging from 311 complaints to crime incidents by neighborhood to the location of every street tree in the city.

“A fair city is an open city. NYC Open Data puts the data we use to make decisions in government back in the hands of all New Yorkers. Every day, New Yorkers in all five boroughs use open data to improve their communities,” said Mayor Bill de Blasio. “Open Data Week is about highlighting those stories and giving all New Yorkers inspiration to make a difference.”

“The NYC Open Data Portal is a powerful tool that ensures transparency and fosters civic innovation within our City to help improve the quality of life for all New Yorkers,” said Samir Saini, Commissioner of the Department of Information Technology and Telecommunications.

“While New York City has impressive open data stats to boast: nearly 2,000 published data assets and 20,000 visitors to the site per week on average - much of its value happens behind the scenes in making our government more data-driven,” said Emily W. Newman, Acting Director of the Mayor's Office of Operations. “Congratulations to the Mayor’s Office of Data Analytics and DoITT--they make it possible for NYC Open Data to engage dozens of City agencies and City-affiliated organizations. Open Data is transforming the way our City thinks about data-sharing and the power of analytics to drive change for New Yorkers.”

“Since committing to Open Data for All in 2015, we have dismissed the idea ‘If you build it they will come’ and taken efforts to engage more New Yorkers than ever in the data created by their City,” said Adrienne Schmoeker, Director of Civic Engagement & Strategy at the Mayor’s Office of Data Analytics. “We are thrilled to celebrate Open Data Week with all our partners who make transparent government possible.”

Highlights of Open Data Week include:

Brooklyn (March 2-10): Data Through Design: Art Exhibit at the Made in NY Media Center is a free exhibit in DUMBO, Brooklyn featuring eight artists selected in February 2018 exploring the insights and stories that emerge from public data.

Staten Island (Saturday, March 10): Our Stories: The Soul of Data is organized by First Star College of Staten Island (CSI) Academy with the Administration of Children’s Services; the free four-hour workshop will empower 20+ foster youth with the tools to find and use NYC Open Data.

Manhattan (Saturday March 10): Open Data L-Train Innovation Challenge is a day-long event organized by Forum for the Future, Collectively and Grand Central Tech calling designers and civic technologists for a day of problem-solving.

Brooklyn (Thursday, March 8): Identify Affordable Housing Risks with Data at Brooklyn Borough Hall will unveil a new web portal to host housing data from multiple sources, allowing organizations to share, validate, and bolster their findings and research to show trends and threats to affordable housing in Brooklyn. Hosted by Brooklyn Borough President Eric Adams and OpenGov.

In celebration of International Open Data Day on March 3, the City is launching the Open Data Project Gallery which shows 5 examples of how data has been used to address urban problems. This new feature was inspired by Open Data Week 2017, designed and prototyped by the NYC firm Fahrenheit 212 and developed by the Department of Information Technology and Telecommunications Gov Lab & Studio. The City is also offering a chance for New Yorkers and international users to enter their own projects into the running for the following awards, which will be judged by a panel of experts from the City, WNYC, SAVI at the Pratt Institute, NYC Tech Alliance, and Technical.ly Brooklyn:

Mayor's Civics Award

Data Science Award

Open Data Award

Most Creative Award

NYC Innovator Award

The contest officially launches on March 3 and runs through May 1, see the Open Data Website for more details: nyc.gov/opendata.

“Data was born to be free, and NYC's Open Data program ensures it lives that way” said Miguel Gamiño, Jr., New York City Chief Technology Officer. “Democratization of data makes it possible for any entrepreneur and startup to access one of the most valuable ingredients for building technologies that serve the public and make technology work for all people.”

“TLC is proud to celebrate Open Data Week and the sixth anniversary of the City’s Open Data Law!” said TLC Commissioner Meera Joshi. “Since the law passed in 2012, there has been massive growth in the for-hire industries in NYC, and TLC has expanded data reporting requirements to understand the effects of this growth and to better regulate services, making much of this granular trip data available to the public. TLC goes beyond the law’s requirements and has identified new metrics that are most useful for the public in understanding how taxis, app-based services, and traditional for-hire services operate in NYC, allowing the public to see firsthand the growth and its effects, no matter a person’s technical skills with data.”

“With NYC Open Data Week, the City has created a unique opportunity for New Yorkers to step into the world of data,” said Gregg Bishop, Commissioner of the New York City Department of Small Business Services. “Not only is the City opening up data for all but, given the upward mobility that data careers offer, we’re providing free skills training and job connection to open up opportunity for New Yorkers in an increasingly tech-driven economy.”

“NYC Service is thrilled to engage our Youth Leadership Councils in Open Data Week,” said NYC Chief Service Officer Paula Gavin. “Community data is a foundation for civic engagement and the Open Data platform is an important tool that encourages our City's youth to better understand their neighborhoods in new ways, allowing them access to information that can be used to improve policy and practice in all five boroughs.”

“Our Open Data Law has given rise to a dazzling constellation of new apps, research projects, and even businesses aimed at improving New Yorkers’ lives through the creative use of public datasets,” said Manhattan Borough President Gale A. Brewer. “Happy Open Data Week to all the civic hackers out there building things with public data, and thank you to the Mayor’s Office of Data Analytics, DoITT, DoRIS, and the public servants at every city agency working to put government data to work for the people we serve.”

“I believe nearly every sector of our municipal government would be enhanced by a better utilization and activation of civic data, through real-time monitoring, enhanced public accountability, and dynamic agency response. My administration will continue to champion technological advances like these that improve the City’s ability to dig into the crevices of persistent challenges facing New Yorkers. I’m pleased that Brooklyn Borough Hall is part of Open Data Week, and I thank the Mayor’s Office of Data Analytics for organizing this important initiative,” said Brooklyn Borough President Eric L. Adams.

Council Member Peter Koo, Chair of the Committee on Technology said, “Open Data Week demonstrates NYC’s commitment to a transparent, accountable and less-cynical government that takes pride in giving people access to information. Today, accessible data is being used by everyone from students to professionals in an effort to influence policy and change in their communities, so it is with great excitement that we ask for this year’s participants to step up their game, and take full advantage of this tremendous opportunity to create new innovations with NYC’s data, and to create a better city for everyone.”

“New York City's Open Data policy puts us at the forefront of giving residents the information they need to make a difference,” said Council Member Ben Kallos. “Since its implementation, the Open Data Portal has bred countless new laws, apps, and other solutions to improve access to city services for all New Yorkers. Thank you to Mayor de Blasio for leading the way on Open Data for All and using Open Data Week to highlight new innovative ways of presenting public information to empower all New Yorkers.”

The City of New York passed the Open Data Law in 2012. Since then, every City agency has contributed datasets to the portal, with more being added every year. In 2015, Mayor de Blasio introduced Open Data for All, a vision to maximize New Yorkers’ engagement with City data. The Portal was subsequently relaunched in 2017 with a more user-friendly design allowing novices and experienced data researchers alike to find and use the valuable information it offers. Since relaunch, the Open Data Portal has seen record use, recording more than 75,000 average visitors per month. The Mayor’s Office of Data Analytics (MODA) and the City’s Department of Information Technology and Telecommunications (DoITT) partner to form the Open Data team.

Open Data Week was inspired, and is supported by, BetaNYC, a civic tech organization based in New York City. They are the organizers of two Open Data Week events, School of Data and Unlocking Open Data for Community Boards with the Manhattan Borough President’s Office.

“After 6 years and 7 additional laws, New York City has nearly 2,000 public data assets from more than 50 different municipal publishers - truly the world's greatest open data program. All worthwhile work takes time, energy and determination - while the road hasn't been easy it has been worth it! It is a pleasure to be a part of the NYC Open Data community and an honor to kick off the second NYC Open Data Week,” said Noel Hidalgo, Executive Director of BetaNYC. “The BetaNYC community is looking forward to an amazing week of storytelling and workshops. We are fortunate to have an Administration, Council, community groups, and individuals who are will to dive beyond the data points and collaborate to make this City work for all of us. The BetaNYC community looks forward to a fantastic Open Data Week. We hope to see all of you at NYC's community data conference.”

“Once again New York is leading the tech industry by demonstrating that open data serves the public interest and spurs innovation in both government and private endeavors,” said Andrew Rasiej, Chairman of the NY Tech Alliance and CEO of Civic Hall. “Now Six Years after New York City’s landmark open data law was passed Open Data Week confirms that both government and citizens are using information that makes government more effective and accountable to the citizens it serves.”

“Allowing New York's innovators access to the wealth of data collected by NYC agencies creates opportunities for startups, large companies, and technologists everywhere. It gives our entrepreneurs the chance to not only solve public-facing problems, but also the tools to build businesses that matter,” said Julie Samuels, Executive Director, Tech:NYC.

“Six years ago, New York City put a plan in motion to leverage its data as a strategic asset. We at Socrata have been fortunate to be part of that journey, helping to make that data not just open but also accessible and usable to all. New York City has created an ecosystem that fosters startups and economic activity around the city, makes the city government more accountable, supports a vibrant civic community, and improves the daily digital experience of New Yorkers. The success of their program is a blueprint that so many other cities are emulating,” said Kevin Merritt, CEO, Socrata.

“The Museum of the City of New York is thrilled to celebrate Open Data Week by offering curator led tours of our data driven interactive gallery exploring the future of New York City. The Future City Lab, part of our signature New York at Its Core exhibition and home to the world’s largest visual display of data about the city, is made possible only by New York City’s commitment to open data and embodies the idea that meeting the challenges of the future requires access to information in the present. We are thrilled to open our doors to all those who are as excited as we are about Open Data Week.” said Whitney Donhauser, Ronay Menschel Director of the Museum of the City of New York. 

“New York City has set a gold standard for public access to data through initiatives like Open Data for All, the Open Data Portal, and Open Data Week. At Rentlogic, we support the administration's commitment to making data easy to understand, using it to improve the quality of life for all New Yorkers, and leveraging it to bring fairness and equity to our City's housing market,” said Rentlogic CEO, Yale Fox. “Choosing where to live is one of the biggest decisions a person makes, and we're proud that Rentlogic has brought transparency to a difficult process in a simple, user-friendly platform for the public good.”

“At Forum for the Future, we design open and collaborative strategies for a more sustainable world, and being part of New York City Open Data Week amplifies our ability to do this. Together with the Department of Transportation, Dell, Grand Central Tech, and other partners, we are organizing a hackathon that is bringing the public and private sectors together to collaborate around the L-train shutdown sustainability implications for the city. We are thrilled to be working with the Mayor’s Office of Data Analytics and other organizations that are committed to using and sharing open data to create positive impact. We are grateful for all of the work and support of the NYC Open Data Team to make our innovation challenge and NYC Open Data Week possible,” said Rodrigo Bautista, Principal Change Designer at Forum for the Future.

“Fahrenheit 212 is proud to have partnered with the City of New York and the Mayor’s Office of Data Analytics on NYC Open Data. As part of our commitment to using innovation for civic and social benefit, we consider NYC Open Data an invaluable resource and fuel for innovation to improve the lives of our neighbors, communities and local businesses. We share in the Mayor’s Office of Data Analytics’ goal to accelerate New York City’s innovation engine. Partnering with NYC Open Data has been a great way for us to contribute to that mission,” said Todd Rovak, CEO of Fahrenheit 212.

“Open data and open government are not a point-in-time event, but an ongoing process.  In 2012 Local law 11 marked a huge milestone for open data in New York and served as a model for cities across the globe. Six years later, at a time when public trust in our democracy is as shaken as it’s ever been, today’s events offer a powerful renewal of New York’s ongoing commitment to transparent and accountable government. From efforts to ensure more robust oversight and compliance to initiatives to make sure data is not just available to some but useful to all, New York has demonstrated a willingness to work toward that future. There is more work to be done to show not just New Yorkers, but the entire globe a positive, more open vision of the future of American democracy. Let’s all hope the City of New York remains up to the task, and can serve as a model once more in the coming year,” said Stephen Larrick, Director of Open Cities, Sunlight Foundation.

“Data isn’t just data. It’s people’s lives, businesses, and homes; it’s the stories of who we are and how we live. Reboot is honored to partner with New York City, and committed to sharing the lessons from working with this pioneering administration with other cities in the US and around the globe. As we seek to research and design new ways to bring data to life for New Yorkers, we are thrilled to see NYC taking the bold next step of moving beyond opening data to solving problems and changing lives with data,” said Zack Brisson, Principal at Reboot.

“General Assembly is a proud supporter of New York City’s Open Data Week. Our instructors leverage NYC Open Data datasets in our data analytics and data science programs, and they have been an invaluable resource for our students who are hungry for real-life data they can use in their projects and portfolios," said Tom Ogletree, Director of Social Impact and External Affairs at General Assembly. "This is our second year participating in Open Data Week, and we are proud of the many initiatives General Assembly is working on with the City to promote access to technology training and tools for all New Yorkers.”

“We are excited to participate in the OpenData Week and showcase our use of OpenData across the Hunter College curriculum,” said Katherine St. John, professor of computer science at Hunter College.  “We look forward to having attendees at our event explore and visualize NYC OpenData and all that it offers.”

“The Zahn Innovation Center at City College is thrilled to be part of Open Data Week. Our March 6th Summit in Harlem will feature both hands-on instruction on how to use NYC Open Data to tackle local problems, and will celebrate the diversity of tech founders here in NYC. We support the many ways in which the Mayor's Office is making technology innovation accessible through initiatives like Open Data!" said Lindsay Siegel, Executive Director of the Zahn Innovation Center at The City College of New York, City University of New York (CUNY).

“Open data represents opportunity. We are thankful for the ongoing dedication and effort from municipalities like New York for expanding the footprint of available open data. There is much more work to be done- and thanks to the efforts of visionary leaders who see beyond what is in place today to continue to expand the scope, breadth and ease of access to open data to serve all constituents. For companies like Vizalytics, who serve public and private sector clients in the US, Canada, Europe and Australia, open data is critical to our business model. When we first began our work in 2012, it was NYC Open Data that was foundational to our success, and we are grateful for being past NYC Big Apps winners, as well as working with NYC Mayor's Office of Tech and Innovation on neighborhoods.nyc,” said Aileen Gemma Smith, CEO Vizalytics Technology.

“Strategy and innovation consultancy Luminary Labs is hosting a panel discussion, “Correlate & Innovate: using non-traditional data sets for innovation,” at its office during Open Data Week. “From healthcare and finance to transportation and government, organizations are increasingly embracing open data in the pursuit of innovation,” said CEO Sara Holoubek of Luminary Labs. “Open Data Week creates a space and time for these important cross-sector conversations.”

“We applaud the City of New York’s continued efforts to increase not only access to data but also to educate users about data through enhancements such as the NYC Open Data Portal metadata screens and data dictionaries. This, along with endeavors like the NYC Planning Lab’s human-centered design approach, significantly increases the understanding and impact we and our community partners are able to achieve,” said Jessie Braden, Director of Pratt Institute’s Spatial Analysis and Visualization Initiative.

 “New York City’s commitment to open data has promoted and fostered operational efficiency within and across agencies, but also more informed and engaged citizens,” said Jim Barry, Esri developer network program manager. “Much of this city’s open data has some location component to it, and on Monday, March 5, during NYC Open Data Week, Esri will be hosting a hands-on workshop, showing how to use "Insights for ArcGIS" to combine, analyze, and visualize NYC open data on maps and charts as actionable results.”  

“We're proud of our partnership with the NYC Mayor’s Office of Data Analytics and New York City nonprofits where together we enable New Yorkers to put Open Data to good use in their communities. CARTO's community and grants programs empower open data users to analyze, visualize, and act on civic data in new ways,” said Tyler Bird, Community Lead at CARTO.  

“We’re excited to have the Open Contracting Partnership engage as a part of Open Data Week and to begin exploring public contracting in New York City. We look forward to the opportunity to dig into NYC's open data infrastructure and explore ways to implement an open data contracting standard to give everyone a way to track spending from planning through to public contracts with companies and onto the delivery of services. Continuing to drive the values of NYC Open Data through to the contracting process will help businesses to access new opportunities and create a level playing field. It will help concerned New Yorkers follow public money and strengthen NYC’s open government efforts,” said Open Contracting Partnership's Executive Director, Gavin Hayman.

Most Open Data Week events are free of charge and open to the public. Visit

The NYC Open Data portal recently added a new “asset type” called Data Collection. Most City data is stored in relational databases and up until now we did not have a good way of surfacing these to the portal. A relational database is a set of tables related to each other through primary and foreign keys. A primary key uniquely identifies every record in that table. A foreign key refers back to the primary key in another table. In this way the tables are linked and make sense when displayed as a group or Collection rather than individually.

Housing Development Projects Receiving City Funding (Local Law 44) is the first Data Collection on the NYC Open Data portal.

The Department of Housing Preservation and Development (HPD) is required by law to publish data relating to housing projects receiving funding from the city. HPD’s Performance Management and Analytics team initially published the required information by putting the tables into a downloadable zipped folder on the NYC Open Data portal, but wanted a more accessible and user-friendly format to display this data. From that, the idea of a Data Collection was initiated.

 This Collection contains 11 data tables, all prefixed with Local Law 44. The main table is the Project table, with ProjectID as its primary key. Each project can have one or more buildings, these are captured in the Building data table. The ProjectID field in the Building table refers back to the ProjectID from the Project table. Likewise, each of the other tables also have a ProjectID field that refers back to the Project table (see image below). In addition to the way the tables are displayed online, the Data Dictionary is combined into a single document, making it easier to see how the tables are connected.

For instance, if you were interested in buildings receiving financial assistance that were completed in the past year, you would find the ProjectIDs in the Project table that fall within the last year according to the ActualCompletionDate field and then go to the Buildings table to pick out the buildings with those ProjectIDs. The tables can be combined in various ways depending on the question you have or analysis you’re looking to do.

 Other datasets on the Open Data portal that could benefit from being surfaced as a Data Collection include ACRIS and OATH Hearings Division Case Status (formerly ECB violations).

 ACRIS is a database of property records and documents. It currently lives on the Open Data portal in 15 separate datasets. This is an example of a relational database on NYC Open Data that could benefit from more robust documentation and presentation on how these datasets relate to each other. Someone may by chance find one of the datasets and not realize that the information they are looking for exists within a different ACRIS dataset.

OATH Hearing Division Case Status dataset contains information on the outcome of hearings relating to ECB violations.  It is currently presented on NYC Open Data as one merged dataset originating from two data tables in the source system: notice of violations and charges. Each violation can have one or more charges. It is joined on the portal as a single dataset, where each row is a violation and the charges are tacked on as fields. While this format is useful for some purposes, it is a challenge to deconstruct it into a list of charges.

Since the passage of Local Law 11 of 2012, the Open Data team has issued an annual report each July to inform New Yorkers about the progress we’re making – both in complying with the law and in engaging New Yorkers to use Open Data in new, creative ways.

On July 14, we released our Open Data for All 2017 Progress Report and Compliance Plan. For the first time, we were able to publish it as an interactive website. In the report, we highlight our accomplishments over the past year, feature vignettes about everyday users, and update our compliance plan. We want to hear from our users. To provide comment on the report itself, or to tell us your own Open Data story, please comment directly through the interactive website here and scroll to the bottom of the page. We’re accepting feedback through the website through September 1. Users may contact us at any time here.

It’s been five years since New York City signed the strongest open data law in the country. We’ve been busy ever since, and celebrated a LOT in honor of our 5-year anniversary...

We launched a pretty new beta website: www.nyc.gov/opendata (send us any feedback here)

 There’s a new way for you to contact us if you have questions, comments, concerns: www.nyc.gov/opendata/engage/ ; some folks have already noticed the difference…!

Our partners at the Department of City Planning launched their new Facilitates Explorer tool – check it out: https://capitalplanning.nyc.gov

900+ people participated in NYC’s First NYC Open Data Week in 12 events across three boroughs

We taught 15 New Yorkers were taught how to use NYC Open Data via our first NYC Open Data for All: TreesCount! Workshop; the first 1-day data-literacy program developed by the City of New York with NYC Open Data as a core teaching tool

Our NYC Chief Analytics Officer, Dr. Amen Ra Mashariki, announced at the Socrata Connect Conference that he will no longer give speeches on NYC Open Data. Instead, our team will find New Yorkers who have been impacted by open data to speak to its promise.

A few folks said nice things about our work (thanks!) and the future of NYC Open Data: State Scoop, AM New York, GovTech, Harvard Ash Center, Progrss, Politico, Reinvent Albany, Sunlight Foundation, Civicist, Big Apps NYC 2017, Chris Chavez via Medium, Carto

All of these wonderful things would not have been possible without the great support, effort and cheerleading from: BetaNYC, BureauBlank, Reinvent Albany, DoITT NYC Gov Lab & Studio, NYC Parks Analytics, NYC Parks Computer Resource Centers, Capital Planning Team at the Department of City Planning, Office of Digital Strategy, The Department of Small Business Services, Carto, Microsoft, Socrata, Screendoor, Vizalytics, General Assembly, Grand Central Tech, Civic Hall, Civic Hall Labs, The College of Staten Island’s Tech Incubator, The Manhattan Borough President’s Office, the National Geo-Spatial Intelligence Agency (NGA), 18F, DataKind, NYC Big Apps, The Made in New York Media Center, Fabernovel Data & Media, Reaktor, Prime Produce, Startup M/IG, GovLab, Luminary Labs and many others!

A final thank you to all 8.5 million New Yorkers - you inspire us every day to do this work and it wouldn’t be where it is without you all and our ongoing dialogue.

Stay tuned for more great things to come from NYC Open Data! Also – if you have a suggestion for something to be featured on our blog, get in touch with us here.

You may have noticed that NYC Open Data has a new look! We’re excited about our new website and hope you’ll explore and let us know what you think.

The site includes a new “Contact Us” page, which will serve as a one-stop shop for any feedback you have for the NYC Open Data Team. The page includes a Screendoor-enabled tool that will allow us to respond to your questions, data requests, and any other inquiries in a more timely manner.

In order to streamline the collection of inquiries, we have worked closely with Socrata to make the following changes:

Dataset nominations: We disabled the old nominations page and are routing all new dataset requests to the Contact Us page. You can still track the status of previously made requests here. We have removed the response to these dataset nominations from this dataset as the responses were always customer-service related in nature and did not provide insight into the data itself.

Comments: We disabled the ability to post new comments on specific datasets in the Socrata platform. We realized that users were posting questions here, which made it difficult for us to quickly respond to. As a result, we are funneling inquiries to the Contact Us page while we research commenting alternatives to implement in the future. No current comments will be removed.

Joey Cherdarchuk of Dark Horse Analytics first posted this creative use of Open Data in 2014. Several datasets, including this building footprints shapefile, the Department of City Planning’s Bytes of the Big Apple, and census data (here and here), were used to create this mesmerizing graphic representing the ebb and flow of working New Yorkers in Manhattan.