DocumentCloud

The DOJ maintains a website where they’ve continued to publish court filings and other documents related to the capital breach on January 6, 2021. The search options are limited there, however. We were curious about things like relationships between defendants and definitely interested in preserving the documents and in seeing new documents as they are added to the repository. We used DocumentCloud’s Scraper Add-On to gather all those documents into a project on DocumentCloud — try it yourself, by following our step-by-step Guide.

To help drive that community’s impact and collaboration, today we’re launching Add-Ons, an easy way for anyone to launch, maintain, and share new capabilities right within DocumentCloud, ranging from exporting notes to applying machine learning techniques.

To get started, all you need to do is log in to DocumentCloud, select some documents, and then pick an Add-On. It will start running in the background, and then notify you of its progress. Add-Ons can also optionally send you an email, generate files for you, or be configured to integrate with a wide range of external tools, such as Slack, cloud-hosted APIs, or a range of open source packages.

(read a whole lot more)

In addition to the Hello World Add-On template that demonstrates basic functionality, we have a few Add-Ons live now that can also serve as a base for your to fork and build on:

Regex Extractor: Let’s you define a Regex string to pull out specified text matches into a spreadsheet across a selection of documents.

PDF Export: Helps you get your PDFs out of DocumentCloud, adding the selected documents into a Zip file that’s then displayed to you.

Note Export: Extracts all the notes on selected documents and saves them as text files you can download.

Bulk Edit: Let’s you update metadata on many documents at once.

SideKick Document Classification: Makes it easy to train a machine learning model to classify documents by an arbitrary type, such as identifying if a document is likely to be an email, a resident complaint, or other categories of records.

Notification Alerts: A simple example of a scheduled automation that lets you adjust a search query and have DocumentCloud alert you if any new results match it.

Currently, running Add-Ons from the web interface requires submitting them through a review process and giving our team a chance to check in, but you can run Add-Ons from the command line or as scheduled GitHub Actions now, and we’ll be adding the ability to directly import and run your own Add-Ons from within the web interface, with no review process, in the coming months.

Over the past year the DocumentCloud team has been working hard to create better solutions for journalists and the public at large to share, analyze, annotate and, ultimately, publish source documents to the web. In the past six months we have seen great growth in DocumentCloud’s capabilities including the ability to de-index documents, some API changes, new translations of the DocumentCloud website, several new Add-Ons and new paid features.

There were more than 4000 families living in FEMA provided housing (typically trailers) in November 2022. FEMA updates this number weekly in their Daily Operations Briefings but the Briefings are only available as PDFs of slide decks shared to email list. The Data Liberation Project is compiling those PDFs into a searchable repository and extracting the housing counts into a public, structured data set. We’ll look at how Singer-Vine is approaching the project and how you’ll be able to use the tools he is building to develop your own data sets from useful numbers trapped in PDFs 

Join us on January 31 for a conversation with The Data Liberation Project

We're starting our 2023 virtual conversations by chatting with Jeremy Singer-Vine, director of The Data Liberation Project and the voice behind Data is Plural, "a weekly newsletter of useful/curious datasets." Singer-Vine will be chatting with us about his journey, his vision for The Data Liberation Project and his experiences developing an RSS Document Fetcher Add-On for DocumentCloud as part of the Gateway Grant effort he is currently undertaking: creating a real-time archive and public dataset of FEMA housing reports.

RSVP to Join Us

MuckRock’s Gateway Grants support projects that preserve critical document collections

The Data Liberation Project was one of four initial recipients of MuckRock’s Gateway Grants Program. With support from the Filecoin Foundation for the Distributed Web, MuckRock provides grants to explore  ways to leverage technology to preserve access to significantly consequential document collections.

Each grantee has been awarded $10,000 and technical assistance for their projects. The tools they build are open source and available for use by all DocumentCloud users in the form of new capabilities via Add-Ons. DocumentCloud users benefit from the growing library of new functionalities. Any user can write and share a DocumentCloud extension with Add-Ons.

OpenNews has opened up scholarships for conference travel, with deadline of January 25:

These scholarships are for journalists who work in data or code. This program is one way we can help people in those roles build skills and connections to make their newsrooms more equitable and just.