#storytracker

Media - Data :

Today marks the official release of version 1.0 of StoryTracker, a set of open source tools for archiving and analyzing news homepages.

The project started in June thanks to funding from the Reynolds Journalism Institute, a research and development center based at the University of Missouri.

Today I'm presenting the result of our work at "Dodging the Memory Hole," a conference for archivists and professionals hosted at the Institute's headquarters in Columbia, Missouri.

Because the entire codebase is free and open source, you can run wild with the code already published on GitHub and distributed via the Python Package Index.

It offers a menu of options, documented here, for creating an orderly archive of HTML snapshots, extracting hyperlinks with a bonus set of metadata that captures each link's prominence on the page and visualizing a page's layout with animations that show changes over time.

Our work is far from complete, but you can see what is possible today in the slide deck above.

It's already been integrated into PastPages to archive the HTML from a select set of sites, and I would like to soon expand to offering automated analysis of those homepages.

PastPages.org has expanded to archive HTML from a pilot group of news homepages thanks to a recently launched open-source software project sponsored by the Reynolds Journalism Institute.

In addition to more than 1.5 million images already archived from nearly 100 news sites around the world, PastPages users can now download and analyze the raw source code, harvested each hour, from five upgraded sites. They are CNN, The Drudge Report, Google News, the Los Angeles Times and the New York Times.

The new HTML archival system is powered by StoryTracker, a project started in June thanks to funding from the Reynolds Journalism Institute, a research and development center based at the University of Missouri.

The lead developer is me, Ben Welsh, founder of PastPages (and graduate of the Missouri School of Journalism).

My charge is to aid researchers affiliated with the institute in a scholarly effort to track and analyze the hyperlinks, headlines and images published by Internet news publishers. Our shared goal is to pursue that mission by crafting free and open software tools that benefit from the code of others, and, we hope, ultimately benefit others by breaking new ground.

Our project is far from complete, but the new features at PastPages illustrate what it is possible today using StoryTracker’s codebase, already published on GitHub and distributed via the Python Package Index.

There you can access a menu of options, documented here, for creating an orderly archive of HTML snapshots, as well as the outlines of a system for analyzing content that will expand in the coming weeks.

We have also released django-urlachivefield, a custom database field for the Django web framework that, given a few simple lines of code, will automatically archive a URL to the storage backend of your choice. You can see how it is used within PastPages here and here.

There is also storysniffer, the beginnings of an effort to create a straightforward service that can inspect a URL and return an estimate about whether or not it links to a news story.

Drew:

Late Night

Cure to Boredness

Drive Home

Higher than a Kite

Owen

How Excited Can I Get?

Jackson

Adam

Bianca &  Adam | Game Night

Bianca & Adam | Talking

Bianca & Adam | Higher Than a Kite

Bianca & Adam | Another Night of Meeting Up

Bianca & Adam | Hospital Visit

Bianca & Adam | At it Again

Drew

Bianca & Drew | Talking

Bianca & Drew | Hanging Out Again

Bianca & Drew | Bianca's House

Bianca & Drew | Like We Used To

Bianca & Drew | Reunited and it Feels so Good

Bianca & Drew | Cuddling, Promises, and Hospitals

Bianca & Drew | Meeting and Talking

Fitz

Bianca & Fitz | Smoke Away

Bianca & Fitz | Smoking Distractions

Imogen

Bianca & Imogen | Girl Talk

Bianca & Imogen | Ice cream

Kidnapper

Bianca, Fiona, & Kidnapper | Wake Up, Girls

Owen

Bianca & Owen | Hospital

Zig

Bianca & Zig | Hanging Out

Bianca & Zig | Dance Shows & Presents

Bianca & Zig | Hospital Visits

Bianca & Zig | Breaking Out

Drew:

Tori & Drew | An Evening to Remember

Tori & Drew | Bowling Adventures

Jenna:

Her long, slender fingers glided through her straight, glossy brown hair as she looked at herself in the mirror. Imogen smoothed out the blue polo currently hanging off of one closet hanger before grabbing her leggings hanging right beside it. Tugging them on, Imogen pair it with a pair of faded black jean shorts along with a black t-shirt and a lace cami underneath. After finishing her bright, shiny eye shadow and completing the rest of her make-up, Imogen slipped on her converse and headed downstairs. Jake Martin. It was odd she would be going over to his home today to feed him. Just yesterday she was texting Fiona about how attractive he was. With a laugh at the memory, Imogen pulled out some fried chicken from the fridge before setting off the make the waffles. She could just warm up the food once she arrived to his humble abode of a house. Imogen mixed the waffle batter, poured it into her waffle maker, and pressed, spitting out at least 10 waffles. Finishing, Imogen placed the waffles in a separate container from the fried chicken.