365 Days of Code

I wanted to do a project where I use some command line tools to do some color analysis on a collection of photos. Instead I fell down a rabbit hole.

First, I found some test images that were free to use and free to download from the National Gallery of Art. (The NGA is awesome and offers open access to all of their images.)

I decided to take a closer look at trends within the most requested images from their open collection. There are 50 works of art, and 50 is a nice round number.

I found myself clicking on each image to download it.

“This is not efficient, Jenna. Isn’t there a small shell script that could do this for you?”

So, I did a little research and I wrote a small shell script with wget that pulls only jpg images off the page.

wget -r -A.jpg http://websitegoeshere

Things started downloading. Yay, I’m scraping!

Well, I was only downloading the thumbnails because those were the images the National Gallery of the Arts put directly on the page because they want the page to load quickly. The big images would slow things down. Those thumbnails downloaded just fine, but I didn’t want those.

I looked closer at the website source code and the full-size photos don’t live on the homepage. Their URLs were assembled with JavaScript when you click on the download arrow. So what I really needed to do in order to download these larger images was to write some Python using RegEx that will find the number and reassemble the URLs to download the full-size image.

To be continued.

Refreshing my memory on a few exercises. Nothing I want to post on the internet just yet, but in the meantime here’s a picture of my cat:

At NICAR in Denver this year, I took the First News App class taught by Ben Welsh from the LA Times and Derek Willis of ProPublica.

It was a sort of paint by numbers approach to building a news app, using Python, Flask as a framework, JavaScript and a bit of CSS. Here is what I made. It's a map and database of who and where people were killed during the LA riots in 1992. (Any clunky bits are courtesy of your friendly #365daysofcode author and not the curriculum.)

Back. There’s going to be some treading water and relearning what I used to know.

Also, I need to get this thing off Tumblr and fix that god awful template coloring. I’ll probably dump this site on GitHub and call it a day.

Sometimes you want to have the result of a program or shell script to paste to your clipboard. On a Mac, the command pbcopy lets you do that. You just need to pipe the information to pbcopy.

As an example: python example45.py | pbcopy

I'm plugging along on my data analysis class on Coursera with Dr. Mine Çetinkaya-Rundel from Duke University. Some of it is review for me, but I like how thorough and consolidated the class is. I had been working aggressively through the R labs, but I'm trying to focus on catching up on the video lectures so I better understand the theory behind what I'm graphing and analyzing in R.

Some other methods of visualizing data are dot plots and box plots. Çetinkaya-Rundel said the dot plot is useful when the actual individual values are of interest and these kinds of plots can get too busy as sample sizes increase.

Box plots, on the other hand, the darling of people who love data, give a good summary of the data. The professor said these are useful for highlighting outliers, pointing out the median and the interquartile range.

Like histograms, you can also tell if the data is left skewed, symmetric or right skewed based on the box plot. Right skewed indicates that the mean is greater than the median, and left skewed shows that the mean is less than the median.

My Cousera Data Analysis and Statistical Inference class is covering the basics of how to best visualize and describe data. 

For instance, a scatterplot can visualize two numerical variables, said Dr. Mine Çetinkaya-Rundel. A histogram can visualize distribution of a numerical variable that Çetinkaya-Rundel said provides a view of data density and identifying the shapes of distributions. 

Short answer: A lot.

I saw really wonderful panels and had to miss some, too, because you can only be in one place at a time. Alas.

Here are a few tools that sounded useful or that I need to look into:

Censusreporter.org: Helps you create files that are ready to map. Can work with Google Fusion Tables.

QGIS - Free mapping software 

TileMill - Software that makes some seriously fancy maps and lets you customize with a CSS like language. Free for up to 3,000 PVS a month, but than $5 a month for up to 10,000 pageviews and the plans go up from there.

Leaflet.js - A Javascript library that TileMill is made from. I should see if I could use it by its lonesome.

D3.js - Reinforced that this Javascript library is useful for graphics and maps, but not always the best for mobile.

SQL - Great for building databases. I'll probably continue with Zed Shaw's Learn SQL the Hard Way.

And here are highights from a few of the many panels I went to:

Mapping and data viz

I went to a few sessions on mapping, which gave a lot of best practices and software/programming options. The Wall Street Journal's Tom McGinty explained how he used a map to show how many people were within a mile of fracked oil wells. This technique could be applied to a wide range of stories and topics.

In a census reporting mapping session led by WNYC's John Keefe he showed different tools for mapping (Fusion Tables, Tile Mill, CensusReporter.org) and how to pull take census data and create shapefiles for mapping.

In another map panel with WNYC's Noah Veltman and MapBox's Tom MacWright, they said to to avoid colors that impact color blind readers; to not put too much on a map or you overwhelm readers (one panelist said you will “break people’s minds”); to make sure the map works on mobile; to remember to zoom in to the area that’s important; and to enlist critical colleague who will be honest when the map just isn’t working. 

I really loved the lightening talks. Two of the data viz ones were especially fun:

How to use small graphics and charts to tell a digital

How data visualization can go very wrong

Takeaways from the worst of data viz talk were to avoid confusion and too much information; to make sure to fact check and copy edit graphics; and to watch your scale.

Offbeat

Some of the very specific panels that seemed irrelevant to my beat turned out to be very informative and broadly applicable. The inspections session by Michael Pell of Reuters and Joce Sterman of ABC2 pointed out how intrinsic inspections are to everyday life. There's a paper trail (or more likely, a database trail) associated with all of those inspections, too.

Project Management

In the "Proper Workflows for Data Projects" session panelists said to automate and show your work whenever possible to avoid mistakes on digital and data projects. They also said to make sure you have the same review process for web projects (with a copy editor looking at the text) as you do regular stories. Brian Boyer of NPR, Ben Welsh of the LA Times, John Perry of the Atlanta Journal-Constitution and Kevin Schaul of the Star Tribune participated in that panel. 

I got a small information session on version control tonight from Chris because I've been getting into some bad habits and have been putting off really learning Git beyond pushing my work to GitHub. He even downloaded a white board app on his iPad for illustrations.

In summary, this infographic is quite handy.

Anyway, now I'm going to start tracking my work on my computer in Git so it's easier to rollback to earlier versions of my projects and so I can do some experimenting in branches without ruining my working code. Messy creativity without Git just makes things complicated.

And here are a couple of little tricks I learned:

\--patch allows you (technically) to go through each chunk of changes and decide which ones you want to add.

I've been pulling my hair our figuring out that word parsing code bug, but I learned today that I really need to make commenting my friend. What does that function do? Spell it out.

/*Function takes a passage and counts how many of each word. 

Takes an array of strings.

Spits out an object containing each word and its word count. */

The more I work with vi, the easier it gets. I'm still limping around by using the arrow keys when I forget shortcuts, but it's getting better. I found out I can both save my code and then run it at the same time without leaving the vi editor using a pipe |. I installed Node.js so I could run JavaScript on the command line.

:w | ! node %

My Kafka word analysis tool now splits words that are hyphenated into two and removes punctuation, thanks to adding in some regex commands.

I wrote this little script that takes a string, separates it and tells you how many of each word you have in the string.

I ran into a problem with capitalization differences, so I converted all the words to lowercase. Here are a few of the words:

This helped. Notice, though, that there is punctuation in there, messing my count up. Removing the commas, the quotation marks and all the extra punctuation is tomorrow's project.

Even taking into consideration the extra punctuation problem, this translation of Kafka doesn't have much word repetition. I'll run the overall story after I finish the last steps tomorrow to see if this pattern continues throughout.

If you're interested, here's the rest of Kafka's Metamorphosis on Project Gutenberg.

I practiced moving around the Vi editor with the search command. You can search forward by hitting slash:

/: {1:9}

This searches for a colon followed by a number. You can search backwards in the document using a questions mark: