#data driven documents

Open Source GIS and Mapping Tools to Help You Get Started (http://wp.me/pXmHL-Wz) #GIS #Mapping #FOSS

I’ve been an ESRI ArcGIS user since the days of Arc/INFO Ver. 5.0. Having used “command line GIS” since 1991 the rise of open source GIS solutions intrigues me. Trying new products and new software and learning new terminology and getting familiar with many different data formats are valuable jobs skills for me. However, being an educator I’m obligated to expose my students to as many different…

To the tune of The 1975's - Chocolate

Morphin' data into shapes is always a thrill,

We're using Data Driven Doc-ments,

You select, where's the shapes go,

Then bind the data to those DOM nodes,

No matter if they existed,

You know you luv it just admit it, whoa,

We enter and append, exit and remove,

The states in Data Driven Doc-ments

Now everything's in update,

Add styles dynamically,

Use scales and svg,

CSV, call it externally, call it....

How it iterates?, nobody knows,

With magic hidden under it's petticoats,

No we're never gonna quit it,

Making data look delicious, oh

With declarative, that's how it goes,

Say what you want, not how to go,

Its ok being explicit, but its cooler when it gets it, yaknow?

If you ever worked on a D3.js project you may have noticed that you are able to method chain many attr() methods to add attributes to your selection(s). After a while it gets tiring typing attr() over and over again. With multivalue maps you are able to refactor all those attr() methods that are chained together into a single attr() statement. 

Let's say you want to select a 'rect' svg element and add three attributes (x, y, and fill) you would write it like this:

svg.select("rect") .attr("x", 0) .attr("y", 0) .attr("fill", "red");

To refactor this code we can just use one attr() statement and create a property/value pairs object.

svg.select("rect") .attr({ x : 0, y : 0, fill: red });

No matter if functions or styling is used, you can include it on a single attr() statement.

Another example (using functions and styling)

.attr("text-anchor", "middle") .attr("x", function(d, i) { return i * (w / dataset.length) + (w / dataset.length - barPadding) / 2; }) .attr("y", function(d) { return h - (d * 4) + 14; }) .attr("font-family", "sans-serif") .attr("font-size", "11px") .attr("fill", "white");

refactored version

Data, DATA, DATA!!! We all love it, but let's be honest here looking at a bunch of numbers in a spreadsheet or a data structure is not fun BUT if you take that data on your computer, spin it around, sprinkle in a bit of magic dust, add some color and then BA-BAM! You'll probably have ruined your computer. This is why we have sexy magical software to do this for us.

We spend alot of time with data, we owe it to ourselves to spice it up a little. Make the data look purdy (read: purdee). That's where Data Driven Documents (or D3 if you're a sexy beast) comes in. D3 is a Data Visualization library that uses JavaScript to map data to pretty shapes on the DOM. You do 'member what the is DOM right? Kewl.

D3 uses actual DOM elements, so you can mess around with things in the browser dev tools similar to how we do with JQuery. We also use CSS to style our shapes D3. Which is a big freakin' deal since some other visualization libraries don't allow this. 

Since they both work with DOM, D3 follows a similar set of steps as JQuery.

Here's the HTML we'll be working with. 

Now onward to the D3. Some will enter, some will be removed, but they will all be updated (you'll get that joke later...I hope) then we all dance!! 

Step Onesie - Make the selection

We first have to tell D3 where the shapes should go on the page. We do that by making a selection of the HTML elements we want to use to display the data.

This is very similar to selecting elements in JQuery. In line 1, we are selecting the first element on the page with the class "chart". On line 2, we're selecting all the div elements nested inside that first selection we did. Pop quiz! How many divs are inside of the first div with class "chart"? Trick question, there aren't any!

Fooled ya! 

Like in JQuery, these selections will come back as an array like structure (well actually its a subclass of array hooray!) containing the corresponding DOM elements. In this example, bar doesn't contain anything because the selection doesn't return any elements but D3 (Dee-thrizzle) understands bar as representing the potential for a relationship between the data in the next step and elements (kinda like in a real bar when you go out to meet folks for uh...potential. Too far? OK I'll stop.)

Step 2. - Bind/Join Data

Next we need to bind or join the data to the selection. During this step you are trying to connect the data up with the corresponding elements in the selection. 

So on line 3, we are taking the selection of elements that we store in the variable bar and calling the data function on it, passing in the value of the dataArray on line 1, then storing that result in the variable barUpdate. This. This right hurr. This folks, is called binding the data. 

Binding is how we associate the data with all the elements on the page. This is done pretty smoothly since data is a property on the elements in the selection.

RIght now barUpdate technically doesn't hold anything but it does represent the exact number of divs that we would need to put the data into. (Like when you goto the bar with a blank piece of paper that you number with the exact number of placeholders that will be soon filled with phone numbers...ya know?...Ok, I'll stop.)

So now we have the data, we have the exact number of elements needed to put each piece of data (or datum...DA-DAM!) in. Let's put some drizzy data in em.

Step Tres Leches - Enter/Update/Exit lifecycle

After the data has been binded (bound? idk) it then has 3 possible states it could be a part of - Enter, Update or Exit.

The Update state contains the data that was matched up directly with an existing element on the page. So for example, if we had 10 pieces of data and 3 divs already in our selection then we would have 3 pieces of data in the Update state. The update state is held in the variable barUpdate. I know what you're thinking..but...but where did all dem other 7 pieces of data go? ::sniff:: ::sniff::...ok maybe YOU didn't cry (you robot) or used a sweet southern accent (you robot) but I did. Thankfully those next pieces of data are in the Enter state.

The Enter state is where the data who did not find a partner element during the data binding stage. falls. I know, very sad but not really! You see D3, being the perfect match maker, knew these pieces of data would need to matched up sooo...the Enter state has the exact number of data bits needed that we can append elements to...like divs! So we're creating the exact number of divs to hold each shape (bar) for each piece of our data.

And last on the other extreme end is the Exit state. The Exit represents the left over html elements (nodes) that have no matching data pieces. This is actually sad because with no other matching data to fill up their hearts these nodes will simply have to be removed. Moment of silence....twas a good node, twas a noble node, but atlas this node shall be no more... 

D3 does alot of dynamic magic to create the exact number of containing elements for the number of values in the data array. This cycle is a bit mysterious and kinda weird at first blush so to get used to things its best practice to start your html with no containing elements only a parent element, like we did, then just add elements as you need them. If you need a much better love analogy make sure you check out my hilarious friend Sarah's fun post on this cycle.

Step 24/6 - Add / update / remove attributes, content or styles

So we gots the data hooked up with matching nodes, I have a bowl of sour brite crawlers, and my hair is did, life is pretty good right now, except for one tiny detail...we can't actually see anything on the page. Whoopsie!

Right now we have a bunch of empty divs. So in order for use to show off the actual visualization of our data we'll need to add some style! We do this by calling this incredibly cryptic function called .style (I know right?!) on the elements in the Update state.

The style function takes two parameters, the first being the attribute you want to affect and the second is function that lets you do all the craziness to beautify your data.

Now I wanna pause right here and take a second to make sure you understand something critical to D3. This. This thing right here! This is really the meat of D3. Having this function gives you the ability to add logic to your attributes dynamically based on the values in the data array! 

So in our example, the width property of each bar will be a function of the value of the data it represents. For example, a bar representing the value 2 will have a width of 10 * 2 = 20px. How? Notice that on line 1, the anonymous function passed to the style function has a parameter variable called d.

That d is a data object that represents each datum (DA-DAM!) value in the data array. See on line 2, we do some math based on this variable d and return the result. The return value gets set to the width property for each node we bound (binded? I'll never know) data to. 

Now there some magic going on here and we know, if there's one thing that programmers are afraid of it's butterflies (Fact!) but a close second is under-the-hood, non-expressive code.

Turns out D3 is declarative, meaning you just write what you want as an end result and it just works (Kinda how you declare your love for someone in the bar and they just love you back right away...OK, I'll stop).

There's some background implicit iteration going on. So that data object, d actually only represents one element in the array at a time. You can kind of think of it as a block in Ruby where this block of code (d * 10 + "px") will be applied to each element in the data collection.

SCALING

So right now our width is based on an equation but "10" is a hardcoded value. Since we want things to look appropriately sized on the client based on displays and whatnot, we don't wanna pass in hardcoded values. We can use LinearScale for this bar chart to help things look uniform. (Like in a bar when you where a uniform...sorry...)

The details of the Linear Scale and scaling in general are very dense and interesting topics. I put some comments in the codes but I encourage you to learn more by seeking out various resources online and then calling me and giving me the answers.

SVG

SVG is another huge topic but I figure we can just touch on it here just for funsies. SVG (which does NOT stand for Standing Vertical Giraffes. Just wanted to point that out before things got awkward.) is a drawing library that's all about making shapes. It gives us more flexibility to work with more shapes unlike just trying to force it with vanilla HTML alone. We have to change our elements in HTML to SVG instead of DIVs though cause well...they said so and you don't wanna question them. 

SVG is cool and all but this post is getting a bit long and I'm hungry and you must be too. Bring me a sandwich? KThx. So I'm just going to drop a few links below along with a ridiculously long gist, which you can check out at your leisure.

The end result looks alil something like this! 

Thanks for reading folks. Cheerio....mmmmm Cheerios does sound good. Bring me some of those too! KThx.

Resources:

Amazing D3 examples - https://github.com/mbostock/d3/wiki/Gallery

D3 Docs - https://github.com/mbostock/d3/wiki

Joins - http://bost.ocks.org/mike/join/

Scales -  http://alignedleft.com/tutorials/d3/scales

SVG - https://developer.mozilla.org/en-US/docs/Web/SVG

Let's say you have a set of numbers and you want to visualize those numbers as different sized circles, where each circle represents 1 number in your dataset, and the size of each circle directly corresponds to the value of that number.

D3 would be a fine solution to visualizing that data, especially if those numbers might change over time, and some might be removed while others were added. Rather than having to go into your code and manually update the values and circles, D3 would happily do that for you automatically.

This leads me to the enter / update / exit lifecycle of D3, which is a system of keeping the visualization up to date when data is added or removed from the dataset. 

The way this lifecycle works reminds me a lot of how tracking changes in git works. Think of changes in your local repo being akin to changes in the D3 dataset. And using git to add / commit / remove these changes in order to keep the repo up to date, being similar to the D3 life cycle. Take a look at these diagrams:

D3

GIT

The enter state in D3 represents data that does not currently have a corresponding node, meaning the visualization is not update with that data. In order to add matching nodes, we use the enter and append methods.

bar.enter().append("g"); // where bar is the selection and g represents a node

This is like having changes in your local directory and then adding / committing them to git. 

The update state represents all the data elements that have corresponding nodes. After calling enter and append, all of the data elements have nodes. Similarly, once changes have been committed to git, the repo is up to date with all of those changes.

Finally, the exit state represents data that not longer exists in the dataset, for which nodes need to be removed. You would remove these extraneous nodes by calling the exit and remove methods on the selection.

bar.exit().remove();

You could imagine a similar situation in git when you delete files on your local repo and need to tell git to remove them in the repo as well. If you don't deliberately remove them, they would remain in your repo, just as extra nodes would remain in a D3 visualization if not removed.

So D3 is an incredible JavaScript library that visualizes data in a dynamic way.  The visualizations can get very complex, like this one below and these other examples on the D3 website.

I've started learning the basics of how to turn data into something like this, and while I'm still a ways away from rendering something as complex as the network effect above, it definitely seems feasible. D3 is a lot like jQuery in terms of the syntax and structure of the library, so nothing about how to use it is terribly complex. 

Today I worked on something simple: creating a simple bar chart in D3. It looked like this: 

Okay, I know it's not super exciting. BUT, what's cool is:

The size of all the bars were calculated using D3, so if the dataset changes, the bar chart will change accordingly. 

The colors were also generated with D3 and vary with the data -- the larger the number, the lighter the color.

Rather than dive into the code step by step, I'll point you to my Github repo, which has extensive comments in the code. Also check out this tutorial, which is better if you're looking for explanation. 

Instead, I'd like to talk about how D3 works on a high level. I mentioned that it's similar to jQuery in some ways. In my last post, I talked about how to bind events with selections using the .on() method. Similarly, D3 binds data with selections using a .data() method. It will follow a pattern like this:

var vizUpdate = ("selection").data(data);

where...

"selection" can be a set of HTML or SVG elements

.data() is a D3 method that binds the selection to the data, and

data is a variable that represents a dataset in the form of an array