#lubridate

My 3 video presentations on “Essential R”

In this post I include my  3 video presentations on the topic “Essential R”. In these 3 presentations I cover the entire landscape of R. I cover the following

R Language – The essentials

Key R Packages (dplyr, lubridate, ggplot2, etc.)

How to create R Markdown and share reports

A look at Shiny apps

How to create a simple R package

Essential R – Part 1 This video cover basic R data types –…

I really wanted to put something together for this series on the twitteR package. Unfortunately, at the moment the number of interesting things than can be done with twitteR, as opposed to through API calls and RCurl, is limited. Regardless, I have Yet Another Invented Application to illustrate a pretty typical use-case for twitteR: grabbing Tweets by search term.

I've done this before, for sentiment analysis of Tweets about Republican presidential primary candidates, and indeed, despite its limitations, the searchTwitter() function can be useful. Since the number of Tweets one can grab appears to be limited to 1000, this Gist attempts to infer term popularity by frequency -- with only minor success, as you can see in the plot below.

We've recently seen how to parse XML for the data that goes into producing graphs such as the one on this page, comparing the Romney vs. Obama polling average.

Today's Gist shows how to approximately replicate that very figure, with two trend lines indicating each candidates' polling average, plus an area plot of the difference between the two. This is done in ggplot2 by way of a semi-hack, which uses the scales package to transform the y-axis so that part of the continuous scale is "cut out."

The figure above is built in 5 steps:

Plot the two competing trendlines

Add an area plot showing Obama's (possibly negative) lead

Write a function (scaleBreaker()) that takes a continuous vector and makes a break in it, between a lower and an upper threshold

Use this function to make a custom axis transformation with trans_new()

Add standard ggplot2 axis breaks and labels, apply the custom transformation, and draw a line to emphasize the disjunction

Now that the election is over, you may want to use polling data in a model of the campaign. Simon Jackman has thoughtfully made his daily state-by-state predictions available for download, but a commonly-used dataset is the RealClearPolitics polling average.

As you can see when you go to RCP, they have a nice HTML5 graph (screenshot above), over which you can hover with your mouse to reveal daily point estimates. Unfortunately, the numbers that compose those point estimates are a little tricky to tease out -- at least, it was tricky for me. Fortunately, I managed to wrangle out the Romney vs. Obama daily averages, which you can download here [CSV].

Fortunately, RCP uses stores their time series data in XML, meaning that the method I used to get those Romney vs. Obama numbers can be used to collect any RCP data, such as from this comparison of Obama & Bush Job Approval. Just view source, and [CTRL-F] for "xml," and try to identify the XML file from which the graph is drawing data:

In this case, the file appears to be o_vs_b6.xml, which we can find listed in this directory of all RCP XML files and graph-drawing code.

From there, you can just use the R package XML and the following code as a guide for neatly folding the XML data into a data.frame. It will take a little effort on your part (i.e. it's not just "CTRL-A, CTRL-R"), but the XML should be consistently-formatted, and thus not too difficult to parse.

As a Political Scientist, it could not be more gauche to talk about the Presidential debate in terms of a winner and a loser, but the occasion provides the opportunity to show how to do (at least) three really useful things:

Directly load price and volume data for InTrade contracts. All you need to know is the InTrade contractId, which can be found in the URL of a page like this one, for Barack Obama to win the third Presidential debate.

Convert the character-class time and date from the InTrade .CSV to a POSIXt-class time and date, which many R functions recognize and use in a special way. This is done with parse_date_time from lubridate, which really is as amazing as people say it is. (More on lubridate here.)

Superimpose the price trend over indicators of trade volume, using ggplot2. There is some odd (to me) behavior in the plotting code, meaning that in one place we have to specify the time as.numeric, but it gets the job done.

R's lubridate package really helps here. How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)?

month=rep(1:12,times=100) year=rep(1901:2000,each=12) d=paste(year,month,1) sum(wday(ymd(d))==1)