Statlandia

From Andrew Gelman's blog:

There’s so much that goes on with data that is about computing, not statistics. I do think it would be fair to consider statistics (which includes sampling, experimental design, and data collection as well as data analysis (which itself includes model building, visualization, and model checking as well as inference)) as a subset of data science. . . .

The tech industry has always had to deal with databases and coding; that stuff is a necessity. The statistical part of data science is more of an option.

To put it another way: you can do tech without statistics but you can’t do it without coding and databases.

Just a few words of thanks to the peer reviewers of my submittal for the final Passion Driven Statistics assignment. I'm pleased by your comments and scores. Thanks to all the peer reviewers for making it to the end of the course and taking the time to look at your assigned blogs.

I enjoyed the experience of a MOOC and I thought Lisa Dierker organized and delivered the material extremely well. She explained the material clearly and the pace was perfect.

Another attraction was to be able to use SAS a bit. Although our use was limited, it gave me small taste of what the application was all about and how one goes about doing an analysis in SAS. After that limited exposure, I still prefer R, but wouldn't mind using SAS more extensively - but the cost is pretty steep for an individual not supported by a business or as an independent scholar. That fact makes R even more appealing.

Final Project: The Association between Life Expectancy and Income in Countries with Different Levels of Democracy

Florian Walchak, Denver, Colorado, USA

Introduction

It is a reasonable supposition to believe that the citizens in countries with higher per capita income enjoy longer lives. Higher incomes provide increased access to food, medical care, and education. We investigate the relationship between per capita income and life expectancy in this paper to see if the supposition is correct.

We also investigate whether the correlation between income and life expectancy is influenced by the level of democracy in a particular country. One might surmise that access to food, medical care, and education could be limited or rationed to citizens in countries that do not allow citizens a full voice in the governance of their country. Such limits could negatively influence their life expectancies. That is, life expectancies in countries with similar per capita income would differ (be lower) in those countries with lower levels of democratic governance.

Research Questions

1. Is life expectancy associated with income?

2. Is life expectancy the same between those countries with similar incomes but with different levels of democracy?

Methods

I used the Gapminder dataset for the analysis. This dataset has collected information regarding 16 variables from 289 countries around the world. The variables I used for this analysis were “lifeexpectancy”, “incomeperperson”,  and “polityscore”.  The lifeexpectancy variable is the 2011 life expectancy at birth in years. That is, it is the average number of years a newborn child would live if current mortality patterns in a country were to stay the same. The information came from four sources: (1) Human Mortality Database; (2) World Population Prospects; (3) Publications and files by history professor James C Riley; and (4) Human Lifetable Database.

The incomeperperson variable is the 2010 Gross Domestic Product per capita in constant year 2000 U. S. dollars. The information comes from the World Bank Work Development Indicators.

The polityscore variable is the overall 2009 polity score from the Polity IV dataset, calculated by subtracting an autocracy score from a democracy score. It is a summary measure of a country's democratic and free nature. -10 is the lowest value, 10 the highest.

Each variable column in the dataset had missing data relative to the 289 observations potentially available in the dataset. The lifexpectancy variable had 197 valid observations ; incomeperperson had 191 observations; and polityscore had 116 observations. Missing observations were stripped from the variable columns in the data preparation step and were not used in the analysis. SAS was the primary statistical analysis tool used with R being used to facilitate quick-look plots and replicate statistical procedures and analyses originally carried out in SAS.

During the exploratory data analysis phase of this study, I plotted each variable with an appropriate univariate visualization tool. In the cases of lifeexpectancy and incomeperperson, I displayed the data with a histogram. In the case of the polityscore variable, a bar chart shows the frequency distribution. Figures 1, 2, and 3 show the initial distributions of the data for each variable. Figure captions include other descriptive statistics as appropriate for the type of variable such as mean, median, and standard deviation.

       Figure 1. Distribution of life expectancy for 197 countries. The minimum life expectancy in the dataset was 47.79 years for Sierra Leone. The maximum value was 83.39 years for Japan. The mean is 69.86 years, the median is 73.24 years, and the standard deviation is 9.67.

Figure 2. Distribution of Income. This histogram shows the 2010 Gross Domestic Product per capita in constant 2000 US$. The inflation but not the differences in the cost of living between countries has been taken into account. The lowest income is for the Democratic Republic of the Congo at $103. The highest is for Monaco at $105,100. The mean is $8,705. The median is $2,550. The standard deviation is $14,234.

Figure 3. Frequency distribution for the “polityscore” variable. This variable assigned a “level of democracy” score for 162 countries. The range of the indicator is from 10 (fully instititutionalized democracy) to -10 (hereditary monarchy). The category with the largest population is 10 (the fully institutionalized democracy level) with 33 countries included for 20.37% of the total. For levels -10 through 0, the combined count was 52 countries for 32.10% of the total.

  Univariate results that describe each variable are as follows:

For the lifeexpectancy variable, notable statistics for the 197 countries are:

·         Mean = 69.86 years

·         Median = 73.24 years

·         Standard Deviation = 9.67 years

·         Minimum = 47.79 years

·         Maximum = 83.39 years

·         Range = 35.6 years

For the incomeperperson variable, univariate data of interest are:

·         Mean = $8,705

·         Median = $2,550

·         Standard Deviation = $14, 234

·         Minimum = $104

·         Maximum = $105,147

·         Range = $105,044

For the polityscore variable we show a frequency table from which the bargraph in Figure 3 was produced.

Table 1. Levels of Democracy for 162 Countries. This table shows the count and frequency for each democracy level.

Bivariate data is treated in two ways due to the data types being compared. First, Lifeexpectancy and incomeperperson are considered continuous (numerical) variables while polityscore is considered as categorical data (albeit present in the ordinal or ranked fashion). The correlation procedure was used against the two numerical variables with incomeperperson as the predictor variable and lifeexpectancy as the response variable. The relationship was also visualized by a scatterplot. Table 4 shows the correlations result and Figure 4 contains the scatterplot of the bivariate data.

Table 2. Pearson Correlation between Income and Life Expectancy. The procedure shows a strong correlation of 0.601(highlighted in yellow) between the variables with a p-value of less than 0.0001.

A visualization of the data looks like this:

Figure 4. Income vs. Life Expectancy. The plot shows definite structure to the data and suggestive of a strong association. SAS’s correlation procedure confirms this.

Though suggestive, the graph really indicates a non-linear relationship, but for our analysis we want to stay in the linear domain, so we transformed the incomeperperson variable by taking the natural logarithm of the predictor data and plotting those results against lifeexpectancy. The results illustrate the strong linear relationship indicated by Pearson’s Correlation Constant. In the log-scaled case, R-squared is 0.6446 and the p-value is much less than 0.001.

Figure 5. Using a log-scale for the predictor variable. The scatterplot now shows a strong linear relationship of income with life expectancy.

The second method of treating bivariate data was used when one of the variables being considered was categorical. We can use the analysis of variance (ANOVA) technique to compare categorical predictor variables to numerical response variables. We can also use ANOVA to illuminate the effect of a hidden or moderating variable – in our case the polityscore – on the response variable. I binned both the incomeperperson variable and the polityscore variable into four levels thus effectively turning them into category variables. The ANOVA again showed a strong positive relationship with the binned data. The Duncan Multiple Range Test shows that the means for each income category are significantly different.

Finally, I investigated polityscore as a potential moderating variable. The ANOVA technique was used with the provision for a moderating variable. To reduce the number of tables that is the output of the SAS procedure, I binned the 21 possible levels if democracy into 4 categories.

Results

There is a strong correlation of income and life expectancy. The scatter plot in Figure 5 shows the log-transformed income data that better illustrates the linear relationship the data have. The results illustrate the strong linear relationship indicated by Pearson’s Correlation Constant. In the log-scaled case, R-squared is 0.6446 and the p-value is much less than 0.001. There is very strong statistical evidence that income and life expectancy are strongly and positively associated.

  Figure 6. Log-scale Income vs. Life Expectancy. When predictor variable is transformed into its logarithm, the strong linear relationship between income and life expectancy becomes clear.

Less clear, however is the moderating effect, if any, that the level of democracy has on a country’s life expectancy. The p-value for the Category 1 polityscore variable (which includes polityscores  -10 to -5) was 0.34. This means we cannot reject the null hypothesis that a low level of democracy has no influence on the life expectancy for a given income level. Thus, the alternative hypothesis that low democracy levels are associated with low life expectancies for a give income level is not affirmed by the statistical evidence.

However, In the cases of polityscore categories politycat=2 (levels of democracy -5 to 0), politycat=3(levels of democracy 0 to 5), and politycat=4 (levels of democracy 5 to 10) the p-values are 0.0044, 0.0012, and <0.0001, respectively. We can reject the null hypothesis and say that higher levels of democracy are associated with increased longevity for a given income level.

 Discussion

What the results might mean?

The results clearly show that higher income is associated with a longer life. Since a longer life is normally considered beneficial, economic development is important for individual and national well-being. The data also show a very large range in income - four orders of magnitude – from about $100 GDP per capita to just over $105,000. They also show that 75% of the countries surveyed have less than $9,500 GDP per capita. There is a significant challenge to raise the standard of living of over three-quarts of the world’s countries to match those in the first quartile.

The analysis also suggests an ambiguous relationship of life expectancy with level of democracy. We cannot clearly say that a low level of democracy influences life expectancy. Yet, the analysis showed that the highest levels of democracy were associated with longer life expectancies.

Strengths

This was a large dataset that allowed for analysis on most, if not all, the sovereign nations in the world. It allows the analyst to see the “big picture’ relative to world development and progress

Limitations

GDP per capita may not be the best way to measure the income per person in a country. In addition, the accuracy of GDP figures can vary depending upon the reporting country and agency. The World Bank no doubt strives for accuracy, but their data is no better that what is given to them.

Recommendations for Future Research

It would be interesting to see if low income levels were associated with lower levels of democracy. Since we know the low income is strongly associated with low life expectancy, we might then more clearly determine what role a low polity score might play in life expectancy.

Appendix I

SAS Program

  libname mydata "/courses/u_coursera.org1/i_1006328/c_5333" access=readonly;

/*

Investigating four variables from the Gapminder dataset: incomeperperson, armedforcesrate,

lifeexpectancy, and polityscore. Hypothesis 1: Armed Forces Rate and Polity Score are correlated with a larger armed forces rate associated with a lower "Level of Democracy" 

polity score.                                                                                                  

  Hypothesis 2: Income and polity score are correlated to life expectancy.

Specifically, longer life expectancies are positively associated with greater income per

person and increased levels of democracy, with income being the strongest influence. 

*/

  DATA new; set mydata.gapminder_pds;

/*grouping continuous variables into categorical variables for graphing and analysis*/

/* starting with the incomeperperson variable*/

if incomeperperson eq . then incomecat=.;

else if incomeperperson le 1000 then incomecat=1;

else if incomeperperson le 5000 then incomecat=2;

else if incomeperperson le 10000 then incomecat=3;

else if incomeperperson gt 10000 then incomecat=4;

    /* grouping the armedforcesrate variable */

if armedforcesrate eq . then armedforcescat=.;

else if armedforcesrate le 1.0 then armedforcescat=1;

else if armedforcesrate le 2.0 then armedforcescat=2;

else if armedforcesrate le 3.0 then armedforcescat=3;

else if armedforcesrate gt 3.0 then armedforcescat=4;

    /* grouping the lifeexpectancy variable */

if lifeexpectancy eq . then lifecat=.;

else if lifeexpectancy le 55 then lifecat=1;

else if lifeexpectancy le 65 then lifecat=2;

else if lifeexpectancy le 75 then lifecat=3;

else if lifeexpectancy gt 75 then lifecat=4;

  /* grouping the polityscore variable */

if polityscore eq . then politycat=.;

else if polityscore le -5 then politycat=1;

else if polityscore le 0 then politycat=2;

else if polityscore le 5 then politycat=3;

else if polityscore gt 5 then politycat=4;

/*Variables are adequately and understandably named, but I will rename some for readability*/

  LABEL      incomeperperson = "Income Per Person"

           armedforcesrate = "Armed Forces Rate"

           lifeexpectancy = "Life Expectancy"

           polityscore = "Level of Democracy";

  PROC SORT; by country;

PROC ANOVA; CLASS incomecat;

     model lifeexpectancy=incomecat;

     means incomecat /DUNCAN;

/*PROC ANOVA; CLASS polityscore;

     model lifeexpectancy=polityscore;

     means polityscore /DUNCAN;*/

PROC ANOVA; CLASS politycat;

     model lifeexpectancy=politycat;

     means politycat /DUNCAN;

PROC SORT;by polityscore;

PROC ANOVA; CLASS incomecat;

     model lifeexpectancy=incomecat;

     means incomecat; by politycat;

PROC FREQ; TABLES polityscore; title 'Levels of Democracy';

PROC FREQ; TABLES polityscore*armedforcescat /CHISQ;

PROC UNIVARIATE; var incomeperperson;

     histogram incomeperperson;

     title 'Per Capita Income Distribution of 191 Countries';

     inset n = 'Number of countries surveyed in the Gapminder dataset' /position=ne;

PROC UNIVARIATE; var armedforcesrate;

     histogram armedforcesrate;

     title 'Size of Armed Forces as a Percentage of Population in 164 Countries';

     inset n = 'Number of countries surveyed in the Gapminder dataset' /position=ne;

PROC UNIVARIATE;var lifeexpectancy;

     histogram lifeexpectancy;

     title 'Distribution of Life Expectancy in 197 Countries';

     inset n = 'Number of countries surveyed in the Gapminder dataset' /position=ne;

PROC CORR; var incomeperperson lifeexpectancy;

run

Appendix II

SAS Output

Click on the link below to access the SAS Output file

John McCarthy

This is a progress report regarding the assignments for the Passion Driven Statistics course. I’ve gotten a little ahead of the class schedule, and that’s probably a good thing. All the posted results are from SAS, although I am also doing the analysis in R. I’ll post R results (plots, graphs, and code) as we go along. Today I’m summarizing where I am at with respect to the analysis of the research question. I’m also anticipating the need to pull together a poster-paper at the end of the class describing the research questions and the results, and I suspect many of the items I report on today will find their way into the poster.

Assignment 1 – Research Question

After taking a look at all the data sets to see which one might be most appealing to hypothesize about, I settled on the Gapminder dataset. A couple questions come to mind after looking at all the variables and a quick look at the data:

(1) Is there a relationship between a nation’s “amount” of democracy and the size of its military in terms of percentage of the total population?

(2) What can looking closely at the relationship among the “democracy”, “income”, and “longevity” variables tell us?

My hypothesis for (1) is that the larger the military, the less democratic a country is.

The hypothesis for (2) is that income and the level of democracy positively correlate with longer life expectancy and, (2a), that income will have the stronger correlation.

  Assignment 2 – Frequency Table

We were asked to use the FREQ Procedure to produce a frequency table of a categorical variable relevant to our research topic. I submitted the Log file associated with this table:

Assignment 3 – Data Management

I did not do very much data management for the dataset that I used. The variable entries were used as presented. I did rename variables for reading clarity.

Assignment 4 – Graphing 1

This assignment asks us to use SAS's graphing wizard to graph a single quantitative or categorical variable and submit the Log file that is automatically generated. If we choose a quantitative variable, we should also separately run the Univariate Procedure to better understand the graph's shape, center and spread.

Using the Bar chart wizard in SAS, I made a bar chart of the Polity Score categorical variable (actually an ordinal variable). The wizard used a default binning algorithm, so it’s more of a histogram than a bar chart. Not all values of the Polity Score variable are shown.

I was able to produce a nicer plot (in color, no less!) in R.

  Finally, applying the univariate process to the remaining three variables (income per person, life, armed forces rate, and expectancy) we get:

These basic plots were accompanied by the tabulated statics (e.g. mean, standard deviation, etc.) for the variable. It would be too tedious to list them all here.

  Assignment 5 – Graphing 2

This assignment asks us to use the SAS graphing wizard to visualize an association between two variables. We are to make sure to follow the Graphing Decisions Flowchart in order to decide what type of graph to use and if any variables need to be managed first. We are to submit the Log that is generated automatically. We are also asked to note that the Univariate Procedure may help to describe any quantitative variables you choose.

For this assignment I didn’t use the graphing wizard, but went directly to the scatter plot menu. I created two graphs showing the relationship between Income and Life Expectancy. The first shows that most observations are crowded into the lower end of the Income domain. The second shows the Income variable transformed into its log.  The scatter plot is more spread out and easier to read. It also shows a definite positive correlation.

For comparison's sake I'm showing the plot of income vs. life expectancy with the x-variable (income) on a linear scale. I'm also showing the regression line in red. much more local crowding with the data points compared to the log plot shown in the previous post.

The loess curve does a regression on small subsets of the data then pieces it together. It shows another aspect of the data shape. The scatterplot command also generates histograms for the quantitative data associated with each variable. Executing the command once the dataset in downloaded (making sure the car package is installed and the car library opened) is pretty simple:

This is an informative set of slides from  Carlos Somohano (founder of Data Science London) - particularly if you are just starting down the data analyst path:

Just saw the e-mail from PDS saying the Gapminder data set was incorrect regarding the polity score and a new data set has been posted. My updated frequency charts are posted below. (The box plots and histograms of the other variables are still valid). The first plot uses mostly default parameters for the hist() command in R. I spiced-up the second frequency chart (same data) with a bit of color.

A couple of graphic formatting commands I do use on a regular basis came from Andrew Gelman's blog Statistical Modeling, Causal Inference, and Social Science. Gelman is a professor of statistics and political science at Columbia Univeristy. The formatting commands are:

par (mar=c(3,3,2,1), mgp=c(2,.7,0), tck=-.01)

Spent the afternoon poking around in R and plotting three of the four variables that I am interested in from the Gapminder dataset: Income, Life Expectancy, and Armed Forces Rate. I did a frequency plot of the fourth variable - Level of Democracy - the other day.

I made several different plots for the variables, each having its own way of displaying the data. This gives us a more comprehensive characterization and understanding of the variable.

Let's start with Life Expectancy. I have three plots: A box plot, a histogram, and a dot chart (I did the same for the other variables).

Income is next. I added a histogram and a box plot on a log scale so we could better see the distribution.

The dot chart looks like this:

Finally, I have the same three types of charts for the Armed Forces Rate (i.e., the size of a country's armed forces as a percentage of the population):

For the dot chart, there are too many countries to clearly list them on the y-axis. However, the observations are in the same order of countries that we find in the Gapminder dataset.

It's a challenge to figure out all the formatting commands and labeling quirks of R. For instance, some of the titles in the Armed Forces series of charts are cut off because I haven't quite figured out how to set plot areas and margins. Nor do I have the proper log numbers on the x-axis for the log-scale income charts, but I think you can get the gist of what the data are saying. Now, to do the same thing in SAS!