Funky Stats

My colleague always complains about the fact that it rains all the time on week-ends or when she is on vacation. This is statistically significant. At least for the last 60 days.

What I hear every Monday

Every Monday morning, same song: “it rained all week-end long, I couldn’t do anything out.” My colleague thinks she is cursed, as every time she is off on vacation or on week-ends, the weather gets bad.

We decided to check if this curse is statistically significant. We asked the question “are my colleague’s days off significantly associated to bad weather?”

Collecting the data

We filled in the following table day after day, during 60 days:

Another way to formulate our question are the “No’s” in the Working day column often associated to Bad’s” in the Weather column?

Working day and Weather are nominal variables, also referred to as qualitative variables. The values they take come from sets of categories (Yes/No; Good/Bad). They are not numbers.

A compact and informative way to summarize the association between two nominal variables is to use a contingency table, which places the categories of one variable in rows and the other one in columns:

The core of the table contains counts of days corresponding to the combination of categories. For example, over the 60 assessed days, there are 10 non-working days with bad weather.

It may be more interesting to visualize it as a row-percentages table:

Interesting: 71.4% of non-working days are associated to bad weather. Let’s go further and try to see if there is an association between working (yes/no) and weather (good/bad).

Investigating the significance of the curse

Let’s put things properly in a statistics language: we want to evaluate the association of ( = correlation between) two variables. Two qualitative variables. According to this grid, a way to assess the correlation between qualitative variables is to run a chi-square (χ²) test on the contingency table.

We ran the test and obtained a p-value of 0.03. If we compare this p-value to a 0.05 risk threshold of being wrong, we may say that there is a statistically significant correlation between weather and working days. By saying this, we are taking a low (3%) risk of being wrong.

Design by freepik

Conclusion: days off, bad weather

This conclusion holds for the 60 last days, at least. It is not necessarily true for all days in life (unless these 60 days are representative of days in general).

Do days off encourage the weather to be bad?

Significant correlation does not necessarily imply causality. This result does not let us state that days off induce bad weather or that bad weather induces days off. It simply says that there is some kind of a significant association between working days and weather (with a quantified risk of being wrong).

Now let’s get back to work and count the hours until next week-end. Nice weather outside!

Playing with statistics in a Goblin world.

The context

A Goblin wizard invents a spell to boost the strength of Goblin warriors and help them shout victory against the nasty human army. Dreadful humans.

Easy: you scream Gogogox* in front of a Goblin and he starts growing to become a big bad terrifying… thing.

The experiment

The Wizard wants to be sure the spell works well. He randomly picks up 30 Goblins in their training kennels and measures their strength on a punch machine stolen at the Hexteria* funfair.

The spell is then cast on the 30 Goblins. Their strength is measured again afterwards.

And here come the results

The graphic says it all: this experiment was a big fiasco. Instead of increasing after the spell, goblin strength lamely collapsed.

The Goblin in the picture below does not even look like a Goblin anymore.

Running the wrong statistical test

The wizard decided to go further and compare the two average scores statistically. He runs the very commonly used t-test for two independent samples.

Surprise! The p-value he obtains (0.07) is higher than the significance threshold he defined (0.05). In English, this means that the strength scores do not change significantly after the spell has been cast.

Statistically speaking, one might think that the wizard is not guilty of having reduced the strength of 30 goblins. Huge contradiction with the graphic.

Are statistics total nonsense?

Not if they are used the right way.

Actually, the test that was used by the Wizard only works if all measurements are totally independent. In this case, that would mean 30 goblins not submitted to the spell compared to 30 other goblins that were Gogogoxed. This is not the situation we have here: the strength score of every goblin is measured twice (before and after the spell). We say that data are organized by pairs.

A more appropriate way to test the difference between two paired samples… is to use a t-test for two paired samples. Easy as Goblin pie.

Running a more appropriate test

The goblin statistical committee runs this test on the same data and obtains a p-value < 0.0001. So this spell really seems to drive Goblin strength down.

This Wizard may be bad at creating spells. This does not prevent him to be very clever at doing anything to claim his innocence.

 Only for statistical addicts among the readers: Student’s t-tests for independent or paired samples are parametric and are thus reliable only under specific conditions. Non-parametric robust equivalents are Mann-Whitney’s U test (for independent samples) or Wilcoxon’s W test (for paired samples).

 Statistical tests were performed using the XLSTAT software.

What?

Several scores from theNBA All-Star Game Box Score acquired from the basketball-reference.com website were united in a single plot obtained thanks to the terrific Principal Component Analysis (PCA) tool.

Interpreting the PCA

A larger size of the plot can be seen here.

If you never heard about PCA before, do not try to interpret the axes. Just consider that two players who are close to each other are relatively similar with regards to the scores we used in the analysis (Field Goals, 3 Points, Rebounds, Assists, Steals, Turnovers, Personal Fouls). It looks like the Gasol brothers have very close scoring profiles (upperleft side of the plot). Players who follow the same horizontal or vertical direction of a specific score according to the central (0,0) point are simply good at that score. James, Brook and MVP Westbrook are good at marking points Whereas Duncan is bad.

A larger size of the plot can be seen here.

Feel free to comment our PCA plot further below!

Technical mumbo jumbo

Well, that could partly explain what comes out of a dataset published by the Guardian, at least.

As I was looking foran open dataset to do a funky datamining analysis, I stumbled upon the guardian’s list of 1000 songs to hear before you die, classified by theme and by year (guardiandata).

I played with the data and this is what I found.

First step: fishing for interesting stuff in a big dataset

First of all, I decided to remove all songs recorded before 1960, since they only accounted for 7% of the data. Then, I computed the percentage of songs belonging to each theme within each decade. This produced the ugly table below.

Let’s make it prettier through a - don’t be afraid - correspondence analysis.

The interpretation is a piece of cake. If you are new to correspondence analysis, don’t try to interpret the axes. Simply consider that those two dots, close to each other… How may we say it… well… are close to each other. Compared to other dots, of course.

For example, in the 1990’s, the Guardian selected more songs linked to Life & Death, compared to the 1980’s, where they were more keen on party and politics songs. Interesting, huh?

On the other hand, in the 1960’s, the Guardian seemed to vote more for love songs, compared to the 1970’s, 1980’s and 2000’s... What?

Second step: from You Really Got Me (The Kinks) to Umbrella (Rihanna)

Let’s take a closer look at how the proportion of love songs selected by the Guardian evolves across time.

As you can see, the red line decreases through time. And a logistic regression analysis confirms that this decrease is statistically significant (we may say this with a very tiny risk of being wrong).

What would that mean?

Many hypotheses may be true, including:

1)      Love songs were more dominant in the 1960’s repertoire compared to now.

2)      Love songs were not more dominant in the 1960’s, but were of better quality.

3)      A mish-mash of hypotheses 1 and 2 (+ other possibilities which we did not think of).

Please help us with the interpretation

I did the statistical treatments but I know nothing about music. I need you to provide exciting interpretations in the comments below. Go go go!

… A very unserious study from FunkyStats to start this blog and to wish you a Merry Christmas.

The problem

This morning, my niece asked me if the number of presents one receives on Christmas is really a matter of good behavior.

Okay

Let’s play with statistics. I computed questionnaire-based good behavior scores (spanning from 0 to 10) in a sample of 100 children all around the world and then asked them how many gifts they received from Santa last year. This is what I got:

The results

(produced with plot.ly)

Clearly, the figure shows an increasing number of gifts according to the good behavior score. However, my niece wasn’t satisfied by this piece of answer. She wanted to know if this increasing trend is statistically significant. Oh dear.

“Why don’t you complete your analysis with a linear Poisson regression?“, she said

The Poisson regression is commonly used to model count data including many low values (such as 0’s or 1’s). If we look at our number of Christmas Gifts variable, this seems appropriate.

I used a linear Poisson regression to model the effect of Good Behavior score on the number of gifts, and got a positive effect (slope = 0.237) which is very significant (p <0.0001).

(statistics were computed using the xlstat software).

So yes, children’s good behavior significantly boosts the number of gifts they receive on Christmas.

- That was the only serious paragraph in the article -

What happened after

My niece used her own Good Behavior Score and the Poisson model coefficients to predict the gift number she would receive this year. She concluded that she really needs to improve her score.

(clipart obtained from openclipart.org)

Why the fish clipart?