#p hacking

Oh, I've heard about this guy! I think they talked about him on an episode of Freakonomics, or else somewhere on NPR.

Academic dishonesty, especially regarding scientific publications, it's something very near and dear to my heart. And the fact that this guy produced so many in easy, delightful, daytime talk show style. Scientific headlines means that I think I know the angle I would want to take with this just because somebody says that something is science, doesn't mean it is. It's not a matter of you. Can't trust the mainstream media, it's a matter of people need scientific literacy in order to be trustworthy when reporting science.

But the reason why I'm responding to this ask instead of just making a lesson plan, Which I do intend to do, is that I never ever actually took any classes on scientific methodology. All I know about p-values I learned from Khan Academy and Wikipedia.

So do you think we could collaborate?

If I draft up a lesson, could you proofread it for me to make sure that it's got all the information that you would like to see students learn?

By Jerry Adler

Pacific Standard: The Science of Society

Originally posted April 28, 2014

Here is two excerpts to a long, yet exceptional, article on research in the social sciences:

OUTRIGHT FAKERY IS CLEARLY more common in psychology and other sciences than we’d like to believe. But it may not be the biggest threat to their credibility. As the journalist Michael Kinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”

P is a central concept in statistics: It’s the mathematical factor that mediates between what happens in the laboratory and what happens in the real world. The most common form of statistical analysis proceeds by a kind of backwards logic: Technically, the researcher is trying to disprove the “null hypothesis,” the assumption that the condition under investigation actually makes no difference.

(cut)

WHILE IT IS POSSIBLE to detect suspicious patterns in scientific data from a distance, the surest way to find out whether a study’s findings are sound is to do the study all over again. The idea that experiments should be replicable, producing the same results when run under the same conditions, was identified as a defining feature of science by Roger Bacon back in the 13th century. But the replication of previously published results has rarely been a high priority for scientists, who tend to regard it as grunt work. Journal editors yawn at replications. Honors and advancement in science go to those who publish new, startling results, not to those who confirm—or disconfirm—old ones.