#phacking

"when he read about the idea of natural selection. The GRIM test, short for granularity-related inconsistency of means, is a simple way of checking whether the results of small studies of the sort beloved of psychologists (those with fewer than 100 participants) could be correct [..] To understand the GRIM test, consider an experiment in which participants were asked to assess something (someone else’s friendliness, say) on an integer scale of one to seven. The resulting paper says there were 49 participants and the mean of their assessments was 5.93. It might appear that multiplying these numbers should give an integer product—ie, a whole number—since the mean is the result of dividing one integer by another. If the product is not an integer (as in this case, where the answer is 290.57), something looks wrong."

"something that might easily have been predicted. But worryingly, the process of replication, by which published results are tested anew, is incapable of correcting the situation no matter how rigorously it is pursued. [..] The researchers’ conclusion is therefore that when the ability to publish copiously in journals determines a lab’s success, then “top-performing laboratories will always be those who are able to cut corners”—and that is regardless of the supposedly corrective process of replication. Ultimately, therefore, the way to end the proliferation of bad science is not to nag people to behave better, or even to encourage replication, but for universities and funding agencies to stop rewarding researchers who publish copiously over those who publish fewer, but perhaps higher-quality papers."

"The idea that papers are publishing false results might sound alarming but the recent crisis doesn’t mean that the entire scientific method is totally wrong. In fact, science’s focus on its own errors is a sign that researchers are on exactly the right path. [..] If you never find mistakes, or failures to reproduce in your field, you’re probably not asking the right questions. [..] there’s already widespread discussion within psychology about pre-registering trials (which would prevent researchers from shifting their methods so as to capture more eye-catching results), making data and scientific methods more open, making sample sizes larger and more representative, and promoting collaboration."

"hink of an experiment on analogy with a recipe. Suppose your friend tells you that your recipe for apple pie is a failure, the pie might have tasted great when you used the recipe to make it yourself, but when he tried it, it turned out awful. The pie does not replicate! Does this mean you should abandon the recipe? Well, maybe. But what if turns out that your friend used moldy apples, that his oven is broken, and he neglected to add the crust. The failure wasn’t due to the recipe, but to the cook. Plainly, a failure to replicate means a lot when it’s done by careful and competent experimenters, and when it’s clear that the methods are sensitive enough to find an effect if one exists. Many failures to replicate are of this sort, and these are of considerable scientific value. But I’ve read enough descriptions of failed replications to know how badly some of them are done.  I’m aware as well that some attempts at replication are done by undergraduates who have never run a study before. Such replication attempts are a great way to train students to do psychological research, but when they fail to get an effect, the response of the scientific community should be: Meh. [..] It’s important for non-scientists to have some degree of scientific literacy, beyond a passing familiarity with certain theories and discoveries. Scientific literacy requires an appreciation of how science works, and how it stands apart from other human activities. A public discussion about how scientists make mistakes and how they can work to correct them will help advance scientific understanding more generally."