#reproducibility project

This is a huge problem. We don’t know what we know without replication! (and well designed studies from independent researchers, i.e. those not funded by those with conflicts of interest!) And even then, biology is COMPLICATED!

Rigorous replication effort succeeds for just two of five cancer papers

The first results of a high-profile effort to replicate influential papers in cancer biology are roiling the biomedical community. Of the five studies the project has tackled so far, some involving experimental treatments already in clinical trials, only two could be repeated; one could not, and technical problems stymied the remaining two replication efforts.

Some scientists say these early findings from the Reproducibility Project: Cancer Biology, which appear tomorrow in eLife, bolster concerns that too many basic biomedical studies don’t hold up in other labs. “The composite picture is, there is a reproducibility problem,” says epidemiologist John Ioannidis of Stanford University in Palo Alto, California, an adviser to the project whose attention-getting analyses have argued that biomedical research suffers from systemic flaws.

Gray Matter: Why Do So Many Studies Fail to Replicate?

[ad_1] Photo Credit Oscar Bolton Green LAST year, a colleague asked me if I would send her the materials needed to try to replicate one of my published papers — that is, to rerun the study to see if its findings held up. “I’m not trying to attack you or anything,” she added apologetically. I laughed. To a scientist, replication is like breathing. Successful replications strengthen…

It’s been called the “decline effect,” “the proteus phenomenon,” and “the reproducibility crisis”: the startling realization that a lot of seemingly solid scientific research doesn’t pan out under repeated testing. The latest blow to scientific confidence comes from the Reproducibility Project, which attempted to replicate 100 published psychology studies and found that, when the experiments were repeated, half or more failed to uphold the original findings. So is it time to start doubting the credibility of research in general? Stanford University psychologist and Reproducibility Project participant Mike Frank joined us to explain what the results really mean, misconceptions about statistical rigor in science, the various ways experimenters blunder and sometimes delude themselves, and the gradual, cumulative nature of scientific progress. 

Click the play arrow above to hear the interview, or the download icon on the upper right to get your own mp3. Click the share icon (the box with arrow) to embed the interview in a tweet, Facebook post, etc.

One of the bedrock assumptions of science is that for a study's results to be valid, other researchers should be able to reproduce the study and reach the same conclusions. The ability to successfully reproduce a study and find the same results is, as much as anything, how we know that its findings are true, rather than a one-off result.

This seems obvious, but in practice, a lot more work goes into original studies designed to create interesting conclusions than into the rather less interesting work of reproducing studies that have already been done to see whether their results hold up.

That's why efforts like the Reproducibility Project, which attempted to retest findings from 100 studies in three top-tier psychology journals, are so important. As it turns out, findings from the majority of the studies the project attempted to redo could not be reproduced. The New York Times reports on the new study's findings:

Now, a painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.

This week, we are interviewing Mallory Kidwell, one of our Metascience Project Coordinators! Lots of good insights into what she does here at the Center for Open Science!

Name: Mallory Kidwell

Position: Metascience Project Coordinator

Hometown: Columbus, Ohio

College/University: Wake Forest University ’14

Major/Study/Scientific Interest: Psychology, with research interests in post-traumatic growth and anxiety.

Do you have a fun fact about yourself? As an undergraduate, I double majored in English – not something many people expect!

What does Open Science mean to you?  Fundamentally, Open Science means collaboration and communication – both with close colleagues and outside researchers. The opportunities and outlets for communication have increased exponentially in the past few decades; our standards and tools to organize and disseminate research effectively should increase accordingly.

What drew you to COS? I was a part of several great psychology labs during my undergraduate studies, but I saw room for organizational improvement. I was drawn to COS by its unwavering commitment to a mission that resonated with my experiences: a call for increased transparency in research. Openness makes it easier for scientists to document their research process and share it with their colleagues.

What are you working on right now? As a Metascience Project Coordinator, I have been overseeing the Reproducibility Project: Psychology (RP:P). The aim of this project is to investigate factors that influence reproducibility, with the goal of finding what practices researchers can implement to increase the rate of reproducibility.

Describe some of your day-to-day duties and responsibilities. I spend the majority of my time communicating with the research teams that have volunteered to conduct a replication for the RP:P. Depending how the study is progressing, this can sometimes require a lot of problem solving to ensure each replication mirrors its original study as closely as possible. 

My activities include (but are never limited to) reviewing replication protocols, drafting grant proposals, locating or recreating original stimuli, communicating with original authors, and checking in with replication teams to track their progress.

What’s it like living in Charlottesville, VA: Charlottesville is an incredibly vibrant community. There is so much talent and innovation all around! As a recent transplant from North Carolina, I have appreciated the welcoming atmosphere and been impressed with how much there is to offer.