The Problem of Irreproducible Results

By Elizabeth Iorns

Cutting edge scientific research is often interdisciplinary. Geneticists team up with biostatisticians, physicists with computer scientists, and biologists with chemists. Partially because of this increasing complexity, however, the scientific literature is littered with papers that cannot be replicated, either because of the good faith efforts of scientists who fail in their attempts to be jacks-of-all-trades or because corners are cut deliberately.

We are trying to fix that.

Science and technology have always been intertwined. The first scientist to describe the cellular composition of life built his own microscope. These days, however, it is safe to say that the majority of cell biologists would not have any idea how to grind a lens or engineer a light box. Most cognitive psychologists probably have only a rudimentary understanding of the physics behind functional magnetic resonance imaging (fMRI), let alone the image processing steps.

These are common examples of the growing specialization required for the modern scientific enterprise to advance.

One positive result of this specialization is the rise of core facilities in the best universities, in which departments intent on recruiting and keeping top scientific talent provide incentives that include specialized facilities for animal breeding, microscopy, genomics, etc. Core facilities are run by dedicated professional scientists who are responsible for maintaining the highest quality and most up-to-date technologies in their designated domains.

Core facilities often have excess capacity that goes unused by researchers in their home institutions. That’s one reason we started Science Exchange. We wanted to bring the efficiencies of electronic marketplaces into the service of science by connecting investigators and service providers, matching needs and expertise regardless of geographic location. Our aim is to improve the quality and efficiency of scientific research, while reducing costs.

Additionally, Science Exchange provides a tool to address the problem of reproducibility. In our network, we have over 1,000 service providers, who work on a fee-for-service basis. What if we could harness those experts to independently replicate the most important research findings?

Earlier this summer, an advisory group of scientific leaders and I launched the Reproducibility Initiative using Science Exchange as a platform. The initiative has two tracks. First, if they have the funds to pay for the service, researchers may submit findings that they want to have independently replicated, perhaps because of its potential translational or commercial value. Where we have providers with the necessary expertise, we will randomly assign replication studies to them.

Second, as part of a broader, more inclusive survey of reproducibility, we invite authors of recently published papers to nominate their work for independent replication. Our partner, Mdndeley, will track the download and citation statistics of invited papers, and our funding partners will support the replication of a fraction of the nominated papers, effectively creating an audit mechanism. Our expectation is that papers that are nominated for replication are more likely to be reproducible and robust. In effect, we are implementing a behavioral screen for papers in which authors demonstrate full confidence.

We are working with PLOS ONE – an open-access publisher which provides articles free of charge to the public – to give authors of replicated studies the opportunity to publish their results, regardless of outcome. Publishing "negative" data, in which experiments fail to replicate, is just as important as publishing a confirmation of results. In this way, we hope to encourage and reward robust, reproducible research.

The beauty and power of using core facilities to perform independent replications of scientific studies is that their incentives are clear. They want to provide the best service possible, within the limited domain of their expertise, for a fee. And importantly, there is no conflict of interest, since they have no vested interest in the outcome of any particular research study.

Using this unique strategy, we believe our Initiative will help solve the problem of reproducibility.

Elizabeth Iorns is the co-founder and CEO of Science Exchange.
http://www.realclearscience.com/articles/2012/11/10/solving_the_problem_of_irreproducible_results_106402.html