Mice, P-hacking, and More Irreproducible Science

Peter Wood

  • Article
  • September 18, 2018

Further evidence continues to accumulate about the extent of the irreproducibility crisis—the combination of flawed statistics, arbitrary research techniques, and groupthink that afflict modern scientific research. A recent article at Nature Communications, “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons,” appears to be a poster-child for the irreproducibility crisis, for it states in the discussion of its procedures that “we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable—you are supposed to determine your sample size first, not just stop collecting data at the moment that you happen to get a statistically significant result. Is this what happened? Nature claims to be committed to “substantive steps” to address the irreproducibility crisis, and we trust that the commitment also applies to Nature Communications. We have sent an inquiry to Nature Communications—our letter is at the bottom of this post.

We found out about this article from People for the Ethical Treatment of Animals (PETA), whose primary concern is that the article required painful, fatal animal research. The NAS’s fundamental commitment is rather to freedom of (scientific) inquiry. Yet we don’t relish the thought of p-hacking compounded by hacking apart mice. If animals are going to die for science, we’d rather it was for science done right.

It is possible that the articles’ authors simply explained badly what was actually a properly done scientific experiment. But we believe Nature Communications ought to clarify the facts at issue.


Dear Nature Communications,

Nature recently published “Scientific Rigour and Reproducibility” (https://www.nature.com/collections/byblhcfwhw), which stated that “There is growing alarm about results that cannot be reproduced.  Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions, and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community.” Yet you recently published “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons” by Yang Dan, et al. (Nature Communications 9, 354, https://www.nature.com/articles/s41467-017-02765-w) , which contain this statement: “For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable. Is this what happened? I urge Nature Communications to re-examine this article to see whether it meets Nature’s standards—and to take appropriate measures if it does not. I also urge Nature Communications to provide a public explanation to its readership as to how it came to publish an article that seems openly to avow p-hacking.


Respectfully yours,

Peter W. Wood


National Association of Scholars


Image: Spark // Creative Commons

  • Share

Most Commented

May 1, 2017


Binghamton University Responds to NAS’s Report

SUNY Binghamton offers critiques and corrections to Outsourced to China. ...

March 23, 2011


Looking for Answers? Ask a Scholar!

NAS is partnering with Intellectual Takeout to answer questions that call for scholarly judgment and can't be answered by Wikipedia. ...

September 5, 2014


Post Traumatic Slave Syndrome

"[PTSS] provides an evidence-proof explanation that lifts away moral responsibility from those engaged in self-destructive, anti-social, and criminal behavior."...

Most Read

May 15, 2015


Where Did We Get the Idea That Only White People Can Be Racist?

A look at the double standard that has arisen regarding racism, illustrated recently by the reaction to a black professor's biased comments on Twitter....

June 15, 2020


See Clearly, Decide Wisely, Act Justly

An Open Letter to the Converse College Community in response to its administrative decisions relating to diversity and inclusion on campus. ...

June 12, 2020


Defending Against the Jacobins

As college presidents shirk academic reform in favor of more activist programs, the question remains: Will boards of trustees walk with higher ed. leadership over the edge of the cliff?...