Mice, P-hacking, and More Irreproducible Science

Peter Wood

Further evidence continues to accumulate about the extent of the irreproducibility crisis—the combination of flawed statistics, arbitrary research techniques, and groupthink that afflict modern scientific research. A recent article at Nature Communications, “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons,” appears to be a poster-child for the irreproducibility crisis, for it states in the discussion of its procedures that “we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable—you are supposed to determine your sample size first, not just stop collecting data at the moment that you happen to get a statistically significant result. Is this what happened? Nature claims to be committed to “substantive steps” to address the irreproducibility crisis, and we trust that the commitment also applies to Nature Communications. We have sent an inquiry to Nature Communications—our letter is at the bottom of this post.

We found out about this article from People for the Ethical Treatment of Animals (PETA), whose primary concern is that the article required painful, fatal animal research. The NAS’s fundamental commitment is rather to freedom of (scientific) inquiry. Yet we don’t relish the thought of p-hacking compounded by hacking apart mice. If animals are going to die for science, we’d rather it was for science done right.

It is possible that the articles’ authors simply explained badly what was actually a properly done scientific experiment. But we believe Nature Communications ought to clarify the facts at issue.

***

Dear Nature Communications,

Nature recently published “Scientific Rigour and Reproducibility” (https://www.nature.com/collections/byblhcfwhw), which stated that “There is growing alarm about results that cannot be reproduced.  Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions, and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community.” Yet you recently published “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons” by Yang Dan, et al. (Nature Communications 9, 354, https://www.nature.com/articles/s41467-017-02765-w) , which contain this statement: “For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable. Is this what happened? I urge Nature Communications to re-examine this article to see whether it meets Nature’s standards—and to take appropriate measures if it does not. I also urge Nature Communications to provide a public explanation to its readership as to how it came to publish an article that seems openly to avow p-hacking.

 

Respectfully yours,

Peter W. Wood

President

National Association of Scholars

 


Image: Spark // Creative Commons

  • Share

Most Commented

May 7, 2024

1.

Creating Students, Not Activists

The mobs desecrating the American flag, smashing windows, chanting genocidal slogans—this always was the end game of the advocates of the right to protest, action civics, student activ......

March 9, 2024

2.

A Portrait of Claireve Grandjouan

Claireve Grandjouan, when I knew her, was Head of the Classics Department at Hunter College, and that year gave a three-hour Friday evening class in Egyptian archaeology....

April 20, 2024

3.

The Academic's Roadmap

By all means, pursue your noble dream of improving the condition of humanity through your research and teaching. Could I do it all again, I would, but I would do things very differently....

Most Read

May 15, 2015

1.

Where Did We Get the Idea That Only White People Can Be Racist?

A look at the double standard that has arisen regarding racism, illustrated recently by the reaction to a black professor's biased comments on Twitter....

October 12, 2010

2.

Ask a Scholar: What is the True Definition of Latino?

What does it mean to be Latino? Are only Latin American people Latino, or does the term apply to anyone whose language derived from Latin?...

September 21, 2010

3.

Ask a Scholar: What Does YHWH Elohim Mean?

A reader asks, "If Elohim refers to multiple 'gods,' then Yhwh Elohim really means Lord of Gods...the one of many, right?" A Hebrew expert answers....