Congress Needs to Strengthen Scientific Reproducibility Standards

David Randall

Editor's note: This article was originally published by The Hill on December 2, 2018. 

America is suffering from a crisis of irreproducible science. In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology, but could only replicate six. That same year Janet Woodcock, director of the Center for Drug Evaluation and Research at the FDA, “estimated that as much as 75 percent of published biomarker associations are not replicable.”

The federal government bears some blame. According to a 2015 study, government funds two-thirds of preclinical research in America and half of that research is irreproducible. Of the $28 billion our country wastes each year in irreproducible preclinical research, the government share is $19 billion.

The federal government makes policy based on research that can’t reproduce. The EPA used the National Ambient Air Quality Standard (NAAQS) for fine particulate matter (pm25) to justify many costly regulations, such as the Clean Power Plan. When Dr. James Enstrom acquired the original data and conducted an independent re-analysis, he “found that there was no significant relationship between pm25 and death from all causes.”

Earlier this year the National Association of Scholars (NAS) published The Irreproducibility Crisis in Modern Science, which analyzes the effects of the improper use of statistics, arbitrary research techniques and political groupthink. Together these have produced a reproducibility crisis that afflicts a wide range of scientific disciplines.

The irreproducibility crisis spreads across a host of departments and agencies. They make regulations based on scientific research, trusting in peer review, but not requiring that the research be reproducible. Peer review has proven to be a weak protection against bad science. The recent study proclaiming — falsely — that the oceans are heating 60 percent faster than expected passed multiple layers of peer review. Just days later it was exposed as having a fundamental statistical error and the headline claim was withdrawn.

Political advocates often prefer to rely on irreproducible research, since that makes it easier to come up with a “scientific” justification for the regulations they want to impose.

Some parts of the government are now trying to restore reproducibility to science. The National Institutes of Health directs grant applicants to make reproducibility a top priority. The Environmental Protection Agency has proposed a rule to strengthen transparency in its own use of scientific research.

But there are dozens of federal agencies that are moving too slowly. The federal government needs to make a global change to its reproducibility requirements.

Congress can do something now — in the weeks before the new Democratic majority takes over the House. Democratic members of Congress have so far opposed all reproducibility reforms. But one simple change by the current reform-minded majority will strengthen the federal government’s reproducibility standards.

Congress should first state that “best available science” must be “publicly accessible research,” defined as research whose registered report (including protocols), research data, associated protocols, computer codes, data analysis scripts, recorded factual materials, statistical analyses and algorithms are archived on an online digital repository in a manner sufficient for continuing independent inspection, replication, reproduction and verification.

Congress should then require that all grant programs that fund “best available science” must fund only “publicly accessible research.”

Congress should also require that any significant regulatory action (SRA) that depends on “best available science” must also depend on “publicly accessible research.”

A congressional requirement that all federal agencies that use “best available science” rely on “publicly accessible research” will have an immediate effect throughout the tangle of federal government—but it will only affect those agencies that already rely on “best available science.” This change will be both simple and tailored.

The federal government will need to make far greater changes to solve the irreproducibility crisis. Each agency needs to take responsibility for instituting specific measures to improve reproducibility standards in its own grants and regulatory processes. Some agencies, for example, might need to require that SRAs rely on science that isn’t just publicly accessible, but which has actually been reproduced. It isn’t unreasonable to ask that a major piece of regulation depend upon two independent works of scientific research, not just one.

But Congress can make a great step forward now. If it does, it will champion America’s longstanding commitment to constantly improve its scientific standards. It will also act as a responsible steward of American taxpayers’ money and a prudent governor of our regulatory state.

David Randall is director of research at the National Association of Scholars.

  • Share

Most Commented

January 9, 2023


NAS Celebrates the Nomination of Reform-Minded Trustees to the New College of Florida Board

The National Association of Scholars is delighted with Governor Ron DeSantis’ nomination of six education reformers to the Board of Trustees of the New College of Florida....

December 7, 2022


New Study Tracks Rise of DEI in STEM Departments, Associations, Grants, and Literature

A new study published today by the National Association of Scholars, Ideological Intensification, offers an in-depth quantitative analysis of just how far DEI has advanced into STEM fields....

January 5, 2023


NAS Condemns the Attacks against Jordan Peterson

The National Association of Scholars condemns the unrelenting illiberal attacks being levied against Dr. Peterson and against anyone who dares push back against the enemies of intellectual f......

Most Read

May 15, 2015


Where Did We Get the Idea That Only White People Can Be Racist?

A look at the double standard that has arisen regarding racism, illustrated recently by the reaction to a black professor's biased comments on Twitter....

October 12, 2010


Ask a Scholar: What is the True Definition of Latino?

What does it mean to be Latino? Are only Latin American people Latino, or does the term apply to anyone whose language derived from Latin?...

September 19, 2022


How Many Confucius Institutes Are in the United States?

UPDATED: We're keeping track of all Confucius Institutes in the United States, including those that remain open, those that closed, and those that have announced their closing....