Congress Needs to Strengthen Scientific Reproducibility Standards

David Randall

Editor's note: This article was originally published by The Hill on December 2, 2018. 

America is suffering from a crisis of irreproducible science. In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology, but could only replicate six. That same year Janet Woodcock, director of the Center for Drug Evaluation and Research at the FDA, “estimated that as much as 75 percent of published biomarker associations are not replicable.”

The federal government bears some blame. According to a 2015 study, government funds two-thirds of preclinical research in America and half of that research is irreproducible. Of the $28 billion our country wastes each year in irreproducible preclinical research, the government share is $19 billion.

The federal government makes policy based on research that can’t reproduce. The EPA used the National Ambient Air Quality Standard (NAAQS) for fine particulate matter (pm25) to justify many costly regulations, such as the Clean Power Plan. When Dr. James Enstrom acquired the original data and conducted an independent re-analysis, he “found that there was no significant relationship between pm25 and death from all causes.”

Earlier this year the National Association of Scholars (NAS) published The Irreproducibility Crisis in Modern Science, which analyzes the effects of the improper use of statistics, arbitrary research techniques and political groupthink. Together these have produced a reproducibility crisis that afflicts a wide range of scientific disciplines.

The irreproducibility crisis spreads across a host of departments and agencies. They make regulations based on scientific research, trusting in peer review, but not requiring that the research be reproducible. Peer review has proven to be a weak protection against bad science. The recent study proclaiming — falsely — that the oceans are heating 60 percent faster than expected passed multiple layers of peer review. Just days later it was exposed as having a fundamental statistical error and the headline claim was withdrawn.

Political advocates often prefer to rely on irreproducible research, since that makes it easier to come up with a “scientific” justification for the regulations they want to impose.

Some parts of the government are now trying to restore reproducibility to science. The National Institutes of Health directs grant applicants to make reproducibility a top priority. The Environmental Protection Agency has proposed a rule to strengthen transparency in its own use of scientific research.

But there are dozens of federal agencies that are moving too slowly. The federal government needs to make a global change to its reproducibility requirements.

Congress can do something now — in the weeks before the new Democratic majority takes over the House. Democratic members of Congress have so far opposed all reproducibility reforms. But one simple change by the current reform-minded majority will strengthen the federal government’s reproducibility standards.

Congress should first state that “best available science” must be “publicly accessible research,” defined as research whose registered report (including protocols), research data, associated protocols, computer codes, data analysis scripts, recorded factual materials, statistical analyses and algorithms are archived on an online digital repository in a manner sufficient for continuing independent inspection, replication, reproduction and verification.

Congress should then require that all grant programs that fund “best available science” must fund only “publicly accessible research.”

Congress should also require that any significant regulatory action (SRA) that depends on “best available science” must also depend on “publicly accessible research.”

A congressional requirement that all federal agencies that use “best available science” rely on “publicly accessible research” will have an immediate effect throughout the tangle of federal government—but it will only affect those agencies that already rely on “best available science.” This change will be both simple and tailored.

The federal government will need to make far greater changes to solve the irreproducibility crisis. Each agency needs to take responsibility for instituting specific measures to improve reproducibility standards in its own grants and regulatory processes. Some agencies, for example, might need to require that SRAs rely on science that isn’t just publicly accessible, but which has actually been reproduced. It isn’t unreasonable to ask that a major piece of regulation depend upon two independent works of scientific research, not just one.

But Congress can make a great step forward now. If it does, it will champion America’s longstanding commitment to constantly improve its scientific standards. It will also act as a responsible steward of American taxpayers’ money and a prudent governor of our regulatory state.

David Randall is director of research at the National Association of Scholars.

  • Share

Most Commented

June 13, 2022


Regime Change: Repelling the DEI Assault on Higher Education

The last few decades have seen a total transformation of higher education's mission in American society. What was once an institution that encouraged innovation, built character, and sou......

May 12, 2022


Mornin’ Ralph, Mornin’ Sam in Anthropology Today

Professor Lowrey recounts her latest encounter with academic cancel culture, this time with an acceptance-turned-rejection at Anthropology Today....

May 31, 2022


Can America’s Colleges and Universities Be Saved?

We urge American citizens and policymakers to reform our higher education system while there is still time....

Most Read

June 13, 2022


Regime Change: Repelling the DEI Assault on Higher Education

The last few decades have seen a total transformation of higher education's mission in American society. What was once an institution that encouraged innovation, built character, and sou......

May 15, 2015


Where Did We Get the Idea That Only White People Can Be Racist?

A look at the double standard that has arisen regarding racism, illustrated recently by the reaction to a black professor's biased comments on Twitter....

June 21, 2022


Press Release: Confucius Institutes Rebrand to Circumvent U.S. Policy, Report Finds

A new report from the National Association of Scholars finds that colleges and universities continue to host Chinese-funded programs similar to Confucius Institutes under new names......