The Establishment Defends Irreproducibility

David Randall

A recent article by Robinson Meyer in The Atlantic continues the drumbeat of opposition by the scientific establishment to the EPA’s proposed rule Strengthening Transparency in Regulatory Science. (The NAS has provided a public comment in favor of that rule.) Meyer quotes various scientists as saying that forbidding the use of un-transparent data will cripple public health regulation. In making this argument, both Meyer and the scientists he quote fail to mention several previous points of contention. Readers interested in this issue should find these points worth reviewing.

Meyer does not mention the possibility of de-identifying previous research, rendering it anonymous, and thus preserving the anonymity of participants. There is legitimate debate as to how much such research would cost: the EPA claims that it would cost more than $250 million a year—although this may be an inflated estimate, provided to deter would-be reformers. There is also the question of how effective de-anonymizing techniques can be, in an age of massive computational power.

Americans should debate the cost of anonymization, its priority within EPA spending, and how best to de-anonymize data. The NAS favors a substantial shift of resources by the EPA to anonymize data, using the best available techniques, sufficient in scale to allow for the de-identification within 10 years of the majority of research used by the EPA to inform its regulations. We urge the scientific establishment to join us in this call, which will provide the EPA properly reproducible scientific data.

Meyer also does not mention the possibility that scientists can use publicly accessible data to study public health issues—as some have already begun to do.

“This is a very highly contentious political climate, and we are taking the extra step to be as transparent as we can be,” says biostatistician Francesca Dominici of the Harvard T. H. Chan School of Public Health in Boston, who led the studies. In one, published in The New England Journal of Medicine in June 2017, she and her colleagues used publicly accessible air pollution data and records compiled by the federal government’s Medicare health insurance program to show that even modest pollution reductions could save more than 10,000 lives per year. In another, published in JAMA last December, they linked short-term exposure to air pollution levels below current limits to premature death among the elderly.

Stricter transparency regulations may annoy scientists, and add marginally greater costs to their procedure—but the contention that science will be crippled depends on the presumption that scientific ingenuity cannot find workable ways to meet strict transparency standards. Dominici’s research already provides evidence that this presumption is incorrect.

The NAS urges the scientific establishment to take account of these issues and developments as it critiques the EPA’s proposed rule. And we repeat our recommendation to the EPA that it adopt strict transparency regulations—with redoubled confidence that American scientists can produce solid research that meets these regulatory standards.

  • Share

Most Commented

September 5, 2014


Post Traumatic Slave Syndrome

"[PTSS] provides an evidence-proof explanation that lifts away moral responsibility from those engaged in self-destructive, anti-social, and criminal behavior."...

September 21, 2010


Ask a Scholar: What Does YHWH Elohim Mean?

A reader asks, "If Elohim refers to multiple 'gods,' then Yhwh Elohim really means Lord of Gods...the one of many, right?" A Hebrew expert answers....

April 9, 2018


How Many Confucius Institutes Are in the United States?

Download an updated PDF that lists all of the Confucius Institutes in the United States, along with information on who to contact if you have concerns....

Most Read