Comment on The Irreproducibility Crisis: Daniele Fanelli

Apr 19, 2018 |  NAS

Font Size  


Comment on The Irreproducibility Crisis: Daniele Fanelli

Apr 19, 2018 | 


Dr. Daniele Fanelli, Fellow in Quantitative Methodology, Department of Methodology, London School of Economics and Political Science, has commented on The Irreproducibility Crisis at his blog--partly in support and partly in dissent. Dr. Fanelli has requested that we not reproduce his text on our website, since he may revise it in future. We provide this weblink, so you may read his commentary there.

Dr. Fanelli takes a more cautious view about the prevalence of irreproducible research, and thinks a number of our policy suggestions would be counterproductive.

The National Association of Scholars is grateful to Dr. Fanelli for providing this comment. We urge the public and policymakers to look at his critique of our position, so that they may know the range of professional opinion about the irreproducibility crisis. We are happy where he agrees with our recommendations--and we are respectfully listening to his dissent, which we will incorporate as we continue our work on the irreproducibility crisis.

Henry H. Bauer

| April 22, 2018 - 8:31 AM


The “crisis of irreproducibility” is only a symptom, The central crisis concerns the role of science in society, namely, that what official sources are now disseminating as supposedly reliable is actually untrustworthy in many ways on many matters — about health risks and drugs, for instance, or global warming.
There are two interlocking facets of this crisis. One is the actual nature of contemporary scientific activity, which conduces to untrustworthiness. The other is society’s ignorance of and misunderstandings about the nature of contemporary scientific activity, related to pervasive ignorance of and misunderstandings about the history of science. A further complication is that fundamental changes have taken place in science during the era of modern science (usually dated from about the 17th century), and society as a whole is ignorant of that.
In some respects, society’s views about science, the conventional contemporary wisdom about science, would not be so inaccurate about early modern science, but they are badly misleading about contemporary science. In a seriously oversimplified nutshell:
The circumstances of scientific activity have changed, from about pre-WWII to nowadays, from a cottage industry of voluntarily cooperating, independent, largely disinterested ivory-tower intellectual entrepreneurs in which science was free to do its own thing, namely the unfettered seeking of truth about the natural world, to a bureaucratic corporate-industry-government behemoth in which science has been pervasively co-opted by outside interests and is not free to do its own thing because of the pervasive conflicts of interest. Influences and interests outside the scientific community now control choices of research projects and decisions of what to publish and what not to make public.
The enormous expansion of support for science following WWII stimulated the training of increasing numbers of people for scientific work, and the consequent demands for resources for research have far outstripped what is available, bringing cutthroat competition and the associated evils of dishonesty including an increased frequency of outright fraud; illustrated by the need for an Office of Research Integrity and related regulations and activities.
These details about how science has changed explain why this crisis is with us now.
The crisis with respect to the hard sciences has come because of these changes in the character of scientific activity, of which the irreproducibility crisis is only a symptomatic part. For example, that the unproven hypothesis of human-caused climate change is so widely believed and acted on is not owing to misused statistics or irreproducibility: it is owing to the outdated reliance on “science” as reliable, the taking as automatically true what official sources say about science, the ignorant assumption that computer models can substitute for reality; and it illustrates President Eisenhower’s prescient warning, in his farewell address, that “public policy could itself become the captive of a scientific-technological elite”.
Society needs an independent, disinterested procedure for adjudicating scientific claims. The best candidate appears to be a Science Court, proposed more than half a century ago and discussed on a number of occasions since then.
For example: On the issue of whether human generation of carbon dioxide is the proven prime cause of global warming and climate change, a Science Court could require proponents and dissenters to prepare and submit their cases and to defend them in open court, under cross-examination. The Court could bring other experts as well, and the presiding judge or panel would have the services of expert consultants as appropriate. Fuller details about how a Science Court might be organized are set out in the mentioned book, Science Is Not What You Think: How It Has Changed, Why We Can’t Trust It, How It Can Be Fixed (McFarland 2017)
As Randall and Welser acknowledge, “At the highest level, however, the crisis of reproducibility also derives from science’s professional culture…. The two most dangerous aspects of this professional culture are the premium on positive results and groupthink… The gravest casualty of all is the authority that science ought to have with the public, but which it begins to forfeit when it no longer produces reliable knowledge”.
The recommendations included at the end of THE IRREPRODUCIBILITY CRISIS OF MODERN SCIENCE are unexceptionable in terms of what should happen, what should be done by a range of actors; but there is no conceivable way to ensure that those actors will follow those excellent recommendations. The aim that public policy be consonant with the best evidence could be met, however, by the long-ago-proposed mechanism of a Science Court.


Stuart Hurlbert

| April 23, 2018 - 8:30 PM

NAS has published some very good articles both online and in its journal, Academic Questions. But this report does not fall in that category.

A thorough appraisal of the NAS report has been provided by Daniele Fanelli, April 19, 2018, on his blog at . Perhaps because he was invited to do so, Fanelli was very gentle in his wording. But he basically argued against most of the major claims and and recommendations of the report, as I think most A-grade statisticians would.

In acknowledging Fanelli’s criticisms, NAS makes light of this fact, and then adds:

“We urge the public and policymakers to look at his critique of our position, so that they may know the range of professional opinion about the irreproducibility crisis.”

But the report authors’  opinions were not “professional” as they were historians without any credentials in the technical subject matter. Nor was the report subject to peer review by experts in the subject matter,  the standard process for getting independent “professional opinion” prior to publishing scholarly works.

Now it’s “water over the dam” time. The damage done to NAS’s reputation as intelligent critic of the foibles of academia and the scientific community has been seriously tarnished.

Stuart Hurlbert

| April 23, 2018 - 8:40 PM

In the preceding, my final sentence should be corrected to read:

“The damage done to NAS’s reputation as intelligent critic of the foibles of academia and the scientific community has been GREAT”

Stuart Hurlbert

| April 23, 2018 - 9:11 PM

Hugh Bauer, if you had much knowledge of the literature of statistics and statistical criticism of the last half century, I’m sure you would want to retract much of what you say here. I’ve publlished extensively in this area. And all my papers, unlike the NAS report, underwent very tough peer review.

A Science Court does not sound very feasible. If it were to be run by competent scientists in the same discipline or research area under examination, wouldn’t their time be better allocated to more rigorous peer review and editorial review for the journals or better statistical curricula in their universities?  The real source of our problems is too many people trying to publish too much and a great shortage in all fields of truly knowledgeable and competent reviewers and editors.


| April 26, 2018 - 6:45 PM

“It is well understood, but too hastily forgotten, that research results may not replicate for reasons that have nothing to do with flaws in their methodology or with the low scientific integrity of their authors. Reality can be messy and complex, and studies that try to tackle complex phenomena (which is to say most social and biological studies) are bound to yield evidence that is incomplete, erratic, sometimes contradictory and endlessly open to revision and refinement.”

Studies of that try to tackle complex phenomena sometimes do, *and sometimes do not* yield evidence that is incomplete, erratic, sometimes contradictory and endlessly open to revision and refinement. Arguably one that does has not *succeeded* in tackling the complex phenomenon. Some phenomena are intractable. Also, some scientists are better than others.

What civic justification can be given for government restriction of human conduct on the basis of evidence that is “incomplete, erratic, sometimes contradictory and endlessly open to revision and refinement”?