Fixing Science: Practical Solutions for the Irreproducibility Crisis
When: February 7 and 8
Where: Independent Institute | 100 Swan Way, Oakland, California, 94621‑1428
Theme
Many headline scientific findings in recent years have turned out to be false. They can’t be reproduced—and if you can’t reproduce a result, it isn’t science. The headlines are just the tip of the iceberg. A huge amount of ordinary scientific findings published in peer-reviewed journals don’t replicate. Something has gone terribly wrong in contemporary scientific procedures.
Scientists’ failure is called the Irreproducibility Crisis. It is the result of improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results. By some estimates, half of recent scientific research could be irreproducible.
“Fixing Science” brings together scholars from several disciplines to talk about practical ways to fix how scientists work. Panelists will suggest how research, scientists, academics, government officials, and philanthropists should reform scientific research. Special panels will describe how the irreproducibility crisis affects specific disciplines, including economics and climate science. Other panels will examine topics such as groupthink and falsifiability. Throughout, “Fixing Science” will focus on which reforms have worked so far, to help us craft and propose initiatives going forward. The conference will help advance practical solutions to the Irreproducibility Crisis by scientists, civil society, and lawmakers.
Speakers:
Louis Anthony Cox: Professor, Department of Biostatistics and Informatics, University of Colorado
Tim Edgell: Principal, Environmental Services, Stantec, Inc.
James Enstrom: Retired UCLA Research Professor and President, Scientific Integrity Institute
Daniele Fanelli: Fellow in Quantitative Methodology, Department of Methodology, London School of Economics and Political Science
Lee Jussim: Chair, Psychology Department, Rutgers University
Yuri Lazebnik: CSO, Scite, Inc.
David M. Levy: Professor, Economics Department, George Mason University
Deborah Mayo: Professor Emerita of Philosophy, Virginia Tech
Patrick J. Michaels: Senior Fellow, Competitive Enterprise Institute
David Randall: Director of Research, National Association of Scholars
Mark Regnerus: Professor, Sociology Department, University of Texas at Austin
Nathan A. Schachtman: Of Counsel to Ulmer & Berne LLP, and Lecturer in Law at the Columbia School of Law
Michael Shermer: Presidential Fellow, Chapman University; Host of the Science Salon Podcast; Founding Publisher of Skeptic magazine
Barry Smith: SUNY Distinguished Professor of Philosophy and Julian Park Chair, Philosophy Department, University at Buffalo
David Trafimow: Professor, Psychology Department, New Mexico State University
Anastasios Tsonis: Distinguished Professor, Department of Mathematical Sciences, Atmospheric Sciences Group, University of Wisconsin Milwaukee; and Adjunct Research Scientist, Hydrologic Research Center, San Diego, California
Richard K. Vedder: Member of the Board of Directors, National Association of Scholars; Senior Fellow, Independent Institute; Distinguished Emeritus Professor of Economics, Ohio University
Ronald L. Wasserstein: Executive Director, American Statistical Association
Stanley Young: CEO of CGStat and Research Fellow, National Association of Scholars
Discounts and Codes
Until December 31, 2019, all tickets will remain at early-bird discount pricing for students and general registration. After that date, ticket prices will increase.
Members of the National Association of Scholars will receive a discount code by email or may write to [email protected]
Members of the Independent Institute will receive discounts from the Independent Institute.
Accommodations
The Independent Institute is located right next to the Oakland Airport; there are many hotels located very near to the Institute. We recommend the Springhill Suites by Marriott. Hotels on the same block as the Independent Institute include the Hilton Oakland Airport and the Holiday Inn Hotel & Suites Oakland – Airport. The Econo Lodge Inn & Suites Oakland Airport, the Oakland Airport Executive Hotel, the Holiday Inn Express & Suites Oakland-Airport, the Courtyard by Marriott Oakland Airport, the Best Western Plus Airport Inn & Suites, Motel 6 Oakland Airport, Quality Inn Oakland Airport, and La Quinta Inn & Suites Oakland Airport Coliseum are all a short distance from the Independent Institute.
Tentative Conference Schedule
Friday, February 7, 2020
2:30 pm Registration open
3:00 pm Welcome and Opening Remarks
David Theroux, Founder, and President, Independent Institute
3:10 pm Welcome and Opening Remarks
Peter Wood, President, NAS
3:30 pm Opening Address
Nathan A. Schachtman, Of Counsel to Ulmer & Berne LLP, and Lecturer in Law at the Columbia School of Law
"Not Just an Academic Dispute: Irreproducibility of Scientific Evidence Renders Legal Judgments Unsafe"
Irreproducibility in science confounds the already strained capacity of legal actors to interpret scientific evidence. Historically, the law has not respected the requirements of valid scientific inference. Starting with its Daubert decision, the Supreme Court conditioned admissibility of expert testimony on reliability and validity. Federal judges are now “gatekeepers” of scientific evidence, which has improved scientific fact-finding in court. The results, however, are uneven; judges are routinely misled by unreliable and invalid methods, which render legal judgments “unsafe.” I will discuss recent examples of irreproducible and irreplicable scientific studies, and their untoward effect on legal proceedings.
4:15 pm Break
4:30 pm Economics and the Irreproducibility Crisis
Moderator: Lee Jussim
David M. Levy, Professor, Economics Department, George Mason University
“Avoiding Investigator Bias: How to Make Economic Expertise Reproducible”
The first step to recognize that experts have the same motivational package as everyone else and the foresight of having one’s work examined by hostile eyes changes one’s incentives to do replicable work. Sandra Peart and I have proposed that those regulatory bodies adopt and expand the adversarial procedures employed in civil litigation as a way to make unbiased procedures incentive compatible. The paper reviews the consequences of the foresight of discovery and replication on the incentives of experts. The context of democracy as a discovery procedure will be considered.
Richard K. Vedder, Member of the Board of Directors, National Association of Scholars; Senior Fellow, Independent Insitute; Distinguished Emeritus Professor of Economics, Ohio University
“The Economic Origins of the Irreproducibility Crisis”
Economic incentives induce professors to churn out scientific papers. Lucrative federal funding is tied to prior research. Highly productive professors, as measured by published research, are financially rewarded more than those whose "comparative advantage" is in teaching. Large overhead provisions on federal grants induce universities to promote grant proposals, even kicking back some overhead funds to principal investigators. Quality control and replication of research findings is, relatively speaking, neglected. Financial incentives induce scholars often to seek quantity over quality, and, most egregiously, to ignore fields of scholarly inquiry not significantly funded by the federal government.
5:30: pm Falsifiability and the Irreproducibility Crisis
Moderator: Elliot D. Bloom
Brian D. Earp, Associate Director, Yale-Hastings Program in Ethics and Health Policy
"Replication and Falsification and the Philosophy of Science"
This talk will put the reproducibility debate in psychology into a historical and philosophical context. What do (apparently) failed replications have to do with falsification? And how can replication studies be made more informative?
Anastasios Tsonis, Distinguished Professor, Department of Mathematical Sciences, Atmospheric Sciences Group, University of Wisconsin Milwaukee; and Adjunct Research Scientist, Hydrologic Research Center, San Diego, California
“Model Falsifiability and Climate Slow Modes”
The most advanced climate models are actually modified meteorological models attempting to capture climate in meteorological terms. This seems a straightforward matter of raw computing power applied to large enough sources of current data. Some believe that models have succeeded in capturing climate in this manner. But have they? This paper outlines difficulties with this picture that derive from the finite representation of our computers, and the fundamental unavailability of future data instead. It suggests that alternative windows onto the multi-decadal timescales are necessary in order to overcome the issues raised for practical problems of prediction.
6:30 pm Break for Dinner
7:00 pm Dinner Address
Barry Smith, SUNY Distinguished Professor of Philosophy and Julian Park Chair, Philosophy Department, University at Buffalo
“The Irreproducibility Crisis and the Lehman Crash”
Multiple initiatives have been launched to address the replication crisis in social sciences and elsewhere. I will summarize their proposed strategies and identify some obstacles to their widespread adoption. Pre-registration is frequently regarded as a silver bullet, but I will show that pre-registration will fail so long as scientists are allowed to use their own terminology to describe the experiments they plan to perform and the data they plan to collect. I will conclude by pointing to an analogous failure in the domain of finance, and describe attempts there to avoid repetition of financial crises such as the Lehman crash.
Saturday, February 8, 2020
8:30 am Registration and breakfast
9:00 am Morning Address
Daniele Fanelli, Fellow in Quantitative Methodology, Department of Methodology, London School of Economics and Political Science
"Reproducibility Reforms if there is no Irreproducibility Crisis”
Science is not, as a whole, in a crisis, but it is undeniably facing challenges in how research is conducted, interpreted and communicated. Some of the challenges are new, others have been known for a long time but are hard to overcome. These issues are so insidious and subtle that they affect meta-research itself. Indeed, they underlie much of the erroneous perception that there is a “reproducibility” crisis. This talk will illustrate some of these problems using examples from meta-science and the literature on reproducibility, and it will discuss possible directions of progress.
9:45 am Break
10:00 am Environmental Sciences and the Irreproducibility Crisis
Moderator: Anastasios Tsonis
Tim Edgell, Principal, Environmental Services, Stantec, Inc.
“Stylistic Bias, Selective Reporting, and Climate Science”
Is science free from bias, personal beliefs, and sensationalism? Absolutely not. Dr. Edgell investigated patterns of bias in climate change science, a field whose principal findings are described as both a hoax and the most important scientific truth of our time. Ironically, in revealing biases about how scientists write and publish about climate change, we gain confidence in the underlying science and leading theories. This talk will remind us that both scientists and the consumers of science have a responsibility to seek truth behind the headlines. After all, science is a human construct, driven by the humanistic propensity for storytelling and reinforcing beliefs.
Patrick J. Michaels, Senior Fellow, Competitive Enterprise Institute
“Biased Climate Science”
The interaction of public choice, monopolistic funding, and skewed professional incentives with the power politics of environmental policy has resulted in a demonstrably biased canon of knowledge in the domain of climate science. The only quantitative forecasting tools that its practitioners have are undergoing a systematic failure, and yet the community neither admits to the problem nor does anything substantive about it. When the resultant scientific literature is summarized in policy-relevant documents, such as IPCC reviews or U.S. “National Assessments”, if they are accurate, they will reflect a similar bias, resulting in unwise, wasteful and harmful policies.
11:00 am Break
11:15 am Environmental Epidemiology and the Irreproducibility Crisis
Moderator: David Randall
S. Stanley Young, Chief Executive Officer, CGStat, LLC
“How to Evaluate Meta-Analyses of Environmental Epidemiology”
Well over 80% of claims made in observational studies fail to replicate. Some think meta-analysis studies using results from multiple observational studies are suitable for decision making. About 5,000 meta-analysis studies are published each year. I evaluate the reliability of meta-analyses in environmental epidemiology. I evaluate the base studies for p-hacking, trying this and that analysis to get a p-value less than 0.05. Plotting a p-value from each base study, ranked, against the integers, often shows a pattern consistent with p-hacking. I show that the base studies and the resulting meta-analysis are frequently unreliable, and have no sound statistical basis.
James E. Enstrom, Retired UCLA Research Professor and President, Scientific Integrity Institute
“Reproducibility is Essential to Combating Environmental Lysenkoism”
My 2017 independent reanalysis of the ACS Cancer Prevention Study data found no robust relationship between fine particulate matter (PM2.5) and total mortality, contrary to the seminal 1995 Pope analysis that justified the 1997 EPA PM2.5 National Ambient Air Quality Standard. My reanalysis increases the evidence that inhaling about 100 micrograms of invisible fine particles per day does not cause premature deaths. In addition, it demonstrates the importance of access to underlying data to in order to strengthen transparency and reproducibility in regulatory science. Finally, it increases efforts to combat the current form of Lysenkoism that has corrupted environmental epidemiology.
12:15 pm Luncheon
12:45 pm Luncheon Address
Louis Anthony Cox, Professor, Department of Biostatistics and Informatics, University of Colorado
“Causality, reproducibility, and scientific generalization in public health”
Irreproducibility threatens the trustworthiness of public health risk assessments informing regulation, litigation, and public policy. So does too-easy reproducibility of results that are untrue, irrelevant, or misleading. Announced results can becomes anchors for subsequent modeling and confirmation biases. If a research group announces a “link,” between an exposure and a feared health effect, others may confirm it even if exposure is not causally related to risk. Modern causal analysis helps assure that announced results reflect genuine causal relationships, useful for policy making. This talk describes how, and gives several examples from public health.
1:30 pm Break
1:45 pm Groupthink and the Irreproducibility Crisis
Moderator: Nathan A. Schachtman
Lee Jussim, Chair, Psychology Department, Rutgers University
“Intellectual Diversity Limits Groupthink in Scientific Psychology”
Groupthink occurs when there are social pressures for all to say they think, alike. An obvious remedy is embrace of intellectual diversity, the opposite of groupthink. Why is intellectual diversity crucial to a strong psychological science? If science is based on facts, the need for intellectual diversity seems moot. However, the pursuit of scientific facts is plagued by uncertainty at every turn, and intellectual diversity is crucial to resolving that uncertainty. I highlight four separate dimensions of “intellectual diversity”: skepticism, theoretical diversity, demographic diversity, and political diversity, and show how embrace of each is likely to improve psychological science.
Mark Regnerus, Professor, Sociology Department, the University of Texas at Austin
The crisis in reproducibility has reached sociology, insofar as its practitioners conduct replicable forms of research. While there have been calls for greater openness and data transparency, this move is hamstrung by the wide use of proprietary data collected by scholars themselves. A larger problem which data transparency cannot solve, however, is the disregard for creative theory-building in sociology, as well as the longstanding disinterest in studying processes in favor of documenting inequalities. Unlike replicability concerns, these latter challenges are rooted in a tendency toward groupthink and the application of political ideals—such as justice, liberation, and diversity—to research methods.
Diego A. Reinero, Doctoral Candidate, Psychology Department, New York University
“Understanding the Scope and Impact of Political Bias in Psychological Science”
Psychology is a field predominantly composed of liberals, and critics have argued that this liberal overrepresentation may reduce the robustness of research by embedding liberal values into the research and peer-review process. While complete value-free science is elusive, the scope of claims surrounding political bias in science warrant empirical investigation. In this talk I’ll discuss whether the ideological slant of psychology research is associated with scientific replicability, robustness, citations, or online attention. I conclude by advocating for a shift in focus from the politics of scientists to their research practices, and for adversarial collaborations and turnabout tests to guard against groupthink.
3:00 pm Break
3:15 pm The Road to Reform
Moderator: Richard K. Vedder
Yuri Lazebnik, CSO, Scite, Inc.
“Helping Science Recover: A Business Approach”
The irreproducibility crisis is a symptom of the malaise that is also manifested by the slowing stream of new drugs, the expansion of administration, neglected basic research, the lost sense of purpose and motivation, the flight of talent, and a numbing hypercompetition. I would like to suggest that this malaise results from a synergy of two causes: adopting a conceptual foundation that clashes with the nature of scientific discovery and the motivation of creative individuals, and an exponential increase in the information generated by scientific research. I am going to discuss how our company, Scite, Inc., helps to solve these problems.
David Trafimow, Professor, Psychology Department, New Mexico State University
“What Journals Can Do To Fix The Irreproducibility Crisis”
A successful replication is typically characterized by a statistically significant finding in the original study, and in the replication study, in the same direction. However, because significance testing is invalid, this characterization of a successful replication is similarly invalid. Journals should use a different conceptualization, whereby a successful replication entails a strong probability that the sample statistics of interest are close to their corresponding population parameters, in both original and replication studies. The a priori procedure enables researchers to use this alternative conceptualization, and journal editors could insist on it.
David Randall, Director of Research, National Association of Scholars
“Regulatory Science and the Irreproducibility Crisis”
Regulatory science is distinct from research science, and should not be held to identically strict standards. Nevertheless, the irreproducibility crisis suggests that we must also reform how the federal government funds science and uses science. Precisely because government can no longer use statistical significance as a “master protocol” to justify regulations, government should require 1) fully reproducible scientific procedures and data; and 2) new limits and transparency on scientific experts’ discretionary expertise. Government should also generously fund anonymization and replication of key data sets, to ensure regulatory continuity and consensus support from the scientific community for these regulatory reforms.
4:30 pm Break
5:00 pm Keynote Address
Ronald L. Wasserstein, Executive Director, American Statistical Association
“What Professional Organizations Can Do To Fix The Irreproducibility Crisis”
In 2016, the ASA took the extraordinary step of producing a statement on p-values and statistical significance. This was way outside its normal activities. Wasserstein will briefly discuss why ASA did this, how it was accomplished, and what the effects of the statement have been. He will talk frankly about how difficult this was to do, and what it takes to get a job like this done, but also share why he thinks it is important for professional associations to take up issues like this when it is the right time to do so.
5:30 pm Closing Remarks
David Theroux, Founder and President, Independent Institute
6:00 pm Closing Remarks
Peter Wood, President, NAS
The National Association of Scholars is an organization of scholars committed to higher education as the catalyst of American freedom. Our recent work on the irreproducibility crisis includes our report The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform (2018). The Independent Institute’s mission is to advance peaceful, prosperous, and free societies grounded in a commitment to human worth and dignity.
