The Hidden Costs of Journal Peer Review

Robert Weissberg

Over the last thirty-five years as a political scientist I have had lots of experience with peer review, as an author submitting articles to be reviewed and as a referee. In principle, at least in political science, and I suspect the social sciences more generally, it is a decent quality control system. Nevertheless, it can also facilitate ideologically driven control over scholarship, no small matter as real-world politics increasingly invades the academy and publishing in peer-reviewed journals becomes the indispensable ticket to tenure.

First some background regarding how peer review works, at least in the social sciences. You write a paper, circulate a draft among colleagues, and then send it off to a peer-reviewed journal and hope for the best. The editor recruits referees, assigns the paper a number to shield authorship, and after perhaps a few months he must sort out the assessments. If referees initially disagree or express qualms, the editor might request “revise and resubmit” and rely on a third referee to resolve disagreements. Especially in the top journals, outcomes are mainly negative, so rejected authors typically then submit the paper to a less prestigious journal (this hierarchy is well known in social science disciplines though it can be complicated for highly specialized journals). This quest for publication may continue for years with multiple submissions, particularly since only successes count regardless of the number of rejections.

Two features of this process warrant special mention. First, referee judgment is anonymous and this secrecy opens the door to irresponsibility. A referee wanting to decline a submission can misrepresent the author’s views, lie to rebut rock-solid research, or otherwise behave capriciously with zero personal cost. Petty jealousies may even surface when the referee can identify the author.

Writers are generally defenseless against such attacks, no matter how outrageous or patently false. Editors and their staff generally lack the time and energy to meddle in these spats, given the crush of other articles awaiting consideration and their own lack of specialized expertise. Most editors look for ways to say no, and truth be told, most articles do not deserve publication.

Potential irresponsibility also applies to the journal’s editing. No fixed rules exist regarding who gets to review a submission and editors know from experience that certain referees are “killers” who pick everything apart as deficient, while others are far more generous. An editor can “honestly” therefore kill a piece that he dislikes for whatever reason, including ideological aversion, simply by forwarding it to an excessively demanding reviewer. A death sentence can also be given by facilitating an ideological mismatch: sending a submission that reaches a “conservative” conclusion to a referee famous for strident liberalism.

In other words, everything about publishing in refereed journals can, at least potentially, resemble sausage-making—nobody can tell what went into the process prior to the paper appearing in print.

The upshot of potential irresponsibly is for those submitting articles for review to play it very, very safe. The odds are against you in any case (top journals reject nearly everything), but an author should not create even more obstacles by angering anonymous reviewers or the editor. A wise and commonplace strategy would be to stay safely within the confines of accepted wisdom and use ample amounts of professional jargon to indicate familiarity with prevailing professional norms. Better to publish innocuous second-rate stuff in a first-rate peer-reviewed journal than to try to hit a scholarly home run and suffer rejection.

One can also up the odds of acceptance by heaping lavish praise on potential reviewers (however undeserved) while copiously citing friends who can be counted on to say “publish” if the submission comes their way. (These friends have probably read drafts, so the author will be known to them.)

Most important, in all instances avoid trouble by steering clear of anything even slightly “controversial.” Don’t kid yourself about speaking truth to power. If you are writing about urban politics and the prevailing orthodoxy insists that black electoral control has been a boon for African Americans in Detroit, don’t disagree. You can never win an argument with an anonymous reviewer. Journal submissions are not evidence-based debates or marketplaces of ideas—once the essay is in editorial hands, you have no voice in the outcome, no matter how biased or wrong-headed.

The upshot of avoiding trouble is that whole areas of scholarship become dead zones where only the politically ultra-orthodox can survive. This is most obvious in anything having to do with “the groups”—African American studies, women’s studies, Hispanic studies, queer studies, or any other subfield populated by clearly identifiable ideologues.

Even the choice of explanatory factors can lead to papers being DOA regardless of subject matter. Good luck to scholars who employ cognitive ability as measured by IQ when examining economic development or civil violence. How about submitting a paper that meticulously demonstrates that lack of impulse control explains poverty? Actually, when all these off-limits fields and subfields are added up, little remains open to anybody willing to challenge the bien-pensant crowd.

So what’s a scholar to do who refuses to drink the ideological Kool Aid but wants to publish in refereed journals? Stick to fields whose obscurity and technical complexities virtually preclude ideological intrusion. Prominent examples include rational choice, mathematical modeling, and statistical methodology. A savvy publication-minded scholar is also advised to dress up the paper with obtuse mathematics to intimidate readers with limited knowledge of mathematics. After all, nobody can antagonize the thought police for applying stochastic calculus models to a series of wars involving Country A versus Country B while Country C sat on the sidelines deciding whether to enter the fray. All perfect insurance against rejection for being politically incorrect, a throwback to the era when the “wrong” scholarship risked death so sages wrote in convoluted Latin.

The same “keep it obscure and irrelevant” advice even applies to the innumerate—become the world’s leading authority on an obscure fourth-century French cleric and his previously unrecognized contribution to the theory of republican government (and the clincher might be that this scholar was a woman living as a man in a monastery). Again, it’s hard to get into trouble if nobody reads your musings. And who will deny that this is “serious scholarship”?

This avoidance strategy also offers the advantage of reducing academic competition, thanks to the technical knowledge barriers to entry. While there may be thousands toiling in American or European politics generally, those viewing the world from the perspective of rational choice (among several other arcane subfields) may not exceed a hundred, and even better, they may all know one another and I’d guess profitably engage in mutual back-scratching.

All and all, this is bad news for scholarship. When viewed from outer space the landscape is bleak and it is no wonder that fewer and fewer academics writing for peer-reviewed journals become “public intellectuals.” Actually, after years of writing in jargon, some academics may have lost the ability to communicate in plain English. How this affects recruitment of graduate students into the profession is a topic of great concern, but I’ll leave that subject for another day.

An Untimely Death by Peer Review: A Personal Account

To illustrate the power of peer review to create academic dead zones, let me offer a personal account that involved the hot-button topic of race. Back in the late 1980s I presented a paper at the Midwest Political Science Association offering an overview of current research on American racial politics. It highlighted various research lacunae that, at least in my estimation, warranted closer scrutiny.

The panel filled the room with many listening just outside the doorway and others sitting on the floor. This, I should add, was highly unusual at the Midwest convention, where most panels attract only a smattering of observers. Reaction to all the papers including mine was exceptionally lively (but very civil) and might have continued for hours afterward. Nobody criticized anything as hateful, racist, or inappropriate for a disciplinary meeting. I reasonably concluded that a large audience existed for a frank discussion on this “taboo” topic.

I then submitted a revised version of my paper to the American Journal of Political Science, a prestigious journal in my field. After a few months an editor’s rejection arrived accompanied by two reviews. One unequivocally advised acceptance but added, “I disagree with some of the author’s points, but the topic is a critical one, long under-examined and this paper might instigate important, much needed research.” The second reviewer was unambiguously negative.

The “reject” review failed to address my key points regarding how current research on race was incomplete. Instead, it sidestepped everything with a rambling, prolix literature review of sundry well-researched race-related subjects, many about court cases, all of which had been around for decades. In effect, this reviewer was saying “today’s research on race is fine, so no need to venture elsewhere.” Evidently, the one negative review trumped the strong positive one from the other reviewer. With this verdict, there was no need to get a third opinion to settle the one “Yes,” one “No” outcome.

Then something highly unusual happened—the one positive reviewer called me up. I had never met him but knew his work well and it was first-rate. He was a self-defined leftist, a supporter of the civil rights agenda, but also, as he told me, a strong believer in professionalism and let the chips fall where they may. We chatted for over an hour and he told me that he had previously dabbled in racial politics but then abandoned it. We both agreed that when it came to doing research on race, it was futile to struggle to secure research funding, get published, or otherwise move up professionally with so many other noncontroversial research topics available.

What is also unusual about this tale is how far I got into the system before quitting. Other academics but especially those early in their careers are unlikely even to contemplate writing about race unless they desire a career in the African American academic establishment. They know full well that the topic is a minefield where the slightest deviation from the orthodoxy can end an academic career, even for African Americans. Awareness of this reality probably begins in graduate school and is likely hammered home by dissertation advisors who quietly tell advisees about the scholarly version of racial apartheid.

Curing the Ideological Mischief of Peer Review

How can we maintain high scholarly standards while minimizing the pernicious impact of ideology? Let me suggest four solutions—all feasible, none excessively expensive—that can be implemented relatively quickly. The first two involve making the process more transparent and therefore more responsible. The third suggests creating an “alternative universe,” while the fourth would use the Internet to eliminate any and all ideologically-imposed choke points.

As noted, the current system’s anonymity permits bias—the ideologically driven reviewer can irresponsibly say “publish” no matter how shoddy the research, and this low standard further encourages yet more journal submissions with the “right” bias that, ultimately, creates a collection of one-sided research called “the scholarly consensus.” Some of this ideologically motivated exchange of favors can be eliminated by listing those who consented to publication. With this public acknowledgment in place, the strident ideologue might think twice about green-lighting a thinly disguised, sloppily executed polemic. Indeed, for time-pressed readers a quick glance at the referee names might be a shortcut to avoid wasting time, while the editor might also think twice about being exposed for cooking the books. There is also the benefit of curtailing cronyism, no small matter when the intellectual value of a publication is weighed. A published article exclusively approved by the author’s friends will surely be discounted, no matter the quality.

A second route to transparency is to alter the meaning of “peer-reviewed.” Just replace the anonymous referee system with a publicly identified board of review (this already occurs at some journals and is considered legitimate). In some instances, the review board may run in the dozens with multiple specialties. In any case, authors submitting manuscripts could anticipate who will judge their work and it would not take much to sniff out potential killer bias. The other side of the coin would be that the publicly identified editorial board members would have to take full responsibility for approving any ideologically flavored rubbish.

A third possibility might be called “creating an alternative universe by avoiding current, corrupt peer review.” Just establish new journals—even electronic ones explicitly welcoming to those who might challenge prevailing consensus—journals for the ultra-unorthodox, so to speak. (Highly regarded but now defunct, The Public Interest exemplified this approach with its penchant for articles probably not publishable by more “academic” journals.) A variant might be an annual book series with titles like “Controversies in Political Research.” Such publications would encourage multiple viewpoints, not just tit-for-tat counter-balancing the other side’s partiality. This solution rests on my observation that many academics, even liberals and perhaps many radicals are reasonably open-minded when judging journal submissions (recall my personal experience with such a leftist referee).

There is, however, a major problem with promoting openness to “controversial” ideas. It is one thing to get “controversial” ideas in print, but quite another to get these publications accepted as legitimate currency in the academic marketplace. And without this certification, these “controversial” contributions will be ignored, even judged as liabilities in departments dominated by narrow-minded ideologues. The parallel is how many departments disregard anything, no matter how profound, that appears in journalistic outlets. Such writings are disregarded as not being “serious scholarship.” In short, getting ideas out only counts if the vehicle is certified as “academic” and this certification can be politically motivated.

Lastly, we can eliminate all gatekeepers regardless of bias simply by “publishing” everything on the Internet. This is far less radical than it may initially appear and, to play the environmentalist card, it would save millions of trees (and gallons of ink) while reducing carbon emissions. Meanwhile, financially strapped academic departments would no longer have to subsidize editorial costs (including expensive copyediting), but the big winners would be libraries now freed from warehousing expensive (and seldom read) journals. A further dividend might be the reduction of professional dues. And when I say publish “everything,” I mean everything no matter how polemical or poorly executed.

Here’s how. Anybody who wants to publish a paper would simply email an electronic version to the headquarters of his profession, where it would be immediately posted on the association website. Authors would provide keywords to sort out this avalanche and subscribers would automatically be sent daily downloadable unscreened listings in their specialties. “Papers” would include the author’s email address and a link to where readers can post comments. At last, a wide open marketplace of ideas.

Actually, much of this exists in the form of informal networks where authors share prepublication drafts for comments. Obviously, the principle benefit is to eliminate ideologically motivated gatekeepers, but there are intellectual rewards. First, since everything is now “published,” evaluations for tenure, salary increases, and promotion must be based on actual substance, not the shortcut “it must be decent since it was published in the Journal of Modern Obscure Nonsense.

A second, non-obvious intellectual benefit might be called “scholarly birth control.” Since everything, no matter how dreadful, is now “published,” those who unhappily waste years trying (usually unsuccessfully) to turn dross into gold can better spend their time elsewhere, perhaps improving their teaching or correcting student essays. Experienced professors know this phenomenon all too well: the colleague who has been struggling with a paper, constantly revising it and sending it out for consideration. Eventually, it may see the light of day, though truth be told, few will read it and it will die an obscure death. That understood, I’d predict that the number of papers intended for journals might decline once authors recognize that getting published has zero benefits, since everyone is published. Now only those who really have something to contribute would contribute.

Conclusion

Illustrations aside, it should be clear that the potential for ideological control of scholarship via peer review can apply across the political spectrum. Liberals and radicals are today’s guilty party, but tomorrow it could be conservatives and I suspect that in some obscure corners of the academy, conservatives dominate. This analysis is about opening up the dissemination of knowledge, not some scheme to give voice to “conservative” views currently excluded by peer review.

Now for the bottom line. What are the odds of any of these suggestions to promote openness succeeding? Each recommendation is reasonable, not too costly, and all, in one form or another, already exist.

The future looks bleak. With few exceptions today’s academics are not interested in a wide open marketplace of ideas without gatekeepers in the social sciences. After all, liberals and radicals—especially those pushing “group” agendas—have spent decades securing control over the academy, including many of its journals, and are hardly anxious to surrender their prize in the name of some lofty appeal to intellectual openness.

Many of these academics will also insist that one-sided “scholarship” is vital to achieving social justice. Moreover, permitting ideological dissenters a soapbox, they argue, only gives voice to “hate,” homophobia, racism, and similar bad thinking. The sad reality is that as the gap between their ideological fantasy that dominates today’s social sciences and the real world widens, the need to quash contrary views grows more pressing.

Even non-ideologues have a vested interest in keeping peer review somewhat corrupt. More than a few professors, particularly in small, arcane fields, have learned to manipulate the system. What could be more career-boosting than a long vita of peer-reviewed articles in a reject-nearly-everything prestige journal that barely anybody can comprehend? Why abandon a wonderful arrangement that has taken years of mutual favors to achieve?

Despite all this, let me suggest one sliver of hope. This generation of indecipherable social science “knowledge” facilitated by peer review’s incentives may ultimately collapse other than as a currency to distribute professional rewards. Recall the decline of turgid Latin-based medieval scholasticism—millions of words now almost totally gone. At some point no one will actually read anything for content; articles (or, more likely, their abstracts) will instead be examined only to keep abreast with the literature or to populate one’s own research papers with citations.

The result, hopefully, will be a shift in attention toward non-refereed publications as a source of scholarly insights, perhaps publications sponsored by think tanks, foundations, even “intellectual” magazines like Commentary. In other words, with so little substance in peer-reviewed outlets, those seeking real knowledge will turn to well-researched content independent of the certification process. An essay will be deemed “good” for the simple reason that it is good—and the way it managed to find its way into print will, happily, become irrelevant.

  • Share
Most Commented

August 23, 2021

1.

Testing the Tests for Racism

What is the veracity of "audit" studies, conducted primarily by sociologists, that appear to demonstrate that people of color confront intense bias at every level of society?...

April 16, 2021

2.

Social Justice 101: Intro. to Cancel Culture

Understanding the illogical origin of cancel culture, we can more easily accept mistakes, flaws, and errors in history, and in ourselves, as part of our fallen nature....

April 19, 2021

3.

Critical Race Theory and the Will to Power

A review of "1620: A Critical Response to the 1619 Project" by NAS President Peter W. Wood....

Most Read

May 30, 2018

1.

The Case for Colonialism

From the summer issue of Academic Questions, we reprint the controversial article, "The Case for Colonialism." ...

December 21, 2017

2.

March 20, 2019

3.

Remembering Columbus: Blinded by Politics

American colleges and universities have long dispensed with efforts to honor or commemorate Christopher Columbus. But according to Robert Carle, “most Americans know very little about......