The Pros and Cons of Open Peer Review

May 16, 2023 | Uncategorized

Some journal publishers, like PLOS and Frontiers, have relatively recently adopted open review policies. But what we mean by ‘open review’ varies within the academic literature on the topic, as well as in practice from journal to journal. Tony Ross-Hellauer identified seven characteristics or traits of open review in his systematic review of 122 definitions. Let’s take a look at some of these in a bit more detail. The most common characteristic of open peer review is open identities, where the identities of both the author and the reviewer are known. A total of 94% of STEM-focused studies – that is, most of the literature on open peer review – consider open identities to be integral to open peer review. For the humanities and social sciences, this figure falls to 84%. Open reports are another common characteristic, whereby the reviewers’ reports are published alongside an accepted article. A third characteristic of open peer review is open participation, where the journal invites both non-specialists and experts to comment on a given piece of work. Open participation is perhaps not as well-known as the other two characteristics, but 54% of studies in the humanities and social sciences still include open participation in their definition of open peer review – more than double the proportion for STEM studies.

Certainly, there is a lot to be said for open review policies. Open reviews encourage accountability and transparency and promote constructive criticism. Others feel, however, that by losing traditional blind and double-blind review methods, we also reduce scholarly rigour and expose both authors and reviewers to a range of dangers.

The most marked positive of open review is something that we discuss in Lex Academic’s blog post on Reviewer #2: some reviewers seem to have forgotten how to behave towards other humans. The blind system, in other words, can encourage the kind of bad behaviour that a more open system can prevent. For instance, on a popular internet forum, one academic posted comments from the review of their #firstpaper: ‘If I never see another piece of writing started with these lines, it will still be too soon.’ What a devastating comment for a young academic to read. It seems remarkable that some of our most educated people will simply troll their juniors and peers if they know that no one will trace bullying comments back to them. Even having a simple human–human connection, like sharing the names of authors and reviewers, supposedly discourages academic trolling, and pushes reviewers to identify the strengths as well as the weaknesses of a given paper.

Beyond open identities, a publisher might print a reviewer’s report alongside a published article – a measure that some feel enriches the academic record. This is an option in the journal Nature, although the measure lacks teeth when reviewers can choose to withhold their consent and/or remain anonymous. Still, creating a culture where reviewers will themselves be open to scrutiny can help to make them accountable. Open identities and open reports are also arguably more in the traditional spirit of scientific communication. We don’t – shouldn’t – review in order to anonymously trash one another. Rather, we review to embrace debate and extend our scholarship by exchanging ideas. As Ben Goldacre points out in his books and many of his ‘Bad Science’ blog posts, scientists (should) welcome being challenged and, surely, open identities in turn facilitates open dialogue. There are further advantages for readers. If presented with reviewers’ comments, a reader can also get a measure of how a piece of work developed. What problems were identified? How did the author address them? Might there be another way around a given problem? Getting a glimpse into the mechanism of peer review is valuable for early-career scholars. These reviews might serve as models of good practice and help us to recognise the value of a good review.

The advantages of transparent peer review systems are many, and in a culture where bullying can be significant, they are arguably overwhelming. But there are also considerable disadvantages. One is that open reviews make it more likely that authors and reviewers will influence one another (although you might also say that in a world where science is becoming more collaborative, we might fairly encourage a less combative model of peer review). An early-career academic might also be reluctant to criticise the work of a more established figure, especially if they know that the person can make trouble for them in the future. Even more serious – and the reverse of the badly-behaved Reviewer #2 trope – is the possibility that a critical reviewer might open themselves up to reprisals from disgruntled and unstable authors. Criticism and rejection, although common and mostly accepted, can also torpedo one’s mental health and in extreme circumstances, even one’s career. It would take only one revengeful author to cause serious safeguarding problems.

The question of whether we should pursue open peer review is a matter of values. We could preserve a system of examination, where a scholar’s work is assessed by a panel of their peers who decide whether their work should be admitted into a hallowed journal. In such a case, blind review clearly takes the lead. If, on the other hand, we can envisage a review system built on mutual respect and collaboration, rather than the traditional academic swordplay, open review seems like a sensible step towards a more humane system.