Peer review is faith not evidence based, but most scientists believe in it as some people believe in the Loch Ness monster. Research into peer review has mostly failed to show benefit but has shown a substantial downside (slow, expensive, largely a lottery, wasteful of scientific time, fails to detect most errors, rejects the truly original, and doesn’t guard against fraud). Research has also failed to show that “improvements” such as training, blinding, opening up the process, or using checklists make much difference, but the peer review continues, and perhaps “real world evidence” can show us the optimal system.
Atmospheric Chemistry and Physics has “real world evidence” that suggests it might have the best peer review system around. The editor, Ulrich Pöschl of the Max Planck Institute for Chemistry, describes the journal’s system as “multi-stage open peer review.”
The authors submit what the journal calls a “discussion paper,” but you could use the word preprint. It may be an original scientific paper, most are. There is a short interaction with the editors, and the authors may be asked to make “technical corrections.” Some papers may be rejected at this stage, but most go to the next stage of being posted on the journal’s website—with comments and ratings by the editor. At this point the world can see it. Editors select referees, who may choose to name themselves or remain anonymous.
This part bothers me as it’s thus a “partly transparent” rather than fully transparent process. And I always remember the experience of an author complaining about one of the BMJ journals that gave authors the option to name themselves: the author was unsurprisingly upset because his paper, despite receiving a positive review from a named reviewer, had been rejected on the basis of a highly critical review from an anonymous reviewer. Pöschl believes that giving reviewers the choice means both that reviews are of a higher quality—that is, less prone to potential biases–and that it’s easier to get reviewers.
The selected reviewers post their comments for the world to see, and anybody else, including the authors, may add comments. This is in effect an online open discussion of the paper, and the authors are expected to revise their paper in the light of the comments. It’s this final paper that is “published” in the journal, along with the comments from the editors and reviewers with the authors’ responses. (I’ve put “published” in inverted commas because the discussion paper is also “published” once the world can see it, but the final paper “published” in the journal supersedes the discussion paper.) Commenting and ranking continue after the paper is published, and the editors may also highlight papers.
“Real world evidence” suggests that this model is successful in that Atmospheric Chemistry and Physics has swiftly risen to have one of the highest impact factors in its discipline, at the same time as being one of the largest journals—publishing nearly 1000 papers a year—with one of the lowest rejection rates. The overall rejection rate of the journal is only 15%, while most other journals in the subject area have much higher rejection rates (greater than 50%) but achieve less impact, visibility, and volume.
Conventional wisdom is that you increase your impact factor by “having more papers and a bigger wastepaper basket”—that is, increasing submissions but publishing fewer of them. Pöschl thinks that it’s transparency that makes the difference—authors tend to submit only good papers to avoid having their papers sharply criticised in public and the open peer review leads to improved papers. The high acceptance rate also means that the author fee can be kept lower than that of competitors.
There is independent “real world evidence” to support Pöschl’s claims. A study published in the Journal of the American Society for Information Science and Technology concluded that the peer review system of Atmospheric Chemistry and Physicians was good at selecting studies that were then highly cited (a surrogate measure of quality).
The authors concluded: “All in all, our results on the predictive validity of the ACP peer review system can support the high expectations that Pöschl has of the new selection process at the journal: ‘The two stage publication process stimulates scientists to prove their competence via individual high quality papers and their discussion, rather than just by pushing as many papers as possible through journals with closed peer review and no direct public feedback and recognition for their work. Authors have a much stronger incentive to maximise the quality of their manuscripts prior to submission for peer review and publication, since experimental weaknesses, erroneous interpretations, and relevant but unreferenced earlier studies are more likely to be detected and pointed out in the course of interactive peer review and discussion open to the public and all colleagues with related research interests. Moreover, the transparent review process prevents authors from abusing the peer review process by delegating some of their own tasks and responsibilities to the referees during review and revision behind the scenes.’”
Imitation is the sincerest form of flattery, and other publishers have followed where Atmospheric Chemistry and Physics has led. These followers include Economics e-journal, SciPost/arXiv.org, Winnower, Wellcome Open Research, and F1000Research.
I’m working with Wellcome Open Research and F1000Research, and they have a model that has evolved from that of Atmospheric Chemistry and Physics. Firstly, there is no preselection on the grounds of quality: rather “editors” simply ensure things like ethics committee approval and compliance with reporting guidelines. Secondly, editors in the sense of powerful figures who control the process are gone: the process is driven by researchers. Thirdly, the authors must include their data with their study. This is a major development and important for science. Fourthly, authors select reviewers, and the reviewers have to sign their reviews. Fifthly, the study is regarded as “published” the minute it appears on the web; once papers receive two approvals from reviewers they are automatically indexed in Pubmed, but even if they never receive two approvals and might in the old world be regarded as “rejected” they are still on the website, still published. Sixthly, all versions of the paper are citable as are the reviewers’ comments.
Wellcome Open Research has only just started and F1000Research hasn’t been going long. It will be interesting to watch whether “real world evidence” shows them to be as successful, or even more successful, than Atmospheric Chemistry and Physics. The approach of Atmospheric Chemistry and Physics might be described as evolutionary in that journals and editors still exist, whereas F1000Research and Wellcome Open Research are about replacing journals and editors.
Richard Smith was the editor of The BMJ until 2004.
Competing interest: RS is a paid adviser to F1000Research, but he offers only occasional advice and has not been paid for this article. He did receive some expenses, although less than a full refund, for his attendance at a meeting in Copenhagen where he heard Ulrich Pöschl speak.