I’ve been tramping from stage to stage arguing that pre publication peer is slow, expensive ($1.8 billion a year), ineffective, biased, and anti-innovatory and should be dumped in favour of post-publication peer review. But what do I mean by post publication peer review? Despite my best efforts, which are clearly not good enough, people are often mystified.
Many people think of post publication peer review as the comments and letters that appear after scientific articles are published. They point out that these are usually sparse or non existent and don’t add much value. How could they substitute for traditional peer review?
That’s not what I mean by post-publication review, but it is a disappointment to me that comments are so sparse. My friend who blogs for cricket.com may get 2000 comments in response to a short blog, and blogs in the Guardian, for example, will often have several hundred comments. In contrast, the majority of scientific articles attract no comments whatsoever.
This is worrisome if like me you believe in the Popperian model of science—that the job of scientists is to pose falsifiable hypotheses and then try their hardest through experimentation to show them to be false. Comments from readers should be an important part of that strenuous effort at falsification.
I don’t know exactly why scientists don’t comment, particularly when you often hear them declare particular studies to be hopeless. The main reason is probably that there is no incentive to comment. Instead there are disincentives—fear of upsetting seniors, giving away good ideas, or being wrong.
More published comments could be a useful addition to post-publication peer review, but they will always be only a small part of it. The true post publication peer review, which is for me the real peer review right now, is the process whereby scientific studies are absorbed into the body of knowledge. The “marketplace of ideas” decides whether they are important and should lead to new practices and further research or whether, like most studies, they don’t matter much.
Some post publication peer review is formalised—like the production of systematic reviews or guidelines or the publishing of things like Evidence Based Medicine, Faculty of a 1000, or Journalwatch, which attempt to pick out what is important.
Systematic reviews give a sense of scale to post-publication peer review. Usually the authors will identify hundreds of articles but after applying quality criteria will dismiss all but a dozen or so. The authors will also have to search hard for unpublished studies (perhaps excluded by the lottery of pre publication peer review) and will observe that the best studies are all over the place and not concentrated in the “best” journals.
But more important than these formal types of peer review is the informal, the thousands of comments, decisions, and actions from the many that lead to a sorting of studies. I may hear a study presented or read a paper and be impressed. Others in the audience or other readers might also be impressed. We talk to friends about it. We email colleagues. We put it on listserves. Some of the recipients are impressed and start their own cascade. Others are less impressed and see problems. Perhaps a statistician attracted by the clamour reads the clinical article and sees important flaws that she shares with colleagues. Somebody might incorporate the study into a lecture, a review, or a grant application. And so a study might attract increasing attention and assume a prominent place, or it might fade as its receives more attention and more problems are noticed.
Many studies, in contrast, attract no attention—usually, but not always, rightly.
This decision of the many rather than the few is what I mean by post publication peer review. And the sooner we abandon pre publication peer review and let the real peer review begin as quickly as possible, the better.
Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.