Yesterday, I gave evidence to a UK parliamentary inquiry into peer review (as did Fiona Godlee). (The session can be viewed here)
Before the session I tried to think of an analogy for peer review that I could use to explain its usefulness, but also its variety and imperfections, to the MPs. Inspiration often comes from unlikely sources – in this case an episode of the 1980s TV sitcom “Minder” which is based around the dodgy dealings of secondhand car salesman Arthur Daley (about as unlikely a role model for a journal editor as you can find, but I hope Fiona and her fellow editors will forgive the comparison).
So, what has peer review got to do with selling used motors? In the UK, all cars over three years old have to undergo an annual inspection called the MOT test (named after the now nonexistent Ministry of Transport). Here was my analogy. The test is designed to reduce breakdowns and keep traffic flowing. It’s not perfect and it doesn’t guarantee that every tested car will run smoothly for the next 52 weeks but it probably keeps the worst wrecks off the roads and often identifies problems that can be fixed (just like peer review). It’s considered reasonable and is therefore acceptable to most law abiding motorists. Of course, more breakdowns could be prevented if cars had to be tested once a month instead of once a year, but that would be disproportionate and would mean cars were off the road too often while they were being tested. (Similarly, peer review could be made tougher by insisting that journals review or re-analyse the raw data, but few people would consider this acceptable and it would cause substantial delays.)
Another reason why I was pleased with the MOT analogy is that, just like peer review, the test doesn’t always spot deliberate fraud such as a mileometer that’s been “clocked” (i.e. turned back to show fewer miles than the car has actually travelled). If you were buying a used car you would probably want to know more than the fact that it had passed its MOT. You might go to a reputable dealer or get it checked by a friendly mechanic. Users of articles should probably do the same and I guess if the journal editors are the dealers, then statisticians and methodologists are the friendly mechanics who check your potential new purchase and warn you of its imperfections.
And, lastly, you need an MOT whether you drive a Rolls Royce or a grubby white van – the MOT doesn’t tell you which is which. Similarly, peer review is used by a huge range of journals, some of which are highly selective and some of which will publish almost anything so long as it is “roadworthy” (i.e. valid).
So, in one of Arthur’s own splendid malapropisms I’ve now got a nice “allergy” for peer review. But, despite my own South London roots, I’ll resist the temptation of lapsing into rhyming slang for my next blog but challenge readers to suggest a nice Cockney name for peer review.
Liz Wager PhD is a freelance medical writer, editor, and trainer. She is the current chair of the Committee on Publication Ethics (COPE).