An interesting article in Nature gives what may be a glimpse of the future of scientific discourse by telling stories of how social media have done a much better and faster job than traditional prepublication review. Science recently published a paper in which researchers claimed to be able to predict human longevity with 77% accuracy. The paper gathered huge coverage in the media, but almost immediately bloggers and Tweeters recognised major errors. Researchers who regularly used the techniques of the study saw a common pitfall, which was why they reacted so fast. One week after the paper was published the authors acknowledged that they had made a technical error, and shortly afterwards Science issued an “expression of concern,” meaning ignore this paper.
The speed with which the mistake was rectified is in startling contrast to the glacial pace at which most seriously flawed or fraudulent papers are retracted. Indeed, the number of retractions is way below what it should be when we know that the prevalence of serious fraud runs at about 1%. “Most papers sit in a wasteland of silence, attracting no attention whatsoever,” says Phil Davis, a communications researcher at Cornell University, in the Nature article by Apoorva Mandavilli.
It was, of course, bloggers who exposed Hwang Woo-suk, the Korean pioneer of stem cell research who was at the centre of the most spectacular fraud in recent years—also published in Science. And just a month ago another Science paper, which said that bacteria used arsenic rather than phosphorus in their DNA backbone, was rapidly dismembered by bloggers and Tweeters.
We shouldn’t be surprised by these developments. We know that prepublication peer review is a fatally flawed process (my latest critique is available as an open access) and we know that the wisdom of the many is much better than the wisdom of the few. We know too that postpublication review is the real peer review in that most papers disappear into obscurity and just a few emerge as important—and often not papers published in major journals. Our present system of trying to sort information by having the best papers in the best journals not only doesn’t work but it deceives us, giving too much attention to the sexy but often wrong.
Cameron Neylon from the UK Science and Technology Facilities Council is quoted in the Nature article as saying that it makes much more sense to publish everything and filter after the fact. We are moving from a world of “filter then publish” to a world of “publish then filter.” Why, I have wondered, are we not going faster, but then I read Maynard Keynes quote that “It is difficult to get a man to understand something when his salary depends on him not understanding it.”
What is perhaps surprising in all this is that researchers are happy to blog and use Twitter but are reluctant to comment on the websites of journals. Blogs in the Guardian or on Cricket.com attract hundreds and even thousands of comments, whereas articles in journals rarely attract any. Why are scientists Tweeting and blogging but not commenting on articles in journals? I can only speculate that it’s something to do with the stuffiness, formality, and pomposity of journals compared with the happy go lucky, party atmosphere of the blogosphere.