As an editor of The Lancet once stated: “We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.” Though stated quite bluntly, those familiar with the process would quite likely tend to agree to some extent – or admit that though useful, peer review (PR) is a time-consuming and inefficient process, to say the least.
In November 2011 a group of Finnish ecologists set out to ‘fix the woes of peer review without breaking what works’. They founded a commercial service and named it Peerage of Science (PoS). With over 1100 scientists and 20 journals currently participating, PoS has been attracting quite some attention in the media lately.
In short, this is how it works: a scientist uploads a manuscript anonymously and free of charge to the online PoS community, which becomes available for member-peers to review. Upon review, reviewers write a peerage essay (PE), which the authors receive along with comments and a categorical recommendation, ranging from ‘publishable’ to ‘withdraw’. Instead of entering another round of review, reviewers evalu- ate and grade each other. Reviewers could also choose to publish their PEs. Authors re-edit the manuscript, reviewers evaluate the final version using a 1-5 scoring of seven aspects (breadth, impact, originality, data, methods, inference and literature coverage). Finally, a quality index is calculated for the manuscript, the authors and the reviewers. Journals in turn pay to gain insight and can send publishing offers to authors.
I like this approach as it takes care of a couple of existing PR problems. First of all, reviews aren’t lost following rejection – like in a traditional down-the-ladder publication trajectory for instance. Reviewers evaluating and grading each other might also contribute to the making of a more objective system. Authors in turn need not worry about the lottery a new set of reviewers might raise when submitting to a new journal. Second, bravo to quid pro quo; directly linking effort of review to the benefit of having your own work reviewed quite likely acts motivating. Offering the opportunity for reviewers to build up a track record and to publish their reports also makes reviewing less of an one-sided effort. For this PoS can be applauded.
However, the chance of reviewers being biased or selective is still present and could be hard to distinguish from an honest review. The value of anonymous submission is also questionable since in many cases author identity can be easily guessed by peers. As each journal also has its own format requirements, it is not clear to me how PoS deals with this one-size-fits-all approach. Editors, usually charged with the task of managing reviewers for authors, benefit from a reduced workload. But with no-one actively policing and seeking reviewers, there is a chance that some manuscripts might just sink without a trace. Just to name a couple of many concerns that could – and should – be raised.
All in all, I think PoS takes a refreshing approach. Though not the first attempt to improve the efficiency and objectivity of the PR process, this open source and collective initiative seems to be catching on. I cordially invite the early adopters amongst you to participate; others can read the ‘peerage essay’ of Hettyey et al., which will appear in Cell this month.
This was published in the 6th Kavli Delft Newsletter, March 2013.