What does peer review do for science and what does the scientific community want it to do? Should peer review detect fraud and misconduct? Does peer review illuminate good ideas or shut them down? Should reviewers be anonymous?
The Peer Review Survey 2009, a large international poll of authors and reviewers, released its preliminary findings today.
Peer review is considered fundamental to integration of new research findings and it allows other researchers to analyze findings and society at large to weigh up research claims. It results in 1.3 million2 articles published every year and it has been growing rapidly with the expansion of the global research community - and corporate publishers.
With that expansion there are new concerns, like getting the next generation of researchers to review in sufficient numbers, how to maintaining the system’s integrity and whether it can be truly globalized. There also needs to be consideration of new ideas - about alternative quality measures, technologies to prevent plagiarism, rewarding reviewers and training them.
To find out researchers think about peer review and its future, Sense About Science developed the Peer Review Survey 2009 - though with a grant from Elsevier, a corporate publisher.
The survey included some questions from the Peer Review Survey 2007 for comparison, and new questions about future improvements, public awareness and pressures on the system.
Preliminary findings include:
1. Playing an active role in the community is the top reason to review:
90% say they review because they believe they are playing an active role in the community; only 16% say that increasing their chances of having future papers accepted is a reason to review.
2. Researchers want to improve, not replace peer review:
- 84% believe that without peer review there would be no control in scientific communication, but only a third (32%) think it is the best that can be achieved; 20% of researchers believe that peer review is unsustainable because of too few willing reviewers.
- 91% say that their last paper was improved through peer review; the discussion was the biggest area of improvement.
- 73% of reviewers (a sub-group) say that technological advances have made it easier to do a thorough job than 5 years ago. Whilst 86% enjoy reviewing, 56% say there is a lack of guidance on how to review; 68% think formal training would help. On average, reviewers turn down two papers a year.
- Just 15% of respondents felt that ‘formal’ peer review could be replaced by usage statistics.
- 61% of reviewers have rejected an invitation to review an article in the last year, citing lack of expertise as the main reason – this suggests that journals could better identify suitable reviewers.
3. High expectations:
- 79% or more of researchers think that peer review should identify the best papers, determine their originality and importance, improve those papers and, though lower scoring, also determine whether research is plagiarized or fraudulent.
- While 43% of respondents thought peer review was too slow, 65% of authors (a further sub-group) reported that they had received a decision on their most recent paper within 3 months.
4. Reviewers want anonymity:
58% would be less likely to review if their signed report was published. 76% favor the double blind system where just the editor knows who the reviewers are.
5. Understanding of peer review:
Researchers agree that peer review is well understood by the scientific community but just 30% believe the public understands the term.
6. Papers aren’t recognizing previous work:
81% think peer review should ensure previous research is acknowledged; 54% think it currently does. This reflects current concerns in the research community.4
7. Detecting plagiarism and fraud might be a noble aim but is not practical:
A majority think peer review should detect plagiarism (81%) or fraud (79%) but fewer (38%&33%) think it is capable of this.
8. Reviewers divided over incentives:
Just over half of reviewers think receiving a payment in kind (e.g. subscription) would make them more likely to review; 41% wanted payment for reviewing, but this drops to just 2.5% if the author had to cover the cost. Acknowledgement in the journal is the most popular option.
Robert Campbell, Wiley Blackwell Science&Chairman, Publishing Research Consortium, said, "Most researchers give up time to review papers for no charge. The whole scholarly communication system is dependent on this. Why do they do it? This study helps us to answer the question. And I take heart in the finding that 85% of the around 4000 respondents quite simply do it because they enjoy being able to improve papers. "
Irene Hames, Managing Editor The Plant Journal, Author 'Peer Review and Manuscript Management in Scientific Journals: guidelines for good practice', said, “It's very important to know what authors and reviewers actually think about the current status of peer review. Too many commentators make broad generalizations that are not evidence based. So I welcome this large-scale survey from Sense About Science. Once again, the importance with which peer review is viewed comes through, with the great majority of researchers believing that without peer review there would be no control in scientific communication. That is not to say there aren't problems - there clearly are, and improvements and innovative solutions are needed. Crucial in this is the need to professionalize this area of activity, which too often is put in the hands of people who may have great academic reputations and research expertise, but no experience of running a peer-review system.”
The preliminary results of the survey will be presented at Science Fact or Science Fiction: Should Peer Review Stop Plagiarism, Bias or Fraud? at the British Science Festival, Surrey University on Tuesday 8th September 2009, 10:00am. Tracey Brown of Science About Science, David Adam of The Guardian and Peter Hayward of Lancet Infectious Diseases will debate the challenges of publishing research.
Notes:
1. The Peer Review Survey was an electronic survey conducted between 28th July 2009 and 11th August 2009; 40,000 researchers were randomly selected from the ISI author database, which contains published researchers from over 10,000 journals. Altogether 4,037 researchers completed our survey. The error margin was ± 1.5% at 95% confidence levels; reviewers answered a subset of questions aimed specifically at reviewers (3,597 - a subset of the base) the error margin for this group was ± 1.6% at 95% confidence levels.
2. The full findings and report are due to be published in November 2009 and will be available at http://www.senseaboutscience.org.uk/index.php/site/project/29/.
3. Björk et al (2008) ‘Global annual volume of peer reviewed scholarly articles and the share available via different Open Access options’ Proceedings ELPUB2008 Conference on Electronic Publishing – Toronto, Canada – June 2008
4. Sense About Science is a UK registered charity, No. 1101114, to equip people to make sense of science and evidence. It has previously published ‘Peer Review and the Acceptance of New Scientific Ideas: a working party report’ 2004; and publishes the public guide to peer review, ‘I Don’t Know What to Believe’. In 2008, it established the online education resource about scientific publishing and peer review, for schools www.senseaboutscience.net.
5. Publishing Research Consortium (2007) ‘Peer Review in Scholarly Journals: perspective of the scholarly community. An international study’ www.publishingresearch.net/documents/PeerReviewFullPRCReport-final.pdf
6. Chalmers&Glasziou (2009) ’Avoidable waste in the production and reporting of research evidence’ The Lancet; 374: 86–89.
The anonymous refereeing could then be done much faster, the anonymous referees could simply state their opinions without putting as much work in it as sometimes the case now. Then they could in fact come in the public part with their detailed suggestions - and these would now be credited to them, not kept lost as they are now. In the published papers those "contributors" should then be credited if their suggestions are followed. I suppose the original anonymous referees should also play some role in the final decision, probably assisted by a responsible editor.
This would be just small changes to the present process, and it will definitely lead to improvements. Naturally, one could also think about more radical changes, like wikipedia-type publishing, but with elements of the present peer review - more like real Science 2.0 or even 2.5 or 3.0, who knows.