The Journal of Pediatrics
Volume 151, Issue 2 , Pages 107-108, August 2007

Landmark, Landmine, or Landfill? The Role of Peer Review in Assessing Manuscripts

  • William F. Balistreri, MD (Editor-in-Chief)

      Affiliations

    • Corresponding Author InformationReprint requests: William F. Balistreri, MD, Pediatric Liver Care Center, Division of Pediatric Gastroenterology, Hepatology, and Nutrition, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Avenue, MLC 2010, Cincinnati, OH 45229-3039.

The Journal of Pediatrics, Pediatric Liver Care Center, Division of Pediatric Gastroenterology, Hepatology, and Nutrition, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio

Article Outline

 

Editors rely on experts (“peer reviewers” or “referees”) who are not members of the editorial staff for the critical assessment of submitted manuscripts. These experts screen and triage, scrutinize manuscripts for validity and reliability, target manuscripts to the appropriate audience, and endeavor to improve the quality of submitted work. Reviewers thus serve as gatekeepers to the medical literature. Ideally, this peer review process ensures several levels of review and decision making, matching thoughtful experts with works appropriate to their expertise. Peer review is done voluntarily and without compensation. It assumes integrity and avoidance of bias (negative or positive). For most journals, the process is blinded in some way; typically, the author does not know the identity of the reviewer, but the reviewer is fully aware of authorship. Such anonymity allows constructive, but often highly critical, comments.

See related article, p 202

The peer review process encourages the advancement of science and helps worthy manuscripts reach publication; however, the process as currently used by most journals is not without perceived flaws and critics. Recall Winston Churchill’s famous dictum: “Democracy is the worst form of government, except for all those other forms that have been tried from time to time.”1 The same might be said for peer review—it is the worst form of manuscript assessment except for all others that have been tried! There have been requests for qualitative studies to assess the value, fairness, and efficacy of the peer review process.2, 3, 4, 5 Some suggested alternatives have included double-blinded peer review (ie, the reviewer does not know who wrote the manuscript), unblinded/signed reviews, open peer review (ie, the author knows the identity of the reviewer), and open access publication.6, 7, 8 “Double-blinding” may not be effective, because the authorship often becomes apparent to the reviewer based on either content or context (eg, the reference list). In 1999, the British Medical Journal adopted an open peer review system that reveals the reviewers’ identities to the authors, and in 2000 the medical journals published by BioMed Central with open access adopted the open peer review system. No formal evaluation of these approaches has yet emerged.

What is our current procedure for assessing manuscripts at The Journal? We invite unvarnished criticism from reviewers, requesting that they keep the interests of the readers foremost, and treat the authors the way in which they would like to be treated. Specifically, we request that a reviewer (1) immediately return a manuscript if he or she is unable to complete the reviews promptly, (2) inform the editor of any potential conflict of interest, (3) look for scientific validity (methodology, results, and conclusions), (4) indicate strengths and novelty, (5) suggest ways to improve the manuscript (advice and encouragement), (6) report any ethical concerns, and (7) respect the authors’ intellectual property. This rigorous process requires a significant effort and expenditure of valuable time by reviewers.

What is the incentive for reviewers to deliver a timely, well thought-out, carefully prepared review, other than “academic altruism”? The diligent intellectual efforts and dedication of peer reviewers is an outstanding academic tradition—part of the “citizenship” and responsibility of being a member of the scientific community. We can all recall the first few painful reviews that we received as young authors and investigators and how the insight offered by an unknown reviewer improved our work. Is that sufficient incentive? It is not common practice to pay reviewers; publication costs are high, and the cost would have to be passed on to authors, readers, or libraries. Instead, journals may provide incentive and recognition in the form of annual recognition lists, complimentary subscriptions, bound copies of the journal to top reviewers (“frequent filers”), continuing medical education credits for manuscripts reviewed, and letters to promotion and tenure committees. Our choices for new members of our Editorial Board are often based on their excellent performance as reviewers.

So how do editors choose and recruit reviewers who are qualified (recognized experts in the area), unbiased, and able to provide timely assessments? There are several options; they might (1) use the journal’s established database of reviewers, (2) review the reference lists of the manuscripts, (3) conduct a literature search, (4) solicit recommendations from their editorial board, or (5) request suggestions from the authors, who, after all, know the field and the “players.” The last option might raise concerns that the authors would submit a list of individuals who are favorably biased (colleagues, friends, and family). However, the opposite also may be true; when cloaked in anonymity, a reviewer may possess a “license to kill.” By the way, authors are also welcome to indicate potential candidates who should be disqualified.

The question that vexes editors, therefore, is whether to use an author-submitted list of reviewers. Rivara et al,9 in this issue of The Journal, have compared reviews done by editor-selected reviewers with reviews by author-suggested reviewers. Specifically, they examined the quality, timeliness, and recommendations of the 2 sets of reviewers. The turnaround time was the same for both sets of reviewers, and the quality of the reviews was similar, although the editor-picked reviewers did achieve a higher “quality score.” Of note—the editor-selected reviewers recommended acceptance (vs rejection or revision) less often than author-suggested reviewers. These results are as expected; indeed, previously published studies have also suggested that author-selected reviewers are more likely to submit a favorable review.10, 11, 12 However, it is unclear whether this reflects the aforementioned fact that author-selected reviewers are the “true experts” or instead reflects cronyism or familiarly of the reviewers with the authors. In contrast to the study by Rivara et al,9 the previously published studies did not assess the quality of the reviews.

“Reviewers revise, editors decide.”

These findings validate our established procedures here at The Journal. When we solicit the desired 3 reviewers for each manuscript, we carefully consider, and frequently use, author-suggested reviewers in combination with names generated by the other means listed earlier. We then analyze the entire set of responses and draw our own conclusions from them.

The reviewers do not always achieve consensus. When reviewers disagree about the quality or acceptability of a manuscript, we use several strategies to reach a decision. We may solicit an additional review (often from our Editorial Board members) as a tie-breaker or simply offer the author a chance for rebuttal.

The bottom line is that the verdict as to whether the submitted paper is a landmark piece of work, a landmine of controversy or flawed science, or simply landfill is rendered by readers and scientists elsewhere. This is reflected in the letters to the editor and by citations and reproducibility.

Back to Article Outline

References 

  1. Speech to the British House of Commons - Nov. 11, 1947.
  2. Rennie D. Fourth International Congress on Peer Review in Biomedical Publications. JAMA. 2002;287:2759–2760
  3. Horton R. The hidden research paper. JAMA. 2002;287:2775–2778
  4. Jefferson T, Alderson P, Wager E, et al. Effects of editorial peer review: a systematic review. JAMA. 2002;287:2784–2786
  5. Falagas ME. Peer review in open-access scientific journals. Open Med. 2007;1:E49–E51
  6. Ceci SJ, Peters DP. How blind is blind review?. Am Psychol. 1984;39:1491–1494
  7. Williamson A. What happens to peer review?. 2002;Presented at an ALPSP International Learned Journals Seminar, London, April 12
  8. Snodgrass R. Single- versus double-blind reviewing: an analysis of the literature. ACM SIGMOD Record. 2006;35:8–21
  9. Rivara FP, Cummings P, Ringold S, Bergman AB, Joffe A, Christakis DA. Comparison of reviewers selected by editors and reviewers suggested by authors. J Pediatr. 2007;151:202–205
  10. Earnshaw JJ, Farndon JR, Guillou PJ, Johnson CD, Murie JA, Murray GD. A comparison of reports from referees chosen by authors or journal editors in the peer review process. Ann R Coll Surg Engl. 2000;82:133–135
  11. Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA. 2006;295:314–317
  12. Wager E, Parkin EC, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? (Results of a rater-blinded, retrospective study). BMC Med. 2006;4:13

PII: S0022-3476(07)00553-7

doi:10.1016/j.jpeds.2007.05.049

Refers to article:

  • A Comparison of Reviewers Selected by Editors and Reviewers Suggested by Authors

    Frederick P. Rivara, Peter Cummings, Sarah Ringold, Abraham B. Bergman, Alain Joffe, Dimitri A. Christakis
    The Journal of Pediatrics August 2007 (Vol. 151, Issue 2, Pages 202-205)

The Journal of Pediatrics
Volume 151, Issue 2 , Pages 107-108, August 2007

Article Tools