How to review a Paper: Suggestions from the Editors of Surgery and the Journal of Surgical Research

      The following are the proceedings of a breakfast session sponsored by Elsevier, held on February 4, 2016, during the 11th Annual Academic Surgical Congress in Jacksonville, FL. The 4 invited speakers were the current editors of Surgery and the Journal of Surgical Research (JSR). A selected reference list follows these proceedings.

      How to review a manuscript

      The first speaker was Michael Sarr, MD, former and Emeritus Professor of Surgery at the Mayo Clinic and the current Co-Editor-in-Chief of Surgery. The topic of his talk was “How to Review a Manuscript.” Dr Sarr began with several questions to the audience related to the relevance of the overall session:
      • 1.
        When asked how many had reviewed a scientific paper, an overwhelming majority raised their hands.
      • 2.
        When asked how many had done >5 reviews, again, most audience members raised their hands.
      • 3.
        When asked whether they had been coached or taught how to do a review, <20% of the audience voted affirmatively.
      • 4.
        When asked whether they had ever been “offended” by the callousness of reviewers' comments, more than half of the audience agreed.
      • 5.
        Finally, when asked whether they had ever felt that a reviewer had done a bad job, had not read their paper, or had not understood their paper, again, more than half of the audience agreed.
      Having set the stage for the session, Dr Sarr first focused on what an editor wants from reviewers. Although the editor does want the reviewer's opinion on whether the manuscript should be published, what is most important is a critical, unbiased, comprehensive review of both the good and bad points of the paper. The review should include constructive, positive suggestions to the authors. Additionally, in a section usually titled “confidential comments to the editor,” the editor wants a very clear opinion on the manuscript's importance and whether it falls into the following categories: (1) warrants publication, (2) requires a minor revision, (3) requires a major revision, or (4) requires a major revision and re-review. Notably, the timely submission of the review (<2 weeks) always is encouraged.
      Dr Sarr continued his talk by describing the sections of the review expected by the editor. These include the major sections usually titled “confidential comments to the editor” and “comments to the author.” The latter section usually begins with a short summary of the article (3–5 sentences). Then, when indicated, the reviewer should list by number the major points (these represent important points that the author must address or change) and the minor points (these are often correctable by the author or editor with simple editing). Numbering these points facilitates the authors' responses to each point (see suggested outline for a scientific review in Table). When the review is given as a full paragraph with multiple suggestions embedded within the paragraph, it is very difficult for the author to address all of the criticisms in a logical, easily understood manner; likewise, it is very difficult for the editor to review the authors' responses.
      TableSuggested outline for a scientific review
      • SUMMARY: The initial paragraph (2−3 sentences) should mention the name and location of the author(s) with a short summary of the aims and goals of the manuscript.
      • MAJOR POINTS: These are critical points that the authors must address; they should be numbered separately to help the author and the editor to address a revision in an orderly manner.
        • 1.
        • 2.
        • 3.
        • etc.
      • MINOR POINTS: These are points that should be easy for the author to address; again, the individual numbering helps authors and editors.
        • 1.
        • 2.
        • 3.
        • etc.
      • Finally, the reviewer may elect to sign the review with his or her name; if so, then his or her name will be communicated to the author with the review. However, note that some journals keep the review process completely anonymous.
      In the “confidential comments to the editor” section, the reviewer should begin with a brief description of the study (2–3 sentences). Next, the reviewer should provide comments about the importance of the topic for that journal, followed by a brief description of the strengths and weaknesses of the paper, as well as any limitations or problems that would preclude its acceptance. These comments do not need to be as detailed as the comments provided to the authors, but they may be used to help determine the study's importance and potential for publication. Finally, the reviewer can add his or her very candid opinion on whether the article should or should not be published, remembering that the authors will not see this or be able to identify the reviewer. Optionally, if a reviewer enjoys doing reviews, this is a good place to thank the editor and encourage the editor to continue sending manuscripts for review. The reviewers can also sign or type their name in this part of the review; this is good PR for the reviewer and shows the valuing of what we call “academic citizenship.”
      In the “comments to the authors” section, the reviewer provides evidence to the authors (and the editor) that he or she critically read the paper and got the “big picture.” In a short paragraph (2–3 sentences), the reviewer conveys his or her understanding of the study's goals or aims and its methods and results. The reviewer's goals are to review the underlying science even if the English is poor, identify things that must be changed before publication, and make constructive suggestions for important improvements. Also, the reviewer should try to be the authors' advocate (that is certainly what we would all want when someone reviews our own submissions) and start out by being complimentary to the authors, such as with comments like “the authors identified a potentially important topic” or “the authors did a nice study investigating…” The goal of a reviewer is not to refute and criticize everything in the paper. A reviewer should always be courteous, inoffensive, and constructive and remember that, with constructive criticism, there is always a way to say that the study is not good or appropriate in a nice way, such as “the authors attempted to determine the genomic signature, but unfortunately…,” followed by a description of limitations and suggestions for improvement. Overall, the reviewers should convey to the authors that they understood the work and are providing a fair assessment.
      In the subsequent detailed scientific review, the reviewer discusses the following matters. Was the methodology understandable? Were the approach and design correct? Was the statistical analysis understandable, and was there adequate statistical power? Was institutional review board approval needed and confirmed? Was the results section well organized? Did the discussion section discuss all pertinent prior literature and the authors' data in the proper context? Do the data support the conclusions? Are the figures and tables appropriate? Are there too many figures and tables or too few, and do they merely repeat information from the text? Should these sections be reconfigured?
      Dr Sarr next discussed reviews of clinical papers. Whereas basic science papers should be hypothesis driven (always remembering the null hypothesis), clinical papers and reviews are often goal or aim driven. A reviewer should check for the national or international registration of randomized controlled trials that allows other investigators to access the study and should check for mention of adherence to the guidelines of the Consolidated Standards of Reporting Trials (CONSORT) Statement.
      • Schulz K.F.
      • Altman D.G.
      • Moher D.
      for the CONSORT Group
      CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
      Reports of randomized controlled trials must include a power calculation, describe an appropriate randomization process, and ask an important question.
      The following special circumstances were then discussed:
      • 1.
        In database studies, the database needs to be described by the authors. Is the database administrative or clinical? Is the database risk-adjusted? Can the database really answer the question? If so, how can the findings be used to improve outcomes?
      • 2.
        Systematic reviews and meta-analyses should ideally conform to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement guidelines.
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        The PRISMA Group
        Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA Statement.
        The reviewer should ask the following questions:
        • a)
          Is the topic important enough?
        • b)
          Does the manuscript include primary reviews of more than just titles?
        • c)
          Does the review feature only articles written in English?
        • d)
          Is there bias or heterogeneity? (This must be addressed in the results section.)
        • e)
          Is there a true evidence-based conclusion?
        • f)
          Are the conclusions useful?
        • g)
          Does the review end with the conclusion that there are not enough data to make any evidence-based conclusions? If so, is the study really worth publishing?
      Unfortunately, systematic reviews and meta-analyses are only as good as the literature, and the authors cannot fully assess this point until after the study is completed. Because the authors put a great deal of work into doing the study, they all too often submit the manuscript even if it is not really helpful. Alas, if the literature is not robust enough to answer the question, then the study is not helpful and probably does not warrant publication.
      • h)
        When evaluating manuscripts from authors who are not native English speakers, reviewers should start by looking at the science, not the grammar. If the science is good, does the paper need to be rewritten by an English editing service, or can the editor or publisher edit it satisfactorily?
      Dr Sarr summarized his presentation with the following: What an editor wants is for the reviewer to be fair and comprehensive, with the ultimate goal of publishing solid, potentially important work. Sometimes the editor and/or the authors want to publish only “discoveries,” but more commonly in surgery journals, the goal usually is to publish true translational work.
      Finally, the editor (and the author) wants a timely submission of all reviews.

      What an author wants from reviewers

      The next discussant was Kevin Behrns, MD. At the time, Dr Behrns was the Edward R. Woodward Professor and Chairman of the Department of Surgery at the University of Florida-Gainesville. Since January 1, 2017, he has become the Vice President for Medical Affairs and Dean of the Saint Louis University School of Medicine. The topic of Dr Behrns' talk was “What an Author Wants.” His objectives were as follows:
      • 1.
        To review the reasons an author chooses to submit to a particular journal,
      • 2.
        To discuss the expectations an author has of a reviewer and an editor,
      • 3.
        To provide examples of reviews that meet an author's expectations, and
      • 4.
        To highlight additional ways to improve surgical manuscripts through increased communication and collaboration.
      The first objective for authors is to select the right journal for their manuscript. Factors that should be weighed include whether the journal has a national or international reputation and whether the journal has the appropriate readership and content areas for the science. Authors also prefer an easy submission process, timely reviews, and a seamless publication process if the article is accepted. Guidance from the managing editor and an understanding publisher also are plusses.
      From the authors' perspective, the review should make it clear that the manuscript has been read and understood and that the work has been evaluated in context with the literature. An organized review will help authors understand which specific issues must be addressed and whether the deficiencies preclude publication. Dr Behrns' discussion reiterated Dr Sarr's suggestions in that the review should provide a brief summary of the work, comment about the novelty versus addition of incremental knowledge, cite major and minor issues with the work, provide examples of how the work can be improved, indicate whether the study design is substantially flawed, and discuss whether the methodology and/or conclusions can be changed to potentially allow publication.
      Using an example of a manuscript review from Surgery, Dr Behrns provided examples of a brief summary, major comments, and minor comments. In the brief summary (see below), the reviewer identified the first author, reiterated the hypothesis, and briefly highlighted the methods and whether they were standard or novel. The reviewer went on to summarize the important results and interpret the authors’ conclusion:Unknown author et al. conducted a meta-analysis of randomized clinical trials that compared pancreaticogastrostomy (PG) versus pancreaticojejunostomy (PJ) reconstruction during a pancreatoduodenectomy. Seven randomized trials including over 1,100 patients were included. Standardized criteria for meta-analyses were applied. The studies exhibited moderately strong heterogeneity. The results demonstrated that PG was associated with a lower pancreatic fistula rate than PJ. Abdominal fluid collections were more common after PJ, but postoperative bleeding was more frequent after PG. The authors conclude that PG decreases postoperative pancreatic fistulae but does not influence mortality, morbidity, or reoperation rate.
      In the major comments section, the authors want the reviewers to state why any issue is problematic and to suggest potential alternatives to improve the work. In this section (and in the minor comments section), authors prefer comments that denote the location of the issue in the manuscript (i.e., in the results, methods, etc.; page and line numbers also really help the authors to focus on the specific issues raised). In the minor comments section, the reviewer should highlight even small errors to demonstrate the depth of the review.
      Authors tacitly implore that reviewers consider the native language of the authors and hope that they do not reject good science because of poor English. Dr Behrns’ examples included some context of “how annoying” these minor issues are to the reviewer. He made the following suggestions for reviewers:
      • 1.
        If there are only a few issues, reviewers can just point them out.
      • 2.
        If there are many issues, reviewers can indicate that the issues detract from what would otherwise be good work and require the help of an English-speaking scientific editor to make the work readable and interpretable.
      • 3.
        If the work is sloppy overall and it is evident that the manuscript was not reviewed carefully before final submission, a good way to inform the authors is to say that the carelessness in the review of the work submitted is a potential reflection of the carefulness in the investigators' experiments—this usually gets the attention of the author. A comment about whether the senior author has reviewed the work also is in order. After all, the authors attest that they all have reviewed and approved the final submission.
      In his conclusion, Dr Behrns opined that communication, collaboration, and continual improvement of our scientific work are the goals of the academic surgery community. The reviewers can help achieve these goals by signing their reviews (we understand that there are pros and cons to signing a review), providing helpful examples from experts or senior investigators, and even offering to discuss the manuscript with a junior author who has done good work but may need scientific guidance. This approach can establish a novel type of mentoring relationship. The authors’ motivation should be to advance the body of knowledge, report results, show progress, perhaps aid with their promotion and tenure, and market their program and themselves.
      In essence, the authors expect and deserve the reviewers to deliver a clear, concise, organized, and timely review that places their work in the context of the literature. The review should offer constructive criticism and options for improvements. Overall, it must satisfy the authors’ sense of a fair review.

      Top 10 list of reviewer “Not-to-Do's”

      The next discussant was David McFadden, MD, Chairman of Surgery at the University of Connecticut and Co-Editor-in-Chief of the JSR (now Editor Emeritus). The topic of Dr McFadden's talk was “Dave's Top 10 List of Reviewer ‘Not-to-Do's.’” Dr McFadden started out by exhorting the group to remember that although reviewing manuscripts is a voluntary, unpaid service, it is part of what we think of as a “culture of service” to our profession. Peer review is an age-old process established to ensure that a published paper contributes to scientific truth and that the work reflects a substantial contribution. It was first used in the modern sense in the Philosophical Transactions of The Royal Society in 1,665 and then was used routinely beginning in 1,731 in Medical Essays and Observations published by the Royal Society of Edinburgh. Since the inception of peer review, numerous practical and theoretic arguments opposing it have been suggested; each argument has counterarguments, and none has prevented its use in scientific disciplines.
      Dr McFadden then stated that reviewers should not forget the golden rule: Avoid the temptation to dismember the authors with their insightful analysis. Remembering Lord Acton's comments about power tending to corrupt, reviewers must always refrain from taking out their unrelated irritations on well-meaning, hapless authors and should bear in mind that the writing of even a dreadful paper is still hard work. The authors unquestionably view their “baby” with love and hopefully believe that they did the best they could in raising it to adulthood.
      It is uniformly better for the reviewer to direct all statements to be about the manuscript (e.g., “this manuscript suffers from a lack of attention to detail”) rather than about the authors (e.g., “the authors should have paid more attention to detail”).
      The next caveat discussed by Dr McFadden was conflict of interest. Reviewers must acknowledge any personal conflict of interest and should not review work with which they have such a conflict. Such personal conflicts of interest can include competing research interests, a competing author, an article from the reviewer's institution, or an article written by a colleague or friend (or enemy, for that matter). The review must be unbiased, and if a reviewer thinks there may be any semblance of a conflict of interest, the reviewer should politely refuse the review and inform the editor as to the reason; this protects both the reviewer and the journal.
      Dr McFadden next recommended that reviewers should not simply respond with “accept” or “reject” or provide a uninformative review. An example of this is: “I read the manuscript on hepatocyte ischemia that you sent me. I found no problems with it. I think it is the first time that this work has been done. This manuscript should be published. Recommendation: accept.” Another example is: “I read the manuscript on hepatocyte ischemia that you sent me. I found major problems with it. This is not the first time that this work has been done. This manuscript should be rejected. Recommendation: reject.” Such reviews are of no help to the authors or the editors.
      Dr McFadden next made a plea for reviewers not to accept an invitation to review a manuscript unless they intend to do the review in a timely manner. Also, reviewers should respond “yes” or “no” to the invitation quickly to allow the online manuscript management service to solicit the next reviewer in the queue, if necessary. Most journals strive for rapid turnover, so 2 weeks is generally an accepted standard. If for some reason a reviewer cannot meet the deadline in a timely manner, the editor should be notified. The editor will understand, as everyone is busy.
      Also, reviewers should try not to magnify or mention their own work in the field. Reviewers are presumably asked to review manuscripts because of their track record and expertise and should leave their egos out of the review. Stanley S. Siegelman, an editor for Radiology, once suggested that reviewers could be broadly categorized as “zealots” or “assassins.” Zealots were those who support their field and believe that any manuscript in that area deserves publication. Assassins are those who tend to believe only their work is important and that the work of others in their field is inferior. Both do a disservice to the time-honored process of peer review: the former because they are inadequately critical and the latter because they fail to acknowledge the contributions of others.
      It is important that reviewers not send mixed messages to authors and reviewers; in other words, reviewers should avoid being inconsistent with the comments they make to either party. Reviews should not laud a manuscript in the comments to the authors while disparaging it in the confidential comments to the editor. The comments to the authors should match the recommendation in the confidential comments to the editor. It is particularly challenging for an editor if a reviewer lauds a manuscript in the written review to the authors but then chooses “major revisions” or “reject” as the recommendation regarding publication. Also, reviewers should specifically avoid commenting to the authors that the manuscript is or should be accepted or rejected; that is the editor's decision, and any decision communicated by the reviewer that may differ from the ultimate decision of the editor makes it difficult for everyone.
      Finally, the ability to “cut and paste” is a blessing and a curse. Reviewers can use the antiplagiarism software of the editorial platform to assess overlap with the literature. Many journals currently use such a software program to check for plagiarism. If there is reason for concern, the reviewer should notify the editor immediately.

      Why is reviewing manuscripts important for your career?

      The final panelist was Scott A. LeMaire, MD. Dr LeMaire is Professor and Vice-Chair of Research in the Michael E. DeBakey Department of Surgery at Baylor College of Medicine (BCM) in Houston, TX. He also serves as Professor in the Department of Molecular Physiology and Biophysics, as well as the Director of Research in the Division of Cardiothoracic Surgery at BCM, and is a member of the Professional Staff of the Texas Heart Institute. He currently is the Editor-in-Chief of the JSR. The topic of Dr LeMaire's talk was titled “Why is Reviewing Manuscripts Important for Your Career?”
      Dr LeMaire first noted that reviewing manuscripts enables reviewers to develop their own science. Reviewing manuscripts will help them learn about relevant cutting-edge science. They will gain knowledge and insights into their field, related fields, others' work, and others’ perspectives on the field.
      Second, reviewers learn how to write from a reviewer's perspective. Reading and reviewing others' work, including their manuscripts and their responses to reviews, will improve their own writing. Dr LeMaire likened this opportunity to serving on a study section, which helps grant reviewers improve their own grant writing.
      Third, being a reviewer facilitates one's academic career. In many institutions, national and international recognition are crucial criteria for promotion. Reviewing articles can provide excellent visibility toward documenting one's reputation as an expert in the field, especially if the review is comprehensive and insightful. After developing a track record with an editor, a reviewer may consider asking the editor to write a letter of support for an appointment or promotion. Reviewing also may be a path toward becoming an editorial board member and beyond. Reviewers who write prompt, thorough, and insightful reviews are often invited to join editorial boards, which can be the first step to greater leadership roles with a journal and academic organizations. Dr LeMaire presented evidence that in 1998, the JSR had 4 associate editors and 50 editorial board members. Of these, 10 went on to become President of the Association for Academic Surgery or Society of University Surgeons.
      Serving as a journal reviewer also expands one's network. Editors and associate editors do notice people who provide good reviews. Other opportunities may ensue. Serving as a reviewer helps with career development by expanding one's CV and increasing national and international visibility.
      In conclusion, being a reviewer contributes to a fundamental academic process and is a key element of academic citizenship. Reviewing manuscripts helps academic surgeons develop their science and their careers and engages them in a fundamental academic endeavor. When any of us submits a manuscript for potential publication, someone must review it, so we all need to participate in the process.

      Selected reference list

        • Schulz K.F.
        • Altman D.G.
        • Moher D.
        • for the CONSORT Group
        CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
        Ann Int Med. 2010; 152: 726-732
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • The PRISMA Group
        Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA Statement.
        Int J Surg. 2010; 8: 336-341
        • Ali P.A.
        • Watson R.
        Peer review and the publication process.
        Nurs Open. 2016; 3: 193-202
        • Baig S.
        • Ahmed S.
        • Attique H.
        Reviewing a manuscript: disparity amongst peer reviewers' priorities from basic health sciences and clinicians.
        J Coll Physicians Surg Pak. 2016; 26: 677-680
        • Ezeala C.
        • Nweke I.
        • Ezeala M.
        Common errors in manuscripts submitted to medical science journals.
        Ann Med Health Sci Res. 2013; 3: 376-379
        • Bordage G.
        Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports.
        Acad Med. 2001; 76: 889-896
        • Stahel P.F.
        • Moore E.E.
        How to review a surgical paper: a guide for junior referees.
        BMC Med. 2016; 14: 29
        • Kurdi M.S.
        'Scholarly peer reviewing': the art, its joys and woes.
        Indian J Anaesth. 2015; 59: 465-470
        • Alam M.
        How to review a manuscript.
        Dermatol Surg. 2015; 41: 883-888
        • Kapoor S.
        • Sikka P.
        • Saxena K.
        Publication of research article: an art or science?.
        Ann Med Health Sci Res. 2013; 3: 96-98
        • Cornelius J.L.
        Reviewing the review process: identifying sources of delay.
        Australas Med J. 2012; 5: 26-29
        • Govender P.
        • Buckley O.
        • McAuley G.
        • O'Brien J.
        • Torreggiani W.C.
        Does online submission of manuscripts improve efficiency?.
        JBR-BTR. 2008; 91: 231-234
        • Lovejoy T.I.
        • Revenson T.A.
        • France C.R.
        Reviewing manuscripts for peer-review journals: a primer for novice and seasoned reviewers.
        Ann Behav Med. 2011; 42: 1-13
        • Alexandrov A.V.
        • Hennerici M.G.
        • Norrving B.
        Suggestions for reviewing manuscripts.
        Cerebrovasc Dis. 2009; 28: 243-246