Advertisement

Review courses for the American Board of Surgery certifying examination do not provide an advantage

      Purpose

      The purpose of this study was to evaluate whether participation in a commercially available board review course affected the likelihood of passing the general surgery certifying exam (CE) of the American Board of Surgery (ABS).

      Methods

      Candidates who took the ABS CE during the 2012–2013 academic year were surveyed electronically about their participation in commercial board review courses on the ABS website immediately before receiving their exam results. The primary outcome variable was passing the CE. Results were adjusted for background variables shown to correlate with CE pass rates using multilevel logistic regression.

      Results

      Of the 1,386 candidates who took the CE and had sufficient data for analysis, 974 of 1,064 first-time examinees (92%) and 272 of 322 repeat examinees (84%) completed the survey. Nearly 78% of survey respondents took a review course. Repeat examinees (85%) were more likely to attend a review course than first-time examinees (76%, P = .002). There were no significant differences in CE pass rates for first-time or repeat examinees who took a review course compared with those who did not (83.7% vs 80.7% for first-time examinees and 77.8% vs 69.0% for repeat examinees, P = .32 and P = .24, respectively). First-time examinee nonrespondents did not differ in their CE pass rates from those who responded to the survey (P = .113); however, repeat examinee nonrespondents had lesser CE pass rates than survey responders (P = .009). None of the review courses included had CE pass rates that differed significantly from the others after we controlled for program characteristics, ABS qualifying exam scores, medical school, and sex of the examinees.

      Conclusion

      This study used a large, prospectively collected national sample with a high response rate to study the effect of board review courses on CE performance on the ABS examination. On the basis of this survey, there was no evidence that participating in a board review course provided a benefit to passing the CE of the ABS. These results should be considered when preparation for the CE is undertaken.
      To become a physician in the United States, one must take a sequence of examinations (United States Medical Licensing Exams) offered by the National Board of Medical Examiners. Most physicians pursue additional training in graduate residency programs and proceed to achieve specialty certification from 1 or more of the 24 member boards of the American Board of Medical Specialties, which requires passing an additional written examination and, in some cases, an oral examination. Failure to pass these examinations can have important consequences for physicians, because most health care institutions require board certification to maintain hospital or practice privileges.
      • Freed G.L.
      • Dunham K.M.
      • Singer D.
      Use of board certification in hospital privileging: policies for general surgeons, surgical specialists, and nonsurgical specialists.
      In view of the high stakes of these tests, it is natural for examinees to choose various means of preparation to optimize the likelihood of passing. One commonly employed method is to participate in a review course.
      Comprehensive board review courses have become a sizable business in the United States and demand is based on the commonly held perception that these review courses provide an advantage for course participants.

      Test-prep companies see business surge. Wall Street Journal—Eastern Edition. October 10, 2001; 238: p. B2.

      Whether this perception is accurate has not yet been evaluated fully in medical specialties. Multiple reviews of the literature have found scant evidence of benefit to attending a board review course for performance on these medical examinations.
      • McGahie W.C.
      • Downing S.M.
      • Kubilius R.
      What is the impact of commercial test preparation courses on medical examination performance?.
      • Tompkins J.
      Money for nothing? the problem of the board-exam coaching industry.
      Few studies have found any benefit to attending review courses in general, and the benefit, when present, has always been small. These studies have focused typically on performance on the United States Medical Licensing Exams or medical college admission test and have not evaluated the potential effectiveness of board review courses in preparation for medical specialty certification exams. Multiple comprehensive board review courses claim that participation in their course will improve a candidate's performance on the CE, and some of these courses advertise that the pass rates of their participants exceed 90%.

      The Osler Institute. Available from: http://www.osler.org/main/gs.html. Accessed February 12, 2015.

      The purpose of our study was to evaluate whether participation in a commercially available board review course improved a candidates' likelihood of passing the general surgery certifying examination (CE) of the American Board of Surgery (ABS). The CE administered by the ABS is an oral examination that evaluates a candidate's clinical skills in organizing the diagnostic evaluation of common surgical problems and determining appropriate therapy. The examination focuses on the application of knowledge to clinical problems, evaluation of surgical judgment and decision making, management of complications, and assessment of technical knowledge.

      American Board of Surgery. Available from: www.absurgery.org/default.jsp?certcehome. Accessed February 12, 2015.

      This CE is the final step toward certification in surgery. We hypothesized that participation in these courses would not improve the pass rate for candidates taking the CE.

      Methods

      The ABS administered an online survey to all candidates who took the 5, ABS general surgery CEs between October 2012 and May 2013. Candidates were prompted to respond to the survey immediately before they obtained their CE results from the ABS Web site. The survey asked about a candidate's attendance at specific review courses directed at the general surgery board exam and self-study courses, including the American College of Surgeons Comprehensive General Surgery Review Course, Clinical Performance and Oral Exams in Surgery, Exam Master, Med-Challenger Surgery, New York General Surgery Review Course, Odyssey, Osler Institute Board Review, Surgical Education and Self-Assessment Program, and the Pass Machine as well as additional methods of preparation. These courses were selected based on the responses of previous CE candidates to an anonymous survey and Internet searches for surgery board review courses. Candidates had the option to select “I choose not to answer” in the event they were not comfortable disclosing course attendance to the ABS. Participants were also asked their opinion about whether the board review course improved their preparation for the examination.
      Participation in these review courses was compared to examination performance to determine the effect of taking one or more board review courses on passing the CE. The primary outcome variable was the CE pass rate. Those who took one or more review courses were compared with examinees who did not take a course. In addition to the primary outcome variable, background characteristics of candidates and their surgery residency training programs were included in the analyses. These parameters were used to control for other factors that have been shown to influence examination pass rates and to help isolate the effects of attendance at review courses on CE performance. Individual variables included sex, undergraduate medical education (United States or international medical school graduate, USMG and IMG, respectively), and ABS Qualifying Examination (QE) scale scores.
      • Biester T.W.
      • Rubright J.D.
      • Jones A.T.
      • Malangoni M.A.
      Does success on American Board of Surgery general qualifying exam guarantee certifying examination success?.
      • Jones A.T.
      • Biester T.W.
      • Busyke J.
      • Lewis F.R.
      • Malangoni M.A.
      Using the American Board of Surgery in-training examination to predict board certification: a cautionary study.
      The QE scale score is an equated raw score that has a fixed passing score of 75 every year and has been demonstrated to be an important proxy for a candidate's knowledge.
      • Biester T.W.
      • Rubright J.D.
      • Jones A.T.
      • Malangoni M.A.
      Does success on American Board of Surgery general qualifying exam guarantee certifying examination success?.
      Program characteristics assessed included program size, type, and the program's average QE score over the most recent 5-year period. On univariate analysis, these program level variables accounted for sufficient variance to warrant inclusion as controls in the model (P < .10).
      Certifying examination scores were determined by summing the grades of 6 ABS examiners who evaluated each candidate independently according to predetermined set of criteria. Examinees must have obtained a threshold of performance to pass the examination.

      Data analysis

      The effect of individual course participation was analyzed by the use of multilevel regression models for the CE pass rate. Five courses were excluded from the analysis because of small sample sizes (n ≤ 40), including 2 in which no one participated. The alpha level for tests of significance was set at P < .01 based on the large sample size and multiple predictors included in the model. For concerns that this value is too conservative, the Bonferroni corrected alpha would be P < .005, a threshold that is more stringent than the value we selected. We also conducted a post-hoc power analysis for detecting an odds ratio of 1.5 for each course in SAS 9.3 (SAS Institute, Cary, NC) and found a >99% chance of detecting this effect with the study sample size at an alpha level of .01. Review courses were entered as independent variables along with covariates that have demonstrated a relationship with examination outcomes either in published literature or on univariate anlaysis.
      • Biester T.W.
      • Rubright J.D.
      • Jones A.T.
      • Malangoni M.A.
      Does success on American Board of Surgery general qualifying exam guarantee certifying examination success?.
      • Jones A.T.
      • Biester T.W.
      • Busyke J.
      • Lewis F.R.
      • Malangoni M.A.
      Using the American Board of Surgery in-training examination to predict board certification: a cautionary study.
      Although information about specific courses was included in the survey questions, the results are presented anonymously.

      Results

      There were 1,396 candidates who took one of five ABS CEs offered from October 2012- June 2013, including 1,067 candidates who took the examination for the first time and 329 who had failed the CE previously and were repeat examinees. The overall pass rate for the CE was 79.9%; however, the pass rate differed between first-time and repeat examinees (82.1% vs 72.6%; χ2 (1) = 14.0, P < .001). There were 1,256 candidates (90%) who responded to the online survey (Fig 1). Candidates who chose not to answer the survey had lower initial QE scores (t [1,394] = 2.67, P = .008). Ten responders were excluded from the analysis due to missing background information. In both the overall group and among first-time examinees, women were more likely to attend a board review course (Table I).
      Table IDemographics of candidates taking a board review course
      Demographic variableOverall groupFirst-time examineesRepeat examinees
      Total, N (% Pass)Attended a review course, n (%)
      Percent who attended a course.
      Total, N (% Pass)Attended a review course, n (%)
      Percent who attended a course.
      Total, N (% Pass)Attended a review course, n (%)
      Percent who attended a course.
      Sex
      Significant differences in board review course attendance for overall group (P = .01) and first time examinees (P = .003) for sex of the participant.
       Male862 (80.2%)654 (75.9%)663 (81.1%)486 (73.3%)199 (77.4%)168 (84.4%)
       Female384 (84.4%)317 (82.6%)311 (86.8%)255 (81.9%)73 (74.0%)62 (84.9%)
      Medical education
       USMG978 (83.3%)759 (77.6%)767 (84.5%)578 (75.4%)211 (79.1%)181 (85.8%)
       IMG266 (74.8%)210 (78.9%)205 (77.1%)161 (78.5%)61 (67.2%)49 (80.3%)
      Program size
       2121 (82.1%)89 (73.6%)93 (81.0%)65 (69.9%)28 (85.7%)24 (85.7%)
       3209 (81.8%)110 (52.6%)149 (85.9%)59 (39.6%)60 (71.7%)51 (85.0%)
       4204 (78.4%)162 (79.4%)158 (79.7%)130 (82.2%)46 (73.9%)32 (69.6%)
       5183 (81.4%)147 (80.3%)146 (84.2%)114 (78.0%)37 (70.3%)33 (89.2%)
       6213 (87.3%)156 (73.2%)167 (87.4%)115 (68.9%)46 (87.0%)41 (89.1%)
       7 or more316 (83.9%)249 (78.8%)261 (85.9%)200 (76.6%)55 (74.5%)49 (89.1%)
      Program type
       University736 (82.2%)570 (77.4%)583 (84.2%)441 (75.6%)153 (74.5%)129 (84.3%)
       Independent458 (82.6%)365 (79.7%)345 (80.9%)267 (77.3%)113 (87.9%)98 (86.7%)
       Military34 (97.1%)28 (82.4%)30 (96.7%)27 (90.0%)4 (100.0%)1 (25.0%)
      IMG, International Medical Graduates; USMG, United States Medical Graduates.
      Percent who attended a course.
      Significant differences in board review course attendance for overall group (P = .01) and first time examinees (P = .003) for sex of the participant.
      Of the 1,067 candidates who took the CE for the first time in 2012–2013, 977 completed the survey for a response rate of 91.6%. Three examinees had missing background information, leaving 974 first-time examinees for the final analyses (Fig 1). First-time examinees who did not answer the survey had a pass rate of 75.6%, whereas the pass rate for respondents was 82.7%, not statistically significant (χ2 [1] = 2.86 P = .11). Female first-time examinees were less likely to respond to the survey than males (88.4% vs 93.1%; χ2 [1] = 6.91, P = .009); however the pass rate for females who responded to the survey was similar to first-time male examinees. There were no other differences in the background or program characteristics of first-time examinees who responded to the survey.
      Of the 329 repeat examinees, 279 completed the survey (response rate, 84.8%). Seven candidates had missing background information and were excluded from the analyses. Repeat examinees who did not answer the survey had a pass rate of 58.0% while respondents had a greater pass rate (75.3%; χ2 (1) = 6.36, P = .01).
      Nine hundred seventy-one of 1,246 examinees took a review course (77.9%), including less first-time examinees versus repeat examinees (76.1% vs 84.6%; χ2 [1] = 8.9, P = .002). The overall CE pass rate did not differ between those who did and those who did not participate in a review course (82.7% vs 78.9%; χ2 [1] = 1.7, P = .22). The first-time CE pass rate also did not differ for those who did and those who did not participate in a board review course (83.7% vs 80.7%; χ2 [1] = 1.1, P = .32). For repeat examinees, again the CE pass rate did not differ for those who and those who did not participate in a review course (77.8% vs 69.0% χ2 [1] = 1.60, P = .24).
      There was no relationship between course attendance across residency programs of different sizes or by undergraduate medical education status (Table I). The number of examinees who participated in the various review courses ranged from 21 to 706, and pass rates for participants in the various courses ranged from 67 to 83.1% (Fig 2).
      Figure thumbnail gr2
      Fig 2Frequencies and CE pass rates for each review course. *Total does not match total numbers from review courses because some individuals took multiple review courses.
      The mean number of courses taken was 1.25 and 1.54 for first-time and repeat examinees, respectively; 243 examinees (25%) participated in multiple courses (Fig 3). There was no apparent advantage to taking multiple review courses. First-time and repeat examinees who participated in board review courses had similar baseline ability based on mean QE scores (data not shown).
      Figure thumbnail gr3
      Fig 3Frequencies and CE pass rates by number of courses attended.
      Table II shows the results of the multilevel logistic regression model for first-time examinees. Board review courses were dummy coded so that the estimates in the first column of the table were equivalent to the increase or decrease in scores or odds of success from participating in the specified course after accounting for the other variables in the model. Participation in any course did not significantly predict passing the CE. Only QE scaled score was a significant predictor of passing the CE.
      Table IIMultilevel logistic regression analysis for certifying examination (first-time examinees)
      PredictorEstimateSEOdds ratio (99% CI)P value
      Threshold−2.000.250.14<.001
      Course D1.180.523.25 (0.86–12.22).02
      Course E−0.180.320.84 (0.37–1.93).59
      Course F−0.020.240.98 (0.53–1.79).92
      Course G0.300.201.35 (0.81–2.25).13
      International medical graduate−0.100.150.15 (0.62–1.33).51
      Sex−0.420.210.66 (0.38–1.14).05
      QE score0.090.021.09 (1.05–1.14)<.001
      Program size0.020.081.02 (0.84–1.24).77
      University−0.040.130.49 (0.70–1.34).77
      Independent−0.090.140.91 (0.64–1.31).52
      Program average QE score0.040.041.04 (0.95–1.15).24
      CI, Confidence interval; QE, qualifying examination.
      Results of the multilevel logistic regression model for repeat examinees are presented in Table III. Similar to first-time examinees, no course was a significant predictor of passing the CE. None of the 7 review courses had a pass rate that differed significantly from the others after controlling for program characteristics, ABS QE scores, medical school, and sex of the examinee.
      Table IIIMultilevel logistic regression analysis for certifying examination (repeat examinees)
      PredictorEstimateSEOdds ratio (99% CI)P value
      Threshold−0.720.410.48 (0.17–1.39).08
      Course D0.040.451.04 (0.33–3.33).93
      Course E0.130.371.13 (0.43–2.97).74
      Course F0.510.421.67 (0.57–4.90).22
      Course G0.470.321.60 (0.71–3.61).14
      International Medical Graduate−0.760.360.47 (0.18–1.19).04
      Sex0.520.371.68 (0.66–4.33).16
      QE score0.060.031.07 (0.98–1.15).04
      Program size−0.020.120.98 (0.71–1.34).85
      University−0.110.240.90 (0.48–1.68).66
      Independent0.220.261.25 (0.64–2.42).39
      Program average QE score0.110.051.12 (0.98–1.28).03
      QE, Qualifying examination.
      Sixty-four percent of examinees expressed their beliefs that attending a board preparation course improved their preparation for the exam, 19% did not think the review course was helpful, and 17% chose not to answer the question.

      Discussion

      This study has a number of important findings. It has been evident for years that candidates taking the CE for the first time have a greater pass rate than those who failed an earlier attempt. In addition, the pass rates of first-time examinee have had less variability over time. The response rate to our survey exceeded 90%, which is extremely high. Less than 1% of survey respondents were excluded as the result of missing data. Both of these characteristics suggest strongly that this analysis is highly representative of the examination patterns of preparation of candidates for the CE examination. Although the percentage of candidates who did not respond to the survey was small, this group differed from respondents. Their pass rate was lower for both initial and repeat examinees, and this disparity was statistically significant among repeat examinees. Females who took the CE for the first time were less likely to respond to our survey; however, their exam performance was not different than male first-time examinees.
      The majority of candidates attended a review course. This rate was even greater for repeat examinees (85%). The greater course attendance by repeat examinees is not surprising, because the CE is a high-stakes examination, and these examinees would likely be searching for more strategies to pass the exam having failed their first attempt. An internal report conducted by the ABS in 1990 showed that only 29% of CE candidates participated in a formal board exam review course.
      American Board of Surgery
      June 1990 Board of Director’s Agenda Book.
      The increase in participation over the last 2 decades follows the trend for test review courses for education in general, which has become a multibillion dollar industry.

      Test-prep companies see business surge. Wall Street Journal—Eastern Edition. October 10, 2001; 238: p. B2.

      Van Buskirk E. Former test-prep exec plots industry makeover. Wired December 3, 2009. Available from: http://www.wired.com/business/2009/12/test-prep-internet. Accessed February 12, 2015.

      Each course evaluated in this study either has the preparation of candidates for the CE as a stated objective, or at least 50 candidates responded that they used the course to prepare for the CE. With such high participation rates among candidates for the CE, it is imperative to examine the potential benefits that board review courses have on success on examinations. Despite wide variability in the popularity of the review courses, there was no significant difference in CE pass rates among the various courses. Although not all courses reviewed in this study claim to improve the CE pass rate, those review courses that publicize their pass rates claim much greater success than that documented in our study. The difference in examination pass rates for those taking a review course was only 3% for first-time examinees and 8.8% for repeat examinees, neither of which was statistically significant. We did not find that participation in multiple courses improved the likelihood of passing the CE.
      This study is unique in that it examines comprehensively the effect of participation in a board review course directed at performance on an oral examination in a medical specialty. The representative sample of examinees who responded to our survey was large and included both initial and repeat examinees. We controlled for examinee characteristics that have demonstrated previously a relationship with ABS exam performance. Greater emphasis should be placed on the results for first-time examinees, because the sample size is greater and their attendance at review courses would likely be proximate to attempting the CE. Because the response rate to the survey was so high and the exam pass rate was similar to first-time examinees who did not respond to our survey, this cohort should be a highly representative sample for studying the effects of attending a review course. Furthermore, the analyses controlled for potentially confounding factors, such as candidate and program demographics and QE performance. These results should help inform candidates about the relative utility that taking a review course may have in helping them achieve their goal of passing the CE. Given their not-insignificant cost, an independent evaluation of the efficacy of specialty board review courses should provide helpful information for candidates to guide their decision about investing the time, effort, and resources in participation in a review course. Although the majority thought that taking a review course improved their preparation for the exam, more than one third did not believe their experience was helpful or offered no opinion.
      There are limitations to this study. Most importantly, the CE pass rate was less for both first-time and repeat examinees who did not respond to the survey. If one assumes that both of these groups took a review course at the same rate as the responders, the percentage differences in exam performance becomes less for both groups. Course attendance was self-reported, so it is possible that survey respondents reported this inaccurately. We believe that this would be unlikely, because the survey was administered within a few days after the examination and before the results were revealed. Only a few percent of respondents claimed they were engaged in “other” types of examination preparation activities beside courses listed on the survey. Review of the details for that group did not reveal a consistent pattern of preparation. The smaller number of repeat examinees creates limitations in the analysis. Repeat examinees who did not respond to the survey had greater rates of failure and the temporal relationship between when a review course was taken relative to the date of the examination may be more variable. We assumed that first-time examinees took a board review course close to the time of attempting the CE. It is also possible that the summative effects of non-significant differences in baseline ability (QE scores) in combination with other factors not measured in this study could have affected the results. The lesser pass rates among those who did not respond to the survey suggest we need more information about course participation by that group. It is possible that differences in exam pass rates attributable to taking a review course would be different if course participation information from non-responders was available.
      In summary, participation in a review course did not appear to provide a statistically significant advantage to passing the CE of the ABS. Additionally, no specific review course differed from the others in this comparison. Candidates should consider these results when assessing how to best prepare for the CE. Future studies should examine the effects of participating in a review course on written examinations in surgery as well as certification in other medical specialties to gain a better understanding of how these results might be applicable to other settings.

      References

        • Freed G.L.
        • Dunham K.M.
        • Singer D.
        Use of board certification in hospital privileging: policies for general surgeons, surgical specialists, and nonsurgical specialists.
        Arch Surg. 2009; 144: 746-752
      1. Test-prep companies see business surge. Wall Street Journal—Eastern Edition. October 10, 2001; 238: p. B2.

        • McGahie W.C.
        • Downing S.M.
        • Kubilius R.
        What is the impact of commercial test preparation courses on medical examination performance?.
        Teach Learn Med. 2004; 16: 202-211
        • Tompkins J.
        Money for nothing? the problem of the board-exam coaching industry.
        N Engl J Med. 2011; 365: 104-105
      2. The Pass Machine. Available from: http://www.thepassmachine.com/general-surgery-certifying-oral-board-online-course.html. Accessed February 12, 2015.

      3. The Osler Institute. Available from: http://www.osler.org/main/gs.html. Accessed February 12, 2015.

      4. Odyssey. Available from: http://www.surgery-board-review.com/oral-exam-preparation/general-surgery/prep-course-schedule.html. Accessed February 12, 2015.

      5. American Board of Surgery. Available from: www.absurgery.org/default.jsp?certcehome. Accessed February 12, 2015.

        • Biester T.W.
        • Rubright J.D.
        • Jones A.T.
        • Malangoni M.A.
        Does success on American Board of Surgery general qualifying exam guarantee certifying examination success?.
        J Surg Educ. 2012; 69: 731-734
        • Jones A.T.
        • Biester T.W.
        • Busyke J.
        • Lewis F.R.
        • Malangoni M.A.
        Using the American Board of Surgery in-training examination to predict board certification: a cautionary study.
        J Surg Educ. 2014; 71: e144-e148
        • American Board of Surgery
        June 1990 Board of Director’s Agenda Book.
        American Board of Surgery, Philadelphia, PA1990: 6
      6. Van Buskirk E. Former test-prep exec plots industry makeover. Wired December 3, 2009. Available from: http://www.wired.com/business/2009/12/test-prep-internet. Accessed February 12, 2015.