If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements.
Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents.
Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training.
We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice.
Concerns have been raised about the effectiveness of current general surgery resident training in adequately preparing residents for independent practice,
including residents' ability to perform core general surgery procedures independently. Currently the final summative evaluation of each resident's operative competence, which is required in order to take the American Board of Surgery (ABS) qualifying exam, is based primarily on 2 things. The first is the cumulative end-of-rotation assessments, which combine judgment of operative performance with that of other performance parameters. This assessment process is not standardized, and there is significant variability between programs, including the assessment methods used for operative performance. Within programs, operative performance assessments are typically not completed consistently or in a timely manner by faculty.
The second requirement is achievement of minimal procedural numbers as defined by the Accreditation Council for Graduate Medical Education (ACGME)/ABS. These number requirements are not based on individual procedures, but rather on defined categories of related procedures. No data currently exist to demonstrate that there is correlation between these defined category numbers requirements and individual resident competence with specific procedures within the same defined category.
Furthermore, there is limited evidence supporting reliable correlation between the numbers of a specific procedure that trainees are exposed to and the achievement of procedural competence for that specific procedure.
before they can be entrusted to perform this activity independently in clinical practice. This determination is therefore not necessarily dependent on the number of procedures they have performed. Performance of core operative procedures is perhaps the most essential EPA for surgical training. Resident autonomy in performing core operative procedures is a critical indicator of competency because supervising surgeons are unlikely to grant significant autonomy to a resident unless they believe he or she is competent. In this study we will attempt to determine if there is correlation between the numbers of core procedures performed by surgery residents and the level of competency they achieve with that procedure.
Resident operative performance assessments were collected between September 1, 2015, and December 31, 2017, at 14 general surgery residency training programs belonging to the Procedural Learning and Safety Collaborative (PLSC). Data were collected using SIMPL, a smart phone app-based system that was developed to facilitate timely assessment of residents' operative autonomy and performance with every procedure they participate in.
SIMPL assessments include 3 scales used to determine 1) the autonomy level achieved by the resident during the procedure using the Zwisch scale (Fig 1); 2) the performance level of the resident during the operative procedure
After the completion of a surgical procedure the SIMPL assessment process is initiated by the resident (or the supervising surgical faculty member) who first identifies the faculty surgeon (or resident) that they just operated with from a drop-down list in the app. Next they identify the procedure that was performed from a procedure list derived from the ACGME case log taxonomy. To ensure that assessments are performed in a timely manner, those that are not completed within 72 hours of initiation cannot be entered into the SIMPL database.
To facilitate further development of the SIMPL assessment process and collaborative research efforts using the data collected by the app, a consortium of general surgery residency programs known as the PLSC was formed, which, at the time of this study, consisted of 14 U.S. general surgery residency training programs. Before being granted access to SIMPL, all participating surgical faculty and residents at each of these programs were required to participate in a 1-hour frame of reference training session to confirm common understanding of how to use the SIMPL app,
including the 3 assessment scales discussed earlier. Furthermore, individual programs were required to confirm that ≥70% of both residents and faculty had completed this training before general programmatic access to SIMPL being granted to trained users.
Although all procedures performed by a categorical general surgery resident with a supervising surgical faculty attending were included in SIMPL, for this study, our analyses focused on the 5 procedures that were most frequently assessed during the study period (i.e. laparoscopic cholecystectomy, laparoscopic appendectomy, open inguinal hernia repair, ventral hernia repair, and partial colectomy) and that were considered “core” general surgery procedures by national surgical leadership.
This study included data collected from all participating categorical general surgery residents (PGY1-5). For each individual resident participating in the study, a mean Zwisch score was determined based on the autonomy level they achieved during their total experience with each of these 5 procedures they performed during the entire study period. Only residents who had performed the procedure being evaluated at least 3 times were included in the analyses for that specific procedure. Additionally, the number of each of these procedures each resident had performed from the start of their residency until the start of the study period was collected from their ACGME case logs. For each of the 5 core procedures evaluated, individual residents' mean Zwisch scores achieved during the study period were plotted against the number of procedures they had performed before the study period. We determined the best fitted linear line to summarize the relationship between prior procedural experience and autonomy level achieved. Meaningful autonomy was defined as having achieved a mean Zwisch score of ≥3 (based on numerical conversion of the Zwisch scale: i.e. Show and Tell = 1, Active Help = 2, Passive Help = 3, Supervision Only = 4) for the procedure in question during the study period. The point where the best-fit line intersected with this meaningful autonomy level was used to determine the number of procedures at which the majority of the resident population should achieve meaningful procedural autonomy. We refer to this as the “transition point” (Table and Figs 2A–2E). All analyses were performed using R: a language and environment for statistical computing, version 3.2.2.
TableRelationship of Procedural Numbers with Meaningful Autonomy in Surgical Residents
Inception point (IP): The number of procedures at which the resident population first begins to achieve meaningful autonomy, exclusive of performance outliers (i.e. residents who achieved meaningful autonomy after performing ≥5 fewer procedures than any other resident required to achieve meaningful autonomy).
Transition point (TP): Based on the line of “best fit” assigned to the plotted data points, the number of procedures at which the majority of the resident population transitions to meaningful autonomy.
Based on the Zwisch scale of progressive autonomy, the 2 highest autonomy levels (i.e. Passive Help and Supervision Only).
beyond TP (n)
Open Inguinal Hernia Repair
Ventral Hernia Repair
* Includes all participating residents who had performed the procedure ≥3 times.
† Inception point (IP): The number of procedures at which the resident population first begins to achieve meaningful autonomy, exclusive of performance outliers (i.e. residents who achieved meaningful autonomy after performing ≥5 fewer procedures than any other resident required to achieve meaningful autonomy).
‡ Transition point (TP): Based on the line of “best fit” assigned to the plotted data points, the number of procedures at which the majority of the resident population transitions to meaningful autonomy.
§ Based on the Zwisch scale of progressive autonomy, the 2 highest autonomy levels (i.e. Passive Help and Supervision Only).
We also sought to establish at what number of procedures the resident population first begins to achieve meaningful autonomy for each procedure. We refer to this as the “inception point” (Table and Figs 2A–2E) and define it as the number of procedures at which meaningful autonomy was first achieved by a trainee. To avoid outlier performances in defining the inception point, we excluded any resident performances where meaningful autonomy was achieved in isolation (i.e. no other trainees achieved meaningful autonomy despite participating in at least 5 additional procedures).
During the study period we collected SIMPL assessments for 10,130 total surgical performances (332 different types of procedures) by 536 surgical residents who were assessed by 444 supervising surgical faculty. Of the SIMPL assessments performed, 73.4% (7,437) involved procedures categorized as “core” for general surgery.
For laparoscopic appendectomy, 55 residents performed ≥3 procedures during the study period. Of these, 31 (56.4%) achieved a mean autonomy level of ≥3. Based on the best-fit line of the intersection point with the meaningful autonomy line, the “transition point” was determined to be at 25 procedures, and all 22 (100%) of the residents who performed more than 25 laparoscopic appendectomies achieved meaningful autonomy (Fig 2A). With laparoscopic cholecystectomy, 89 residents performed ≥3 procedures during the study period. Of these, 41 (46.1%) achieved a mean autonomy level of ≥3. The transition point was determined to be at 52 procedures, and 17 (73.9%) of the 23 residents who had performed this many procedures achieved meaningful autonomy (Fig 2B). With open inguinal hernia repair, 54 residents performed ≥3 procedures during the study period. Of these, 21 (38.9%) achieved a mean autonomy level ≥3. The transition point was determined to be at 42 procedures. However, only 2 (28.6%) of the 7 residents who performed this many procedures achieved meaningful autonomy (Fig 2C). For ventral hernia repair, 40 residents performed ≥3 procedures during the study period. Of these, 14 (35%) achieved a mean autonomy level of ≥3. The transition was determined to be at 35, although only 2 (50%) of the 4 residents who performed this many procedures achieved meaningful autonomy (Fig 2D). For partial colectomy, 39 residents performed ≥3 procedures during the study period. Of these, 11 (28.2%) achieved a mean autonomy level of ≥3. The transition point was determined to be at 60, but only 2 residents performed this many procedures, although both (100%) achieved meaningful autonomy (Fig 2E). These results are summarized in Table.
The procedural performance numbers that reflect when the residents first started to achieve meaningful autonomy (i.e. “the inception point”) are also summarized in Table. For laparoscopic appendectomy, the inception point occurred at 13 procedures, as none (0%) of the residents who had performed <13 procedures achieved meaningful autonomy, whereas 36 (87.8%) of those who performed ≥13 procedures achieved meaningful autonomy. For laparoscopic cholecystectomy, the inception point was 19 procedures, as none (0%) of the residents who had performed <19 procedures achieved meaningful autonomy, whereas 39 (62.9%) of those who performed ≥19 procedures achieved meaningful autonomy. With open inguinal hernia repair, the inception point was 9, as only 1 (6.25%) of the residents who performed <9 procedures achieved meaningful autonomy, whereas 20 (52.6%) of those who performed ≥9 procedures achieved meaningful autonomy. With ventral hernia repair, the inception point was 20 procedures, as none (0%) of the residents who performed <20 procedures achieved meaningful autonomy, whereas 14 (60.8%) of those who performed ≥20 procedures achieved meaningful autonomy. Finally, with partial colectomy, the inception point was 23 procedures, as only 1 (5.6%) of the residents who performed <23 procedures achieved meaningful autonomy, whereas 10 (47.6%) of those who performed ≥23 procedures achieved meaningful autonomy.
The findings in this study suggest that specific procedural target numbers that correlate with the achievement of procedural competency in surgical residency training may be identifiable for some core surgical procedures, but will likely be difficult to define for most of the 132 core procedures. We identified what appear to be reliable target numbers for laparoscopic appendectomy and laparoscopic cholecystectomy, the two most commonly performed procedures. With laparoscopic appendectomy, the target number was most reliable, as all (100%) residents who performed at least 25 procedures achieved meaningful autonomy. With laparoscopic cholecystectomy, a significant majority (73.9%) of residents who performed at least 52 procedures achieved meaningful autonomy. Although it is beyond the scope of this study, it will be essential to attempt to establish why the other 26.1% of residents who performed ≥52 procedures did not achieve meaningful autonomy with laparoscopic cholecystectomy. Because this is the most frequently performed procedure by general surgery residents, procedural numbers may be potentially less of an impediment in attaining procedural autonomy than with other procedures, and other factors may have impeded resident progression to meaningful autonomy. Although individual resident ability may be an important factor, the level of autonomy an individual resident achieves may also be influenced by faculty rater stringency, faculty willingness to grant autonomy, programmatic culture, programmatic patient populations, and other factors.
With open inguinal hernia, ventral hernia, and partial colectomy, the target numbers identified using the “best-fit” process need to be further refined once additional observations are available. With open inguinal hernia and ventral hernia, the target numbers did not clearly identify a transition point beyond which the majority of residents achieved meaningful autonomy. For partial colectomy, technically this transition point was identified by the target number, although the majority of residents who achieved meaningful autonomy beyond the target number consisted of only 2 residents. These analyses may be further enhanced by increased data collection.
Another concern is that the procedural target numbers identified using this approach may be higher than the numbers that most residents may experience during their entire residency. We have previously shown that in the last 6 months of their training PGY5 residents achieve meaningful autonomy, with only 69% of the core general surgery procedures they perform,
although higher percentages (≥85%) are achieved with the 4 most frequently performed procedures. For less frequently performed core procedures, it may be unrealistic to expect all surgical trainees to achieve these defined targets during their training.
One limitation of this analysis is the lack of a widely accepted objective standard for defining procedural competence. The criteria we have chosen for this study are based on the mean level of autonomy achieved by the resident performing the procedure in question during the defined study period. Because the autonomy level achieved by the resident is entirely dependent on the autonomy level granted by the supervising faculty, higher levels of autonomy are typically granted only when the resident has gained the trust of the supervising faculty based on their demonstration of competence. One might argue that there is no stronger endorsement of a resident's competence by a supervising surgeon than to let the resident do the procedure with a high level of autonomy. Still, other factors, including procedural complexity, programmatic culture, and individual faculty demeanor, may also influence the amount of autonomy granted, and this may confound our results.
Some express significant concern about granting residents significant autonomy during their training for patient safety reasons, because they are less experienced and therefore less competent than their supervising faculty surgeon. Others believe that granting supervised autonomy during training to ensure residents are ready for independent practice should be considered more acceptable than sending them out into independent practice unprepared and that increasing the focus on progressive autonomy during training may lead to more residents achieving meaningful autonomy during their training.
We have used a mean Zwisch level ≥3 to define meaningful autonomy, which essentially means that the resident has achieved either Passive Help (PH) or Supervision Only (SO) for the majority of times they have performed that procedure during the study period. Although SO may be a more ideal goal for trainees because it most closely mimic independent practice, PH also reflects meaningful resident autonomy and may be a more comfortable scenario for supervising faculty. Furthermore SO requires that someone other than the supervising faculty assists the resident with the procedure, which is not always logistically possible. Furthermore, if SO was considered the required criteria for meaningful autonomy, the procedural number requirements would be even higher and the number of residents who achieve those levels would be lower. For these reasons we have included both PH and SO in the definition of meaningful autonomy.
Regardless, the criteria we have used in this study to define meaningful autonomy may reflect a higher degree of autonomy than is currently being granted to most surgical residents. It is possible that the criteria we have used to define meaningful autonomy for this study is too stringent and may exclude many residents who are actually competent. Perhaps instead of basing the threshold for meaningful autonomy on achieving PH or SO for the majority of procedures performed during a specific period, it may be reasonable to consider basing it on achieving PH or SO for a designated minimal number of performance assessments (i.e. 3), as is the established standard in some other countries.
Of course these or other less stringent criteria would need to be validated in a separate study.
Although we have focused on how a trainee's competence with a specific operative procedure may be influenced by their previous experience with that procedure, we have not looked at the impact of experience with other procedures. This impact of transfer of surgical skills
Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency.
from other surgical procedures has never been adequately studied in operative procedures performed during residency training, but is the basis for the ACGME-defined category requirements. These requirements stipulate that surgical residents perform a minimum number of operative procedures from a defined group of procedures before they are allowed to take their ABS qualifying exams. Specifically, transfer of surgical skills may be a factor for residents who obtain the majority of their procedural experience for a specific procedure during the PGY4 or PGY5 year. Due to a greater overall procedural experience preceding these final years of surgical residency training, these residents may be able to more rapidly achieve meaningful autonomy with a specific procedure than residents who are less experienced.
Ultimately the purpose of establishing target numbers for procedures needs to be carefully re-evaluated. If the intent is to distinguish those in training who have achieved an acceptable level of autonomy from those who have not for the purposes of determining who is eligible for certification, our data suggest that this may work for only a very limited number of individual core procedures. With most procedures, however, high levels of autonomy are not currently achieved by most residents, and those who do achieve them often have done so after performing procedural numbers that are higher than many residents may have access to.
However, if target numbers based on individual procedures or defined categories are used as training milestones to trigger more in-depth review of individual residents' operative competence, this may be an effective way to guide subsequent training for residents by directing those who have achieved competence in a specific procedure to shift their focus to other procedures and allowing those who have not yet achieved competence to obtain extra experience with the procedure in question. Identification of procedural numbers that are associated with the earliest and most significant transitions to meaningful procedural autonomy in the resident population, that is, both the inception and transition points, as we have done here (Figs 2A–2E and Table) may be helpful in guiding this process.
Our data therefore argue against general use of a fixed procedural number as a surrogate for declaring procedural competence for specific procedures. Although procedural numbers are easy to monitor and evaluate, if these numbers are not used in conjunction with a system that accurately monitors progressive competence, they can be misleading in determining who is ready for independent practice. There appears to be some correlation between procedural numbers and progressive autonomy, but there is such a high degree of variability between individual residents that it is likely not possible to select a single procedural target number that would reliably be used to predict procedural competence for all trainees. Therefore serious consideration needs to be given to converting to a system that is based primarily on assessment of competence in the clinical workplace rather than based solely on a procedural target number.
Finally, with ongoing monitoring of residents' progressive autonomy, problems can be identified at the level of individual residents, at the level of programs, and at a national level. To address problems that are identified at any of these levels, interventions can be deployed and their impact monitored. Faculty can also be monitored to identify problems in granting autonomy and initiate appropriate faculty development initiatives to resolve these problems.
Operative experience of residents in US general surgery programs: a gap between expectation and experience.
Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency.
Supported by grants from the American Board of Surgery , Massachusetts General Hospital , and Indiana University and contributions from members of the Procedural Learning and Safety Collaborative (PLSC).
Presented at the Central Surgical Association 2017 Annual Meeting, Chicago, IL, July 26–August 1, 2017.