Advertisement

Defining operative experience targets in surgical training: A systematic review

Open AccessPublished:August 26, 2022DOI:https://doi.org/10.1016/j.surg.2022.07.015

      Abstract

      Background

      The surgical learning curve is an observable and measurable phenomenon. Operative experience targets are well established as a proxy measure for operative competence in surgical training across jurisdictions. The aim of this study was to critique the available evidence regarding the relationship between operative experience in surgical training and trainee competence.

      Methods

      A systematic review of the PubMed, Embase, Web of Science, and Cochrane library databases was conducted in accordance with the Preferred Items for Systematic Reviews and Meta-Analyses guidelines. Articles were sought that defined the relationship between procedural volume in surgical training and trainee competence, proficiency, or mastery. The educational impact of included studies was evaluated using a modified Kirkpatrick model.

      Results

      Of 3,672 records identified on database searching, 30 papers were ultimately included. Fourteen studies defined operative experience thresholds using operative time as a surrogate measure of competence, whereas another 8 used trainer assessments of operative performance (Kirkpatrick level 3). A further 5 studies were able to determine the relationship between trainee case volumes and subsequent patient outcomes (Kirkpatrick level 4b).

      Conclusion

      Many studies have recorded competent trainee performance in key index procedures after reaching experience threshold numbers in excess of currently mandated targets across jurisdictions. The evidence relating current operative experience targets to patient outcomes across a range of surgical subspecialties of surgical subspecialties is lacking. This review supports a move toward criterion-based referencing of operative performance targets in surgical training.

      Introduction

      Competency-based education curricula require surgical training programs to certify the outcome abilities of graduating residents.
      • Gervais J.
      The operational definition of competency-based education.
      The competent performance of operative procedures is a core capability expected of surgeons at the end of training. The surgical ‘learning curve’ is an observable and measurable phenomenon,
      • Hopper A.N.
      • Jamison M.H.
      • Lewis W.G.
      Learning curves in surgical practice.
      and the relationship between higher surgeon volume and improved patient outcomes has been reported for many procedures across subspecialties.
      • Morche J.
      • Mathes T.
      • Pieper D.
      Relationship between surgeon volume and outcomes: a systematic review of systematic reviews.
      Minimum operative experience targets are therefore attractive as proxy measures of competence. Concerns exist regarding the reduced working hours of surgical residents, the concomitant reduction of operative exposure and training opportunities, and the potential downstream effects of these phenomena on both the confidence and competence of graduating trainees.
      • Friedell M.L.
      • VanderMeer T.J.
      • Cheatham M.L.
      • et al.
      Perceptions of graduating general surgery chief residents: are they confident in their training?.
      ,
      • Fonseca A.L.
      • Reddy V.
      • Longo W.E.
      • Gusberg R.J.
      Graduating general surgery resident operative confidence: perspective from a national survey.
      This places scrutiny on the use of operative experience targets as a summative method of certifying trainees and, in particular, raises questions as to how high or low these targets should be set.
      In the United Kingdom and Ireland, experience targets are set for a series of core ‘index procedures’ in surgery training, chosen by the Joint Commission on Surgical Training for their clinical importance.
      • McKee R.F.
      The Intercollegiate Surgical Curriculum Programme (ISCP).
      In general surgery, for example, trainee experience in each procedure is compared to an indicative target number derived from the lowest quartile of operative experience achieved by a graduating cohort of trainees in 2012.
      • Allum W.
      • Hornby S.
      • Khera G.
      • Fitzgerald J.E.
      • Griffiths G.
      General surgery logbook survey.
      Required case minimums can vary substantially across surgical subspecialties.
      • Wood S.
      • James O.P.
      • Hopkins L.
      • et al.
      Variations in competencies needed to complete surgical training.
      Similarly, the Accreditation Council for Graduate Medical Education (ACGME) of the United States has outlined the defined category minimum numbers for surgery residents. General surgery residents, for example, are expected to have logged a total of 850 cases, of which 250 are to be abdominal surgery cases, by the end of training.
      Accreditation Council for Graduate Medical Education
      Defined category minimum numbers for general surgery residents and credit role.
      Ideally, operative experience targets should reference a predefined, robust, and validated performance benchmark. Validated assessment tools have been developed as part of competency-based surgical training curricula internationally. In the United Kingdom and Ireland, workplace-based assessments are regularly conducted using a ‘procedure-based assessment’ (PBA) tool. In the United States, the General Surgery Milestones project has been developed, along with ‘entrustable professional activity’ assessments; workplace-based assessments built around the concept of procedure-specific entrustability.
      • Brasel K.J.
      • Klingensmith M.E.
      • Englander R.
      • et al.
      Entrustable professional activities in general surgery: development and implementation.
      These assessment tools should allow for quantitative analysis of the relationship between operative experience in training and operative competence and further allow for the development of criterion-referenced operative experience targets based on numbers at which trainees are likely to develop proficiency.
      The objective of this systematic review was to identify and critique the available evidence for criterion-referenced operative case minimums in postgraduate surgical training.

      Methods

      This systematic review was reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses 2020 statement.
      • Page M.J.
      • McKenzie J.E.
      • Bossuyt P.M.
      • et al.
      The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
      The primary aim was to explore the relationship between derived or predefined procedural number targets performed in residency or postgraduate surgical training and objectively assessed operative competence (including relevant and similar definitions such as proficiency, mastery, or meaningful autonomy) applied to both procedure-specific competencies and competence across all of the procedures. The review protocol was prospectively registered with PROSPERO - international prospective register of systematic reviews (registration number CRD42021266916).

      Information sources and search strategy

      An initial search of the PubMed (MEDLINE) database was performed using the following search terms: “operative” OR “case” AND “volume” OR “target” OR “threshold” OR “logbook” OR “requirement” and “competence” OR “competency” OR “proficiency” OR “milestone” OR “independent” AND “training” OR “residency” OR “residency education” AND “general surgery.” Further search terms were identified from the titles, abstracts, and keywords of identified relevant articles. A final electronic search (from database inception to the present) of the Cochrane library, PubMed (MEDLINE), Embase, and Web of Science databases was performed in February 2021 using a combination of the following Medical Subject Headings and free-text terms: “general Surgery” OR “specialties, surgical” AND “resident” OR “trainee” OR “training” OR “internship” and “residency” OR “residency education” AND “operative” OR “technical” OR “procedural” OR “surgical” AND “clinical competence” OR “competence” OR “competency” OR “mastery” OR “proficiency” OR “certification” OR “accreditation” OR “credentialing” AND “target” OR “threshold” OR “volume” OR “minimum” OR “learning curve.” The full search strategy is outlined in Appendix 1 (supplemental data).

      Selection process, eligibility criteria, data extraction and data items

      After screening for duplicates, retrieved titles and abstracts were screened for relevance. Full texts were independently reviewed by 2 reviewers (A.O’B., C.T.) according to the inclusion and exclusion criteria outlined in Table I. Articles were not included that documented a correlation between skill or performance with increasing training time or case numbers only, without reference to an objective standard, predefined target, or meaningful definition of competence. Although the relationship between experience and skill is both intuitive and well explored, such studies do not provide meaningful data to support the setting of definitive experience targets in surgical training. Disagreements between reviewers were resolved by consensus; where consensus could not be reached, a third reviewer (D.O.K.) decided on inclusion. Two reviewers (A.O’B., C.T.) further independently extracted the following data points from each included article; study title, year, journal, country, and study design. The method of procedural number data capture, the method of competence assessment, and the relationship between measured or reported competence and procedural numbers were further retrieved (Appendix 2, supplemental data).
      Table IInclusion/exclusion criteria
      Inclusion criteriaExclusion criteria
      All surgical subspecialties (including gynecology and ophthalmic surgery)Nonsurgical specialties
      Relationship between procedural numbers and assessed competence, proficiency, mastery, or autonomy reported, with sufficient information to provide evidence for a criterion referenced experience threshold.Relationship between procedural numbers and skill or performance only, without reference to competence (ie, where skill is not benchmarked against a predefined standard; articles simply demonstrating a correlation between increasing case numbers and outcomes were not included). Similary, articles not reporting the number of procedures required to reach a pre-defined benchmark were excluded.
      Operative proceduresBedside procedures, outpatient procedures, endoscopy procedures.
      Residents/ postgraduate surgical trainees and fellowsMedical students, attending/consultant surgeons, non-surgical trainees.
      No language restrictionsReview articles
      Case reports/technique articles, correspondence/ letters to editors

      Assessment of study quality and educational impact

      The quality of included studies was assessed using the Medical Education Research Study Quality Instrument, a 10-item quality assessment tool scored to a maximum of 18.
      • Reed D.A.
      • Cook D.A.
      • Beckman T.J.
      • Levine R.B.
      • Kern D.E.
      • Wright S.M.
      Association between funding and quality of published medical education research.
      The educational impact of each included study was evaluated using a modified Kirkpatrick model
      • Kirkpatrick D.
      • Kirkpatrick J.
      Evaluating Training Programs: The Four Levels.
      as further developed by Freeth et al
      • Freeth D.
      • Hammick M.
      • Koppel I.
      • Reeves S.
      • Barr H.
      A Critical Review of Evaluations of Interprofessional Education.
      for use in healthcare-related educational studies (Table II). This review was not focused on the perceptions or experiences of trainees or trainers subject to operative experience targets; therefore, it was expected that studies would be identified presenting evidence related to Kirkpatrick levels 2b to 4b (Table II).
      Table IIModified Kirkpatrick Model
      • Kirkpatrick D.
      • Kirkpatrick J.
      Evaluating Training Programs: The Four Levels.
      adapted from Freeth et al 2002
      • Freeth D.
      • Hammick M.
      • Koppel I.
      • Reeves S.
      • Barr H.
      A Critical Review of Evaluations of Interprofessional Education.
      LevelDescription
      1 ReactionTrainees’ views on minimum operative experience targets or their experiences of their use in surgical training.
      2a Modification of attitudes/ perceptionsOutcomes relate to changes in the reciprocal attitudes or perceptions between participant groups toward minimum operative experience targets.
      2b Acquisition of knowledge / skillsAcquisition of knowledge and skills related to or measured by minimum operative experience targets.
      3 Behavioral changeDocuments the transfer of learning to the workplace, or willingness of learners to apply new knowledge and skills because of minimum operative number targets, or a change in behavior as a result of reaching a minimum derived or predefined case minimum.
      4a Change in organizational practiceWider changes in the organizational delivery of care, attributable to reaching minimum operative number targets.
      4b Benefits to patientsAny improvement in the health and well-being of patients because of trainees reaching a minimum derived or predefined operative experience target.

      Results

      Study selection

      A total of 3,672 records were retrieved from database searching, of which 605 were duplicates. On screening of titles and abstracts, 2,810 articles were excluded. A total of 257 full-text articles were independently reviewed, of which 18 could not be retrieved (abstract only). Of the 239 full-text reports assessed for eligibility, 212 were excluded for reasons outlined in Figure 1. On manual searching of the reference lists of the 28 identified studies, 2 further articles were deemed eligible for inclusion. Thirty articles were ultimately included in a narrative synthesis.
      Figure thumbnail gr1
      Figure 1Preferred items for systematic reviews and meta-analyses flowchart. Adapted from Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. https://doi.org/10.1136/bmj.n71.

      Study characteristics and quality assessment

      The study characteristics are outlined in Table III. All included studies were observational: 12 were retrospective studies
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      and 18 were prospective.
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      • Malik M.U.
      • Varela D.A.
      • Park E.
      • et al.
      Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      The mean Medical Education Research Study Quality Instrument score was 14.12 (range 10–16) (Appendix 2, supplementary data).
      Table IIICharacteristics of included studies
      First authorYearCountrySample size, no. of traineesPopulation
      Umali
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      2020Philippines40Ophthalmology third year residents
      Al-jindan
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      2020Saudi Arabia22Ophthalmology residents year 1–4
      Liebman
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      2020USA40Ophthalmology residents
      Masoud
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      2020Canada1Urology trainee
      Merola
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      2020Italy4General surgery trainees
      Shibuya
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      2019Japan6Pediatric surgery trainees
      Hopkins
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      2019UK24Orthopedic HSTs
      Gupta
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      2018India38Ophthalmology residents
      Bracale
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      2018Italy2General surgery trainees
      Abou-foul
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      2018UK8OLHNS trainees
      Van Rijswijk
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      2017the Netherlands9General surgery residents
      Brown
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      2017UK84General surgery HSTs
      Stride
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      2017USA536General surgery residents
      Garg
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      2016India8Urology residents
      Serrano
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      2016USA30Transplant surgery fellows
      Abdelrahman
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      2016UK69General surgery HSTs
      De Siqueira
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      2016UK121General surgery hsts
      Hodgins
      • Hodgins J.L.
      • Veillette C.
      • Biau D.
      • Sonnadara R.
      The knee arthroscopy learning curve: quantitative assessment of surgical skills.
      2014Canada20Orthopedic trainees (PGY 1–5)
      Laeeq
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      2013USA17OLHNS trainees
      Malik
      • Malik M.U.
      • Varela D.A.
      • Park E.
      • et al.
      Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
      2013USA15OLHNS trainees
      Joseph
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      2012USA7Chief general surgery residents
      Taravella
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      2011USA9Ophthalmology residents
      Hiemstra
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      2011The Netherlands9Gynecology residents
      Jelovsek
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      2010USA27Gynecology residents
      Maeda
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      2010Japan1Colorectal fellow
      Francis
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      2010USA9OLHNS trainees
      Papavramidis
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      2010Greece4GS residents
      Zhang
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      2009China5Urology trainees (years 1–5)
      Randleman
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      2007USA15Ophthalmology residents
      Powers
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      2005USA1Senior fellow in gynecology
      GS, general surgery; HST, higher specialist trainee; OLHNS, otolaryngology/head and neck surgery; PGY, postgraduate year.

      Studies demonstrating the relationship between the development of operative competence and operative experience based on observed and measured trainee performance in the operating room (Kirkpatrick level 3)

      The findings of included studies are outlined in Table IV. No included studies presented relevant data pertaining to Kirkpatrick levels 1b to 2b. Fifteen studies used operative time alone as a surrogate marker of operative competence in order to generate trainee learning curves.
      • Hopper A.N.
      • Jamison M.H.
      • Lewis W.G.
      Learning curves in surgical practice.
      ,
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      ,
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      ,
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      ,
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      ,
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      ,
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      ,
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      ,
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      Several studies documented the mean or individual number of cases after which trainee operative time reached a plateau.
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      ,
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      ,
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      Using this technique, Masoud et al tracked the learning curve of a single urology trainee, calculating the learning curve in laparoscopic nephrectomy to be 22 cases.
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      Liebman et al tracked the operative time of 40 ophthalmology trainees performing cataract surgery; operative times decreased until no significant improvement was seen after 150 cases.
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      Garg et al recorded a plateau in the mean operative time of 8 urology residents performing percutaneous nephrolithotomy for simple stones after 30 to 35 cases.
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      Powers et al tracked a gynecology fellow’s time to complete laparoscopic paravaginal repairs until it reached an arbitrarily set cut-off of <155 minutes, recording a learning curve of 25 to 30 cases.
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      Umali et al used the <30-minute cut-off referenced as required for competence performance by the International Council of Ophthalmology-Ophthalmology Surgical Competency Assessment Rubric tool and tracked the learning curve of 40 ophthalmology trainees performing phacoemulsification; competent performance was recorded between 76 to 100 cases.
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      Papavramidis et al compared trainee operative time in minimally invasive video-assisted total thyroidectomy with that of a previously used technique, defining a learning curve of 8 cases.
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      Table IVCase minimums associated with measures of operative competence in included studies
      First authorYearProcedure(s)Method of competence assessmentMethod of determining relationship between competence and experienceOperative experience threshold determinedKirkpatrick level
      Umali
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      2020PhacoemulsificationOperative timeMedian operative time <30 min (ICO-OSCAR standard)76–1003
      Al-jindan
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      2020Phacoemulsification3-point competence rater across 11 procedural steps>90% proficiency in all recorded tasks403
      Liebman
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      2020All cataract surgeriesOperative timeMixed-effect linear regression model relating mean operative time to case numbers (plateau)1503
      Masoud
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      2020Laparoscopic nephrectomyOperative timePlateau in operative time223
      Merola
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      2020Open inguinal hernia repairDuration of surgery, intraoperative and postoperative complications, and 6-mo incidence of chronic pain or numbness (2-y recurrence rates)CUSUM and KPSS (expert benchmarked operative time)37–423
      Shibuya
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      2019LPEC for inguinal herniaOperative timeMean operative time <25 min (expert benchmark)1253
      Hopkins
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      2019A range of procedures in orthopedic surgeryPBA toolMedian number of cases to achieve ‘level 4’ competenceCarpal tunnel decompression: 33; arthroscopy: 101; total knee replacement: 41; compression hip screw: 28; tension band wiring: 9; intramedullary nailing of the femur or tibia: 25; tendon repair: 193
      Bracale
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      2018Laparoscopic inguinal hernia repair (TAPP)Operative time, intraoperative, and postoperative complication ratesCUSUM and KPSS (expert referenced operative time)60–653
      Abou-foul
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      2018TonsillectomyExpert benchmarked operative time and complication rate)CUSUM (expert benchmarked)25–304b
      Van Rijswijk
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      2017Laparoscopic Roux-en-Y gastric bypassThirty-day complication rate, and duration of surgeryLinear regression of resident versus experienced surgeons learning curve (operative time)50–1003
      Brown
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      2017A range of procedures in general surgeryPBA toolMedian number of procedures performed when 3 level 4 PBAs achieved64 inguinal hernia repairs; 83 emergency laparotomies; 87 laparoscopic cholecystectomies; 95 appendicectomies; 45 segmental colectomies; 16 Hartmann procedures3
      Stride
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      2017A range of procedures in general surgerySIMPL toolThe number of procedures at which the majority of the resident population transitions to ‘meaningful autonomy’25 laparoscopic appendicectomies; 52 laparoscopic cholecystectomies; 42 open inguinal hernia repairs; 35 ventral hernia repairs; 60 partial colectomy procedures3
      Garg
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      2016Percutaneous nephrolithotomyOperative time, fluoroscopic time, complication rate using the modified Clavien grading system and success ratePlateau in mean operative time and fluoroscopic time30–35 (simple stones only)3
      Serrano
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      2016Laparoscopic donor nephrectomyIntraoperative time, estimated blood loss, and incidence of intraoperative complications (novice TSF performance compared to senior TSF performance)RACUSUM (expert benchmarked)35–384b
      Abdelrahman
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      2016A range of procedures in general surgeryPBA toolMedian number of procedures performed when 3 level 4 PBAs achieved76 emergency laparotomies; 17 Hartmann procedures; 107 appendicectomies; 64 inguinal hernia repairs; 52 segmental colectomies; 72 laparoscopic cholecystectomies3
      De Siqueira
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      2016A range of procedures in general surgeryPBA toolMedian number of procedures performed when 3 level 4 PBAs achieved90 cholecystectomy procedures; 55 segmental colectomies; 15 Hartman procedures3
      Hodgins
      • Hodgins J.L.
      • Veillette C.
      • Biau D.
      • Sonnadara R.
      The knee arthroscopy learning curve: quantitative assessment of surgical skills.
      2014Knee arthroscopyBAKSSLC-CUSUM13–243
      Laeeq
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      2013Endoscopic sinus surgeryPredefined procedure specific checklistScore ≥3 on each particular task of a procedure-specific checklist42–553
      Malik
      • Malik K.I.
      • Siau K.
      • Dunckley P.
      • Ward S.T.
      Colorectal trainees in the UK struggle to meet JAG certification standards in colonoscopy by the end of their training.
      2013MastoidectomyProcedure-specific checklists; LOWESS regression analysesPlateau in mean scores achieved on checklist10–153
      Joseph
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      2012Single incision Laparoscopic cholecystectomyOperative time, blood loss, Intraoperative complicationsExpert-benchmarked operative time53
      Taravella
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      2011Phacoemulsification3-point proficiency rater across 5 procedural stepsExpert-benchmarked operative time and proficiency scores with increasing case numbers1003
      Hiemstra
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      2011Pooled results across a range of gynecology proceduresOSATSOSATS >2463
      Jelovsek
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      2010Vaginal hysterectomyVaginal Surgical Skills Index and a Global Rating ScaleNumber of procedures required to reach ‘cut-off’ scores determined using a modified Angoff method21–273
      Maeda
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      2010Laparoscopic colon and rectosigmoid resectionsOperative time, and ‘failure’ defined as either: (1) perioperative major morbidity and mortality, (2) intraoperative blood loss >1,000 mL, or (3) long operative time >2 SDs above the department average (>240 min for open surgery, >270 min for laparoscopic resections)Operative time - moving average until plateau reached. Operative ‘failure’ - CUSUMOperative failure: open surgery = 11; laparoscopic = 7. Operative time: open surgery = 22; laparoscopic = 25.4b
      Francis
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      2010MastoidectomyPredefined procedure specific checklistScore ≥3 on a particular task, and maintained in all subsequent evaluations133
      Papavramidis
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      2010Minimally invasive video-assisted total thyroidectomyOperative time, postoperative patient outcomesOperative duration equivalent to previously performed ‘classic operation’ reached83
      Zhang
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      2009Laparoscopic adrenalectomyOperative timeExpert benchmarked operative time (plateau)25–303
      Randleman
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      2007PhacoemulsificationOperative time, intraoperative complication rateMean intraoperative complication rate and operative time (expert benchmarked)1604b
      Powers
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      2005Laparoscopic paravaginal repairOperative timeOperative time <155 min25–30
      Gupta
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      2018Manual small-incision cataract surgeryIntraoperative complications, postoperative complications, and reoperationsMultivariate logistic regression, relationship between case numbers and postoperative complication rate of <2%3004b
      BAKSS, basic arthroscopic knee skill scoring system; CUSUM, cumulative sum analysis; ICO-OSCAR, International Council of Ophthalmology-Ophthalmology Surgical Competency Assessment Rubric; KPSS, Kwiatkowski–Phillips–Schmidt–Shin test; LC-CUSUM, learning curve cumulative sum analysis; LOWESS, locally weighted scatterplot smoothing; LPEC, laparoscopic percutaneous extraperitoneal closure; OSATS, Objective Structured Assessment of Technical Skill; PBA, procedure-based assessment; RACUSUM, risk-adjusted cumulative sum analysis; SIMPL, Society for Improving Medical Professional Learning; TAPP, transabdominal preperitoneal; TSF, transplant surgery fellow.
      A number of further studies have referenced trainee operative time to expert benchmarks in order to generate learning curves.
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      ,
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      ,
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      ,
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      ,
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      ,
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      ,
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      ,
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      Merola et al used cumulative sum (CUSUM) analyses to determine a learning curve of 37 to 42 open inguinal hernia repairs for 4 general surgery trainees.
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      The CUSUM analysis was also used by Bracale et al to define a learning curve of 60 to 65 cases for laparoscopic inguinal hernia repairs.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      Van Rijswijk et al used regression analyses to conclude that trainees reach an operative time similar to expert surgeons in laparoscopic Roux-en-Y gastric bypass procedures after between 50 and 100 cases.
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      Taravella et al found that ophthalmology trainees reach expert-level operative times after performing 76 to 100 cases.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      Similar methods were used by Shibuya et al to record a learning curve of 125 cases for laparoscopic percutaneous extraperitoneal closure hernia repair,
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      whereas the operative times for trainees learning single-incision laparoscopic cholecystectomy in Joseph et al’s 2012 study reached expert levels after 5 procedures.
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      In Zhang et al’s 2009 study of 9 urology trainees, expert-level operative times were observed after 25 to 30 cases.
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      Thirteen further studies used intraoperative assessment tools to determine a relationship between the case volume and development of competence.
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      ,
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      ,
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      ,
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      • Malik M.U.
      • Varela D.A.
      • Park E.
      • et al.
      Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      ,
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      ,
      • Hodgins J.L.
      • Veillette C.
      • Biau D.
      • Sonnadara R.
      The knee arthroscopy learning curve: quantitative assessment of surgical skills.
      Four studies in general and orthopedic surgery from the United Kingdom tracked the number of cases required before trainees are awarded ‘level 4’ scores (corresponding to a competent performance) using the validated PBA tool.
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      The findings from these studies, across a broad range of index procedures, are outlined in Table IV. A newly developed (previously unvalidated) task-specific checklist was used to track competence in phacoemulsification by Al-Jindan et al
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      ; 40 cases were required before >90% proficiency was recorded across all of the procedural steps. Laeeq et al similarly used a (previously validated) procedure-specific checklist to assess performance by calculating a learning curve for endoscopic sinus surgery of between 43 and 55 cases.
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      Malik et al used a validated procedure-specific checklist to assess performance in mastoidectomy, where the learning curve was defined as the number of cases at which a plateau in scores was reached.
      • Brasel K.J.
      • Klingensmith M.E.
      • Englander R.
      • et al.
      Entrustable professional activities in general surgery: development and implementation.
      • Page M.J.
      • McKenzie J.E.
      • Bossuyt P.M.
      • et al.
      The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
      • Reed D.A.
      • Cook D.A.
      • Beckman T.J.
      • Levine R.B.
      • Kern D.E.
      • Wright S.M.
      Association between funding and quality of published medical education research.
      • Kirkpatrick D.
      • Kirkpatrick J.
      Evaluating Training Programs: The Four Levels.
      • Freeth D.
      • Hammick M.
      • Koppel I.
      • Reeves S.
      • Barr H.
      A Critical Review of Evaluations of Interprofessional Education.
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      ,
      • Malik M.U.
      • Varela D.A.
      • Park E.
      • et al.
      Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
      A similar learning curve for this procedure was identified by Francis et al (13 cases), using predefined proficiency benchmarks on a procedure-specific checklist.
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      Taravella et al used a combined metric of operative time within 1 SD of expert performance and proficiency as recorded using a procedure-specific checklist to conclude that a majority (58%) of trainees achieved competent performance after performing 100 cases.
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      Expert benchmarking was used by Hodgins et al to record competent performance in arthroscopy using the Basic Arthroscopic Knee Skills Scoring System tool after between 13 and 24 cases were performed. Hiemstra et al defined an Objective Structured Assessment of Technical Skill score of ≥24 as acceptable performance that was reached by 9 gynecology trainees performing a mix of procedures after 6 cases.
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      Jelovesk used a modified Angoff approach to develop a competent standard using the Vaginal Surgical Skills Index and a global rating scale, noting that trainees achieved this threshold after performing between 21 and 27 vaginal hysterectomies.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      ,
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      In 2017, Stride et al’s study used ‘meaningful autonomy’ (corresponding to a Zwisch scale score of ‘3–passive help’ or greater) as a proxy measure for competence.
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      The number of procedures at which the majority of residents transitioned to meaningful procedural autonomy for key index procedures is outlined in Table IV.
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.

      Studies demonstrating the relationship between the development of operative competence and operative experience based on patient outcomes (Kirkpatrick level 4b)

      No studies explored the relationship between operative experience in training and changes in organizational practice (Kirkpatrick level 4a). Five studies were able to define the learning curves in training with respect to patient outcome measures (Kirkpatrick level 4b).
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      ,
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      ,
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      ,
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      Gupta et al reported that it took 300 cases of manual small-incision cataract surgery before trainees achieved an acceptable postoperative complication rate of <2%.
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      Similarly, Randleman et al used a complication rate of <2% to define a learning curve of 160 phacoemulsification cases; a similar threshold was determined using expert-referenced operative time. Abou-Foul et al used both operative time and postoperative complication rates to define an expert-benchmarked learning curve (using CUSUM analysis) of between 25 and 30 tonsillectomy cases.
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      A similarly combined outcome metric (intraoperative time, estimated blood loss, and incidence of intraoperative complications) was used by Serrano et al to explore the relationship between experience and competence in fellows performing laparoscopic donor nephrectomy (at 35–38 cases), though performance of junior fellows was benchmarked against that of senior fellows and not post-fellowship surgeons.
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      Maeda et al, again, used CUSUM analysis to monitor the learning curve of a single colorectal fellow, using a composite outcome measure of ‘operative failure’ (defined as either of the following: (1) perioperative major morbidity and mortality, (2) intraoperative blood loss >1,000 mL, or (3) long operative time >2 SDs above the department average (>240 min for open surgery, >270 min for laparoscopic resections).
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      A learning curve of 11 open colonic or rectosigmoid resections and 7 laparoscopic resections was generated compared to 22 and 25 cases, respectively, if operative time was used alone as an outcome measure.
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.

      Discussion

      This study identified and critiqued the evidence regarding the use of operative experience thresholds to determine trainee operative competence. Competence is determined by a number of outcome measures—faculty or trainer assessment,
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      ,
      • Al-Jindan M.
      • Almarshood A.
      • Yassin S.A.
      • Alarfaj K.
      • Al Mahmood A.
      • Sulaimani N.M.
      Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
      ,
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      ,
      • Laeeq K.
      • Lin S.Y.
      • Varela D.A.
      • Lane A.P.
      • Reh D.
      • Bhatti N.I.
      Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
      • Malik M.U.
      • Varela D.A.
      • Park E.
      • et al.
      Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      • Hiemstra E.
      • Kolkman W.
      • Wolterbeek R.
      • Trimbos B.
      • Jansen F.W.
      Value of an objective assessment tool in the operating room.
      • Jelovsek J.E.
      • Walters M.D.
      • Korn A.
      • et al.
      Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
      ,
      • Francis H.W.
      • Masood H.
      • Laeeq K.
      • Bhatti N.I.
      Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
      ,
      • Hodgins J.L.
      • Veillette C.
      • Biau D.
      • Sonnadara R.
      The knee arthroscopy learning curve: quantitative assessment of surgical skills.
      patient outcomes,
      • Gupta S.
      • Haripriya A.
      • Vardhan S.A.
      • Ravilla T.
      • Ravindran R.D.
      Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
      ,
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      ,
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      ,
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      or surrogate measures of competence such as operative time.
      • Hopper A.N.
      • Jamison M.H.
      • Lewis W.G.
      Learning curves in surgical practice.
      ,
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      ,
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      ,
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      ,
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      ,
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      ,
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      ,
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      ,
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      Ideally, learning curves should be generated based on objective, relevant, and measurable in-theater or postoperative patient outcome measures. Well-conducted studies in ophthalmology and otolaryngology/head and neck surgery by Randleman et al, Abou-Foul et al, and Gupta et al demonstrate how this data can be used to generate evidence for criterion-referenced operative targets, though similar studies across other surgical specialties are lacking. Lessons can be learned from studies of trainee competence in endoscopy, where well-defined procedural metrics can be used to benchmark performance.
      • Malik K.I.
      • Siau K.
      • Dunckley P.
      • Ward S.T.
      Colorectal trainees in the UK struggle to meet JAG certification standards in colonoscopy by the end of their training.
      ,
      • Wexner S.D.
      • Litwin D.
      • Cohen J.
      • et al.
      Principles of privileging and credentialing for endoscopy and colonoscopy.
      In colonoscopy, for example, learning curves can be generated using the independent caecal intubation rate as a measure of competence.
      • Shahidi N.
      • Ou G.
      • Telford J.
      • Enns R.
      Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review.
      ,
      • Sedlack R.E.
      Training to competency in colonoscopy: assessing and defining competency standards.
      Using such procedural metrics for competence assessments in surgery requires definition of relevant metrics and pass/fail criteria, as well as a method of objectively and reliably assessing against these metrics. Measures of intraoperative error in laparoscopic surgery have been developed and have accrued substantial validity evidence.
      • Seymour N.E.
      • Gallagher A.G.
      • Roman S.A.
      • O’Brien M.K.
      • Andersen D.K.
      • Satava R.M.
      Analysis of errors in laparoscopic surgical procedures.
      • Ahlberg G.
      • Enochsson L.
      • Gallagher A.G.
      • et al.
      Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies.
      • Seymour N.E.
      • Gallagher A.G.
      • Roman S.A.
      • et al.
      Virtual reality training improves operating room performance: results of a randomized, double-blinded study.
      In the future, video-assessment tools and artificial intelligence technology may be useful in prospectively monitoring trainee performance.
      • McQueen S.
      • McKinnon V.
      • VanderBeek L.
      • McCarthy C.
      • Sonnadara R.
      Video-based assessment in surgical education: a scoping review.
      ,
      • Kirubarajan A.
      • Young D.
      • Khan S.
      • Crasto N.
      • Sobel M.
      • Sussman D.
      Artificial intelligence and surgical education: a systematic scoping review of interventions.
      Operative time is used by several included studies as a surrogate marker of operative competence.
      • Hopper A.N.
      • Jamison M.H.
      • Lewis W.G.
      Learning curves in surgical practice.
      ,
      • Liebman D.L.
      • McKay K.M.
      • Haviland M.J.
      • Moustafa G.A.
      • Borkar D.S.
      • Kloek C.E.
      Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
      ,
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      • Powers K.
      • Lazarou G.
      • Greston W.
      • Mikhail M.
      The learning curve in laparoscopic reconstructive surgery: a pilot study.
      • Umali M.I.N.
      • Castillo T.R.
      Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
      ,
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      ,
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      ,
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      ,
      • Papavramidis T.S.
      • Michalopoulos N.
      • Pliakos J.
      • et al.
      Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
      ,
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      However, the relationship between these two variables is not certain. Although some studies have identified operative time as an independent predictor of complication rates
      • Reames B.N.
      • Bacal D.
      • Krell R.W.
      • Birkmeyer J.D.
      • Birkmeyer N.J.
      • Finks J.F.
      Influence of median surgeon operative duration on adverse outcomes in bariatric surgery.
      and technical skill,
      • Julià D.
      • Gómez N.
      • Codina-Cazador A.
      Surgical skill and complication rates after bariatric surgery.
      other studies have failed to identify positive associations
      • Darzi A.
      • Smith S.
      • Taffinder N.
      Assessing operative skill. Needs to become more objective.
      ,
      • Addison P.
      • Yoo A.
      • Duarte-Ramos J.
      • et al.
      Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery.
      ; there may be many trainees who operate well and slowly and others who operate poorly at speed. Tracking operative times until a plateau is reached
      • Masoud M.
      • Ibrahim A.
      • Elemam A.
      • et al.
      Learning curve of laparoscopic nephrectomy: a prospective pilot study.
      ,
      • Garg A.
      • Yadav S.S.
      • Tomar V.
      • et al.
      Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
      ,
      • Maeda T.
      • Tan K.Y.
      • Konishi F.
      • et al.
      Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
      relies on the assumption that the learning curve is ‘L-shaped’; this may not always hold true.
      • Nursal T.Z.
      • Ezer A.
      • Belli S.
      • Parlakgumus A.
      • Caliskan K.
      • Noyan T.
      Reaching proficiency in laparoscopic splenectomy.
      If operative time is to be used as a proxy measure of competence, it should be benchmarked to a validated predefined threshold or expert standard.
      • Merola G.
      • Cavallaro G.
      • Iorio O.
      • et al.
      Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
      ,
      • van Rijswijk A.S.
      • Moes D.E.
      • Geubbels N.
      • et al.
      Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
      ,
      • Randleman J.B.
      • Wolfe J.D.
      • Woodward M.
      • Lynn M.J.
      • Cherwek D.H.
      • Srivastava S.K.
      The resident surgeon phacoemulsification learning curve.
      ,
      • Shibuya S.
      • Fujiwara N.
      • Ochi T.
      • et al.
      The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
      • Bracale U.
      • Merola G.
      • Sciuto A.
      • Cavallaro G.
      • Andreuccetti J.
      • Pignata G.
      Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
      • Abou-Foul A.K.
      • Taghi A.
      • Tolley N.S.
      • Awad Z.
      Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
      ,
      • Serrano O.K.
      • Bangdiwala A.S.
      • Vock D.M.
      • et al.
      Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
      ,
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      ,
      • Taravella M.J.
      • Davidson R.
      • Erlanger M.
      • Guiton G.
      • Gregory D.
      Characterizing the learning curve in phacoemulsification.
      ,
      • Zhang X.
      • Wang B.
      • Ma X.
      • et al.
      Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
      Clear learning curves can be generated from data from sequential workplace-based assessments.
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      This approach is not without its own challenges. It requires clear, objective benchmarks of competence to be set. It also requires multiple assessments to be undertaken on multiple occasions. For example, the relationship between the time to achievement of a trainee’s first ‘competent’ assessment score may not relate to the achievement of subsequent similar scores.
      • Hopkins L.
      • Robinson D.B.T.
      • Brown C.
      • et al.
      Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      Competence assessed at a single timepoint, or even across multiple timepoints, may not guarantee future competence, and the reliability of an assessment tool is important in determining how many procedural assessments at a given level of operative competence will be required to make a summary judgment of a trainee’s ability.
      • Abbott K.L.
      • Chen X.
      • Clark M.
      • Bibler Zaidi N.L.
      • Swanson D.B.
      • George B.C.
      Number of operative performance ratings needed to reliably assess the difficulty of surgical procedures.
      This reliability, in turn, depends on adequate assessor training for consistency.
      • Woehr D.J.
      • Huffcutt A.I.
      Rater training for performance appraisal: a quantitative review.
      It is also important to determine where exactly operative competency thresholds should be set. Experience targets required for a majority (>50%) of assessed trainees to reach a certain level will be different from targets required for 95% of trainees, for example, to reliably reach this level.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      Defining targets at which trainees are expected to achieve competence means accepting that even after reaching these experience targets, many trainees will not be competent. The reverse may also be true; many trainees will be competent without having achieved the necessary experience targets.
      Learning curves also differ for trainees at different stages of training. Joseph et al reported a short learning curve of 5 cases for single-incision laparoscopic cholecystectomy procedures.
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      It is important to note that the residents recruited to this study had already met and exceeded ACGME-defined case numbers, with a mean of 81 complex laparoscopic procedures logged.
      • Joseph M.
      • Phillips M.
      • Farrell T.M.
      • Rupp C.C.
      Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
      This may also explain the phenomenon observed by Abdelrahman et al, who documented a learning curve of 107 cases for laparoscopic appendicectomy and only 15 cases for emergency Hartmann procedure; trainees are likely undertaking more complex procedures later in training by which stage they will have developed substantial and transferrable surgical skills.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      Evidence to date has suggested that targets in the United Kingdom and the United States may be set too low to ensure competency in most graduating trainees. For example, findings from large studies by De Siqueira, Abdelrahman, and Browne demonstrated that the mean number of procedures required for trainees to meet predefined levels of operative competence is greatly in excess of numbers mandated by the United Kingdom training curriculum for cholecystectomy, segmental colectomy, inguinal hernia repair, and appendicectomy procedures.
      • McKee R.F.
      The Intercollegiate Surgical Curriculum Programme (ISCP).
      ,
      • Brown C.
      • Abdelrahman T.
      • Patel N.
      • Thomas C.
      • Pollitt M.J.
      • Lewis W.G.
      Operative learning curve trajectory in a cohort of surgical trainees.
      • Abdelrahman T.
      • Long J.
      • Egan R.
      • Lewis W.G.
      Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
      • De Siqueira J.R.
      • Gough M.J.
      Correlation between experience targets and competence for general surgery certification.
      Similarly, in 2017, Stride et al’s study identified the transition to meaningful autonomy (defined as requiring ‘passive help’ or ‘supervision only’) for the majority of residents in partial colectomy occurring after 60 cases, in excess of the 40 ‘large intestine’ procedures expected by the ACGME.
      Accreditation Council for Graduate Medical Education
      Defined category minimum numbers for general surgery residents and credit role.
      ,
      • Stride H.P.
      • George B.C.
      • Williams R.G.
      • et al.
      Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
      Simply increasing operative experience thresholds, however, may not be feasible in practice; evidence suggests that some training programs in the United Kingdom struggle to meet the current more modest targets that are in place.
      • Thomas C.
      • Elsey E.
      • Boyce T.
      • Catton J.
      • Lewis W.
      Data from two deaneries regarding the ability of higher surgical training programmes to deliver curriculum targets.
      Setting minimum operative number targets for surgical trainees can also have unintended consequences. In a study by Beamish et al, 33.2% (255/768) of surveyed surgical trainees reported feeling pressure to overstate the number of cases they recorded in their surgical experience logbook.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Use of the eLogbook in surgical training in the United Kingdom: a nationwide survey and consensus recommendations from the Association of Surgeons in Training.
      A further 29.3% of respondents reported that the mandated operative experience targets were unachievable.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Use of the eLogbook in surgical training in the United Kingdom: a nationwide survey and consensus recommendations from the Association of Surgeons in Training.
      More worrying is the finding that 76.5% of respondents believed that the Annual Review of Competence Progression panel, which makes annual summative decisions regarding trainee progression, does not regard competence as more important than numbers achieved.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Use of the eLogbook in surgical training in the United Kingdom: a nationwide survey and consensus recommendations from the Association of Surgeons in Training.
      Each operative case logged documents an opportunity for trainees to develop skills; without corresponding data regarding the observed performance and autonomy of the trainee in each case, these numbers alone are a crude proxy measure of competence at best and meaningless at worst. An undue focus on meeting minimum targets that have not been criterion-referenced may inflict undue pressure on trainees without ensuring subsequent competence. Furthermore, the secondary effect that these pressures place on training programs, program directors, and individual surgical trainers has not been well explored. The newly updated Intercollegiate Surgical Curriculum Programme for the United Kingdom and Ireland (2021) has de-emphasized the importance of indicative number targets, stating that other measures of performance (such as workplace-based assessments) can be used as evidence for trainee competence. These alternative measures will become even more important as surgical training recovers from the profound and ongoing impact of the COVID-19 pandemic on in-theater training opportunities.
      • Hope C.
      • Reilly J.J.
      • Griffiths G.
      • Lund J.
      • Humes D.
      The impact of COVID-19 on surgical training: a systematic review.
      ,
      • Joyce D.P.
      • Ryan D.
      • Kavanagh D.O.
      • Traynor O.
      • Tierney S.
      Impact of COVID-19 on operative experience of junior surgical trainees.

      Limitations

      The study populations, methodologies, procedures, and outcome measures of included studies are heterogenous, precluding meta-analysis. The findings of several studies included in this review relied on the accuracy of submitted data, much of which is self-reported by surgical trainees.
      In conclusion, this study appraised the evidence for operative experience targets in surgical training. Methods of competency assessment differ widely across studies. The evidence comparing the clinical outcomes of trainees who have reached predefined experience targets compared with those who have not is lacking. Workplace-based assessment tools such as PBA and entrustable professional activity (EPA) assessments should allow for future studies to characterize the trainee learning curve across commonly performed procedures in surgical training. This data can then be used to generate criterion-referenced operative experience targets in the absence of definitive patient outcomes data. The findings of this review highlighted the fundamental challenges inherent in using operative experience targets as a proxy measure of surgical competence. The real value of logging procedural numbers by trainees may lie in its ability to highlight training posts providing insufficient training opportunities and as a method of determining individual learning curves, which can then be used to target procedure-specific areas for remediation.

      Funding/Support

      Financial support has been provided by the Royal College of Surgeons/Hermitage Medical Clinic Strategic Academic Recruitment (StAR MD) program.

      Conflict of interest/Disclosure

      The authors have no conflicts of interests or disclosures to report.

      References

        • Gervais J.
        The operational definition of competency-based education.
        JCBE. 2016; 1: 98-106
        • Hopper A.N.
        • Jamison M.H.
        • Lewis W.G.
        Learning curves in surgical practice.
        Postgrad Med J. 2007; 83: 777-779
        • Morche J.
        • Mathes T.
        • Pieper D.
        Relationship between surgeon volume and outcomes: a systematic review of systematic reviews.
        Syst Rev. 2016; 5: 204
        • Friedell M.L.
        • VanderMeer T.J.
        • Cheatham M.L.
        • et al.
        Perceptions of graduating general surgery chief residents: are they confident in their training?.
        J Am Coll Surg. 2014; 218: 695-703
        • Fonseca A.L.
        • Reddy V.
        • Longo W.E.
        • Gusberg R.J.
        Graduating general surgery resident operative confidence: perspective from a national survey.
        J Surg Res. 2014; 190: 419-428
        • McKee R.F.
        The Intercollegiate Surgical Curriculum Programme (ISCP).
        Surgery. 2008; 26: 411-416
        • Allum W.
        • Hornby S.
        • Khera G.
        • Fitzgerald J.E.
        • Griffiths G.
        General surgery logbook survey.
        Bull R Coll Surg Engl. 2013; 95: 1-6
        • Wood S.
        • James O.P.
        • Hopkins L.
        • et al.
        Variations in competencies needed to complete surgical training.
        BJS Open. 2019; 3: 852-856
        • Accreditation Council for Graduate Medical Education
        Defined category minimum numbers for general surgery residents and credit role.
        • Brasel K.J.
        • Klingensmith M.E.
        • Englander R.
        • et al.
        Entrustable professional activities in general surgery: development and implementation.
        J Surg Educ. 2019; 76: 1174-1186
        • Page M.J.
        • McKenzie J.E.
        • Bossuyt P.M.
        • et al.
        The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
        Int J Surg. 2021; 88105906
        • Reed D.A.
        • Cook D.A.
        • Beckman T.J.
        • Levine R.B.
        • Kern D.E.
        • Wright S.M.
        Association between funding and quality of published medical education research.
        JAMA. 2007; 298: 1002-1009
        • Kirkpatrick D.
        • Kirkpatrick J.
        Evaluating Training Programs: The Four Levels.
        Berrett-Koehler Publishers, San Francisco (CA)2006
        • Freeth D.
        • Hammick M.
        • Koppel I.
        • Reeves S.
        • Barr H.
        A Critical Review of Evaluations of Interprofessional Education.
        The Higher Education Academy Health Sciences and Practice Network, London (UK)2002
        • Liebman D.L.
        • McKay K.M.
        • Haviland M.J.
        • Moustafa G.A.
        • Borkar D.S.
        • Kloek C.E.
        Quantifying the educational benefit of additional cataract surgery cases in ophthalmology residency.
        J Cataract Refract Surg. 2020; 46: 1495-1500
        • Merola G.
        • Cavallaro G.
        • Iorio O.
        • et al.
        Learning curve in open inguinal hernia repair: a quality improvement multicentre study about Lichtenstein technique.
        Hernia. 2020; 24: 651-659
        • Hopkins L.
        • Robinson D.B.T.
        • Brown C.
        • et al.
        Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective.
        J Surg Educ. 2019; 76: 1569-1578
        • Gupta S.
        • Haripriya A.
        • Vardhan S.A.
        • Ravilla T.
        • Ravindran R.D.
        Residents’ learning curve for manual small-incision cataract surgery at Aravind Eye Hospital, India.
        Ophthalmology. 2018; 125: 1692-1699
        • van Rijswijk A.S.
        • Moes D.E.
        • Geubbels N.
        • et al.
        Can a laparoscopic Roux-en-Y gastric bypass be safely performed by surgical residents in a bariatric center-of-excellence? The learning curve of surgical residents in bariatric surgery.
        Surg Endosc. 2018; 32: 1012-1020
        • Brown C.
        • Abdelrahman T.
        • Patel N.
        • Thomas C.
        • Pollitt M.J.
        • Lewis W.G.
        Operative learning curve trajectory in a cohort of surgical trainees.
        Br J Surg. 2017; 104: 1405-1411
        • Abdelrahman T.
        • Long J.
        • Egan R.
        • Lewis W.G.
        Operative Experience vs. competence: a curriculum concordance and learning curve analysis.
        J Surg Educ. 2016; 73: 694-698
        • De Siqueira J.R.
        • Gough M.J.
        Correlation between experience targets and competence for general surgery certification.
        Br J Surg. 2016; 103: 921-927
        • Randleman J.B.
        • Wolfe J.D.
        • Woodward M.
        • Lynn M.J.
        • Cherwek D.H.
        • Srivastava S.K.
        The resident surgeon phacoemulsification learning curve.
        Arch Ophthalmol. 2007; 125: 1215-1219
        • Powers K.
        • Lazarou G.
        • Greston W.
        • Mikhail M.
        The learning curve in laparoscopic reconstructive surgery: a pilot study.
        Femal Pelvic Med Reconstr Surg. 2005; 11: 329-332
        • Umali M.I.N.
        • Castillo T.R.
        Operative time and complication rates of resident phacoemulsification surgeries in a national university hospital: a five-year review.
        Clin Ophthalmol. 2020; 14: 4065-4072
        • Al-Jindan M.
        • Almarshood A.
        • Yassin S.A.
        • Alarfaj K.
        • Al Mahmood A.
        • Sulaimani N.M.
        Assessment of learning curve in phacoemulsification surgery among the Eastern province ophthalmology program residents.
        Clin Ophthalmol. 2020; 14: 113-118
        • Masoud M.
        • Ibrahim A.
        • Elemam A.
        • et al.
        Learning curve of laparoscopic nephrectomy: a prospective pilot study.
        Afr J Urol. 2020; 26: 24
        • Shibuya S.
        • Fujiwara N.
        • Ochi T.
        • et al.
        The learning curve of laparoscopic percutaneous extraperitoneal closure (LPEC) for inguinal hernia: protocoled training in a single center for six pediatric surgical trainees.
        BMC Surg. 2019; 19: 6
        • Bracale U.
        • Merola G.
        • Sciuto A.
        • Cavallaro G.
        • Andreuccetti J.
        • Pignata G.
        Achieving the learning curve in laparoscopic inguinal hernia repair by Tapp: a quality improvement study.
        J Invest Surg. 2019; 32: 738-745
        • Abou-Foul A.K.
        • Taghi A.
        • Tolley N.S.
        • Awad Z.
        Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis.
        J Laryngol Otol. 2018; 132: 734-738
        • Stride H.P.
        • George B.C.
        • Williams R.G.
        • et al.
        Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.
        Surgery. 2018; 163: 488-494
        • Garg A.
        • Yadav S.S.
        • Tomar V.
        • et al.
        Prospective evaluation of learning curve of urology residents for percutaneous nephrolithotomy.
        Urol Pract. 2016; 3: 230-235
        • Serrano O.K.
        • Bangdiwala A.S.
        • Vock D.M.
        • et al.
        Defining the tipping point in surgical performance for laparoscopic donor nephrectomy among transplant surgery fellows: a risk-adjusted cumulative summation learning curve analysis.
        Am J Transplant. 2017; 17: 1868-1878
        • Joseph M.
        • Phillips M.
        • Farrell T.M.
        • Rupp C.C.
        Can residents safely and efficiently be taught single incision laparoscopic cholecystectomy?.
        J Surg Educ. 2012; 69: 468-472
        • Laeeq K.
        • Lin S.Y.
        • Varela D.A.
        • Lane A.P.
        • Reh D.
        • Bhatti N.I.
        Achievement of competency in endoscopic sinus surgery of otolaryngology residents.
        Laryngoscope. 2013; 123 (2932–2394)
        • Malik M.U.
        • Varela D.A.
        • Park E.
        • et al.
        Determinants of resident competence in mastoidectomy: role of interest and deliberate practice.
        Laryngoscope. 2013; 123: 3162-3167
        • Taravella M.J.
        • Davidson R.
        • Erlanger M.
        • Guiton G.
        • Gregory D.
        Characterizing the learning curve in phacoemulsification.
        J Cataract Refract Surg. 2011; 37: 1069-1075
        • Hiemstra E.
        • Kolkman W.
        • Wolterbeek R.
        • Trimbos B.
        • Jansen F.W.
        Value of an objective assessment tool in the operating room.
        Can J Surg. 2011; 54: 116-122
        • Jelovsek J.E.
        • Walters M.D.
        • Korn A.
        • et al.
        Establishing cutoff scores on assessments of surgical skills to determine surgical competence.
        Am J Obstet Gynecol. 2010; 203 (81.e1–86.e6)
        • Maeda T.
        • Tan K.Y.
        • Konishi F.
        • et al.
        Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision.
        Surg Endosc. 2010; 24: 2850-2854
        • Francis H.W.
        • Masood H.
        • Laeeq K.
        • Bhatti N.I.
        Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
        Laryngoscope. 2010; 120: 1417-1421
        • Papavramidis T.S.
        • Michalopoulos N.
        • Pliakos J.
        • et al.
        Minimally invasive video-assisted total thyroidectomy: an easy to learn technique for skillful surgeons.
        Head Neck. 2010; 32: 1370-1376
        • Zhang X.
        • Wang B.
        • Ma X.
        • et al.
        Laparoscopic adrenalectomy for beginners without open counterpart experience: initial results under staged training.
        Urology. 2009; 73: 1061-1065
        • Hodgins J.L.
        • Veillette C.
        • Biau D.
        • Sonnadara R.
        The knee arthroscopy learning curve: quantitative assessment of surgical skills.
        Arthroscopy. 2014; 30: 613-621
        • Malik K.I.
        • Siau K.
        • Dunckley P.
        • Ward S.T.
        Colorectal trainees in the UK struggle to meet JAG certification standards in colonoscopy by the end of their training.
        Colorectal Dis. 2019; 21: 715-722
        • Wexner S.D.
        • Litwin D.
        • Cohen J.
        • et al.
        Principles of privileging and credentialing for endoscopy and colonoscopy.
        Gastrointest Endosc. 2002; 55: 145-148
        • Shahidi N.
        • Ou G.
        • Telford J.
        • Enns R.
        Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review.
        Gastrointest Endosc. 2014; 80: 410-416
        • Sedlack R.E.
        Training to competency in colonoscopy: assessing and defining competency standards.
        Gastrointest Endosc. 2011; 74: 355-366.e2
        • Seymour N.E.
        • Gallagher A.G.
        • Roman S.A.
        • O’Brien M.K.
        • Andersen D.K.
        • Satava R.M.
        Analysis of errors in laparoscopic surgical procedures.
        Surg Endosc. 2004; 18: 592-595
        • Ahlberg G.
        • Enochsson L.
        • Gallagher A.G.
        • et al.
        Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies.
        Am J Surg. 2007; 193: 797-804
        • Seymour N.E.
        • Gallagher A.G.
        • Roman S.A.
        • et al.
        Virtual reality training improves operating room performance: results of a randomized, double-blinded study.
        Ann Surg. 2002; 236 (discussion 463–4): 458-463
        • McQueen S.
        • McKinnon V.
        • VanderBeek L.
        • McCarthy C.
        • Sonnadara R.
        Video-based assessment in surgical education: a scoping review.
        J Surg Educ. 2019; 76: 1645-1654
        • Kirubarajan A.
        • Young D.
        • Khan S.
        • Crasto N.
        • Sobel M.
        • Sussman D.
        Artificial intelligence and surgical education: a systematic scoping review of interventions.
        J Surg Educ. 2022; 79: 500-515
        • Reames B.N.
        • Bacal D.
        • Krell R.W.
        • Birkmeyer J.D.
        • Birkmeyer N.J.
        • Finks J.F.
        Influence of median surgeon operative duration on adverse outcomes in bariatric surgery.
        Surg Obes Relat Dis. 2015; 11: 207-213
        • Julià D.
        • Gómez N.
        • Codina-Cazador A.
        Surgical skill and complication rates after bariatric surgery.
        N Engl J Med. 2014; 370: 285
        • Darzi A.
        • Smith S.
        • Taffinder N.
        Assessing operative skill. Needs to become more objective.
        BMJ. 1999; 318: 887-888
        • Addison P.
        • Yoo A.
        • Duarte-Ramos J.
        • et al.
        Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery.
        Surg Endosc. 2021; 35: 5303-5309
        • Nursal T.Z.
        • Ezer A.
        • Belli S.
        • Parlakgumus A.
        • Caliskan K.
        • Noyan T.
        Reaching proficiency in laparoscopic splenectomy.
        World J Gastroenterol. 2009; 15: 4005-4008
        • Abbott K.L.
        • Chen X.
        • Clark M.
        • Bibler Zaidi N.L.
        • Swanson D.B.
        • George B.C.
        Number of operative performance ratings needed to reliably assess the difficulty of surgical procedures.
        J Surg Educ. 2019; 76: e189-e192
        • Woehr D.J.
        • Huffcutt A.I.
        Rater training for performance appraisal: a quantitative review.
        J Occup Organ Psychol. 1994; 67: 189-205
        • Thomas C.
        • Elsey E.
        • Boyce T.
        • Catton J.
        • Lewis W.
        Data from two deaneries regarding the ability of higher surgical training programmes to deliver curriculum targets.
        Bull R Coll Surg Engl. 2015; 97: e50-e53
        • Beamish A.J.
        • Johnston M.J.
        • Harries R.L.
        • et al.
        Use of the eLogbook in surgical training in the United Kingdom: a nationwide survey and consensus recommendations from the Association of Surgeons in Training.
        Int J Surg. 2020; 84: 199-206
        • Hope C.
        • Reilly J.J.
        • Griffiths G.
        • Lund J.
        • Humes D.
        The impact of COVID-19 on surgical training: a systematic review.
        Tech Coloproctol. 2021; 25: 505-520
        • Joyce D.P.
        • Ryan D.
        • Kavanagh D.O.
        • Traynor O.
        • Tierney S.
        Impact of COVID-19 on operative experience of junior surgical trainees.
        Br J Surg. 2021; 108: e33-e34