Advertisement

Time crunch: increasing the efficiency of assessment of technical surgical skill via brief video clips

Published:January 18, 2018DOI:https://doi.org/10.1016/j.surg.2017.11.011

      Abstract

      Background

      Video review for assessment of surgical performance is gaining popularity but is time consuming for busy expert reviewers, making review delays inevitable. Decreasing review time and including nonexpert reviewers may facilitate more timely reviews. We hypothesized that a shorter duration video clip would not affect the quality of expert ratings compared with full-length review. A secondary aim was to examine the reliability between expert and novice raters and how it was affected by video clip duration.

      Methods

      Videos of laparoscopic suturing performed on a live porcine model by premedical students, surgery residents, and fully trained surgeons were edited into 3 different durations: full, part, and 30-second versions. Video clips (n = 36) were reviewed by experienced surgeons (n = 3) and novice volunteers (n = 4) using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) assessment. Videos were assigned randomly during 3 review cycles held 1 week apart. Each cycle included 1 iteration of the 12 performance videos. We assessed the impact of the duration of the video on reviewer scores and inter-rater reliability.

      Results

      Variance in scores for 2 of 4 GOALS domains was dependent on the duration of the video clip (P < .05). Total GOALS scores were greater for part and 30-second clips compared with full clips (P < .05). Inter-rater reliability was greatest for full clips (intraclass correlation = 0.68) and decreased significantly for shorter clips.

      Conclusion

      Our hypothesis was rejected as shorter video durations for surgical performance assessment led to inflated reviewer ratings both for expert and novice reviewers. Shortening duration of the video cannot be recommended for accurate performance assessment.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Herrera-Almario G.E.
        • Kirk K.
        • Guerrero V.T.
        • Jeong K.
        • Kim S.
        • Hamad G.G.
        The effect of video review of resident laparoscopic surgical skills measured by self- and external assessment.
        Am J Surg. 2016; 211: 315-320
        • Scott D.J.
        • Rege R.V.
        • Bergen P.C.
        • et al.
        Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation.
        J Laparoendosc Adv Surg Tech A. 2000; 10: 183-190
        • Vassiliou M.C.
        • Feldman L.S.
        • Fraser S.A.
        • et al.
        Evaluating intraoperative laparoscopic skill: direct observation versus blinded videotaped performances.
        Surg Innov. 2007; 14: 211-216
        • Hu Y.Y.
        • Mazer L.M.
        • Yule S.J.
        • et al.
        Complementing operating room teaching with video-based coaching.
        JAMA Surg. 2016;
        • Soucisse M.L.
        • Boulva K.
        • Sideris L.
        • Drolet P.
        • Morin M.
        • Dube P.
        Video coaching as an efficient teaching method for surgical residents-a randomized controlled trial.
        J Surg Educ. 2017; 74: 365-371
        • Bridges M.
        • Diamond D.L.
        The financial impact of teaching surgical residents in the operating room.
        Am J Surg. 1999; 177: 28-32
        • Kairys J.C.
        • DiMuzio P.J.
        • Crawford A.G.
        • Grabo D.J.
        • Yeo C.J.
        Changes in operative case experience for general surgery residents: has the 80-hour work week decreased residents' operative experience?.
        Adv Surg. 2009; 43: 73-90
        • Kairys J.C.
        • McGuire K.
        • Crawford A.G.
        • Yeo C.J.
        Cumulative operative experience is decreasing during general surgery residency: a worrisome trend for surgical trainees?.
        J Am Coll Surg. 2008; 206 (discussion 11-3): 804-811
        • Carlin A.M.
        • Gasevic E.
        • Shepard A.D.
        Effect of the 80-hour work week on resident operative experience in general surgery.
        Am J Surg. 2007; 193 (discussion 9–30): 326-329
        • Antiel R.M.
        • Reed D.A.
        • Van Arendonk K.J.
        • et al.
        Effects of duty hour restrictions on core competencies, education, quality of life, and burnout among general surgery interns.
        JAMA Surg. 2013; 148: 448-455
        • Yeo H.
        • Viola K.
        • Berg D.
        • et al.
        Attitudes, training experiences, and professional expectations of US general surgery residents: a national survey.
        JAMA. 2009; 302: 1301-1308
        • Acton R.D.
        • Chipman J.G.
        • Lunden M.
        • Schmitz C.C.
        Unanticipated teaching demands rise with simulation training: strategies for managing faculty workload.
        J Surg Educ. 2015; 72: 522-529
        • Vaughn C.J.
        • Kim E.
        • O'Sullivan P.
        • et al.
        Peer video review and feedback improve performance in basic surgical skills.
        Am J Surg. 2016; 211: 355-360
        • Lendvay T.S.
        • White L.
        • Kowalewski T.
        Crowdsourcing to assess surgical skill.
        JAMA Surg. 2015; 150: 1086-1087
        • Nesbitt C.
        • Phillips A.W.
        • Searle R.
        • Stansby G.
        Student views on the use of 2 styles of video-enhanced feedback compared to standard lecture feedback during clinical skills training.
        J Surg Educ. 2015; 72: 969-973
        • Dunn M.M.
        • Woolliscroft J.O.
        Assessment of a surgical pattern recognition examination.
        Am J Surg. 1995; 169: 341-343
        • Vassiliou M.C.
        • Feldman L.S.
        • Andrew C.G.
        • et al.
        A global assessment tool for evaluation of intraoperative laparoscopic skills.
        Am J Surg. 2005; 190: 107-113
        • Deal S.B.
        • Lendvay T.S.
        • Haque M.I.
        • et al.
        Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills.
        Am J Surg. 2016; 211: 398-404
        • Govaerts M.J.
        • Schuwirth L.W.
        • Van der Vleuten C.P.
        • Muijtjens A.M.
        Workplace-based assessment: effects of rater expertise.
        Adv Health Sci Educ Theory Pract. 2011; 16: 151-165
        • Watanabe Y.
        • Madani A.
        • Ito Y.M.
        • et al.
        Psychometric properties of the Global Operative Assessment of Laparoscopic Skills (GOALS) using item response theory.
        Am J Surg. 2017; 213: 273-276
        • Gallagher A.G.
        • Ritter E.M.
        • Satava R.M.
        Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training.
        Surg Endosc. 2003; 17: 1525-1529
        • Korndorffer Jr, J.R.
        • Kasten S.J.
        • Downing S.M.
        A call for the utilization of consensus standards in the surgical education literature.
        Am J Surg. 2010; 199: 99-104