Advertisement

Global versus task-specific postoperative feedback in surgical procedure learning

Open AccessPublished:February 12, 2021DOI:https://doi.org/10.1016/j.surg.2020.12.038

      Abstract

      Background

      Task-specific checklists and global rating scales are both recommended assessment tools to provide constructive feedback on surgical performance. This study evaluated the most effective feedback tool by comparing the effects of the Observational Clinical Human Reliability Analysis (OCHRA) and the Objective Structured Assessment of Technical Skills (OSATS) on surgical performance in relation to the visual-spatial ability of the learners.

      Methods

      In a randomized controlled trial, medical students were allocated to either the OCHRA (n = 25) or OSATS (n = 25) feedback group. Visual-spatial ability was measured by a Mental Rotation Test. Participants performed an open inguinal hernia repair procedure on a simulation model twice. Feedback was provided after the first procedure. Improvement in performance was evaluated blindly using a global rating scale (performance score) and hand-motion analysis (time and path length).

      Results

      Mean improvement in performance score was not significantly different between the OCHRA and OSATS feedback groups (P = .100). However, mean improvement in time (371.0 ± 223.4 vs 274.6 ± 341.6; P = .027) and path length (53.5 ± 42.4 vs 34.7 ± 39.0; P = .046) was significantly greater in the OCHRA feedback group. When stratified by mental rotation test scores, the greater improvement in time (P = .032) and path length (P = .053) was observed only among individuals with low visual-spatial abilities.

      Conclusion

      A task-specific (OCHRA) feedback is more effective in improving surgical skills in terms of time and path length in novices compared to a global rating scale (OSATS). The effects of a task-specific feedback are present mostly in individuals with lower visual-spatial abilities.

      Introduction

      Feedback has long been recognized for its positive effect in surgical knowledge and skills training.
      • El Boghdady M.
      • Alijani A.
      Feedback in surgical education.
      It has been shown to be crucial in technical skill development because it increases motivation, prevents incorrect actions, and reinforces correct actions.
      • Porte M.C.
      • Xeroulis G.
      • Reznick R.K.
      • Dubrowski A.
      Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills.
      • Xeroulis G.J.
      • Park J.
      • Moulton C.-A.
      • Reznick R.K.
      • LeBlanc V.
      • Dubrowski A.
      Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback.
      Feedback can be provided based on direct observation of technical skills.
      • Ahmed K.
      • Miskovic D.
      • Darzi A.
      • Athanasiou T.
      • Hanna G.B.
      Observational tools for assessment of procedural skills: a systematic review.
      Within the surgical field, different observational assessment tools are available.
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      Assessment tools assess surgical performance on competences, skills, or surgical-specific items on a checklist. These tools can be used as a medium for feedback to provide information regarding a trainee’s performance to improve on specific items that are being assessed.
      • El Boghdady M.
      • Alijani A.
      Feedback in surgical education.
      ,
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      Two main types of assessment tools can be recognized: global rating scales, which rate general surgical skills and are applicable to all surgical procedures, or procedure-specific checklists.
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      In both categories, many tools have been developed and validated.
      • Ahmed K.
      • Miskovic D.
      • Darzi A.
      • Athanasiou T.
      • Hanna G.B.
      Observational tools for assessment of procedural skills: a systematic review.
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      A commonly used and generally accepted as “gold standard” assessment tool is Objective Structured Assessment of Technical Skills (OSATS), a global rating scale introduced by Martin et al for assessing technical skills of an entire surgical procedure.
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      • Martin J.A.
      • Regehr G.
      • Reznick R.
      • et al.
      Objective structured assessment of technical skill (OSATS) for surgical residents.
      OSATS is a reliable, validated tool that assesses 7 competencies on a 5-point Likert scale.
      • Martin J.A.
      • Regehr G.
      • Reznick R.
      • et al.
      Objective structured assessment of technical skill (OSATS) for surgical residents.
      It is feasible and effective in assessment of surgical skills of trainees in the operating room.
      • Niitsu H.
      • Hirabayashi N.
      • Yoshimitsu M.
      • et al.
      Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room.
      Although global rating scales such as the OSATS are easy in use, these scales can be imprecise.
      • Ahmed K.
      • Miskovic D.
      • Darzi A.
      • Athanasiou T.
      • Hanna G.B.
      Observational tools for assessment of procedural skills: a systematic review.
      A task-specific method may provide more concise and precise feedback.
      • Ahmed K.
      • Miskovic D.
      • Darzi A.
      • Athanasiou T.
      • Hanna G.B.
      Observational tools for assessment of procedural skills: a systematic review.
      A task-specific technical skills assessment method is the Observational Clinical Human Reliability Analysis (OCHRA).
      • Tang B.
      • Hanna G.B.
      • Joice P.
      • Cuschieri A.
      Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy.
      An OCHRA checklist assesses in a stepwise manner whether a substep was correct or incorrect.
      • Tang B.
      • Hanna G.B.
      • Joice P.
      • Cuschieri A.
      Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy.
      Both OSATS and OCHRA assessment tools have shown to be valid for providing constructive feedback.
      • Ahmed K.
      • Miskovic D.
      • Darzi A.
      • Athanasiou T.
      • Hanna G.B.
      Observational tools for assessment of procedural skills: a systematic review.
      ,
      • Niitsu H.
      • Hirabayashi N.
      • Yoshimitsu M.
      • et al.
      Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room.
      However, according to constructive alignment theory, the OCHRA feedback might be more effective when the surgical procedure is also learned in a stepwise manner.
      • Biggs J.
      Enhancing teaching through constructive alignment.
      Although the validity of OSATS and OCHRA is demonstrated, these assessment tools are still based on individual judgments, which are inevitably associated with subjectivity.
      • Levin M.
      • McKechnie T.
      • Khalid S.
      • Grantcharov T.P.
      • Goldenberg M.
      Automated methods of technical skill assessment in surgery: a systematic review.
      Quantifying measures of technical skills may potentially mitigate this subjectivity. For open surgery, different motion tracking devices are described to measure either hand or instrument movements.
      • Datta V.
      • Mackay S.
      • Mandalia M.
      • Darzi A.
      The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model.
      • Shaharan S.
      • Nugent E.
      • Ryan D.M.
      • Traynor O.
      • Neary P.
      • Buckley D.
      Basic surgical skill retention: can patriot motion tracking system provide an objective measurement for it?.
      • Genovese B.
      • Yin S.
      • Sareh S.
      • et al.
      Surgical hand tracking in open surgery using a versatile motion sensing system: are we there yet?.
      • Corvetto M.A.
      • Fuentes C.
      • Araneda A.
      • et al.
      Validation of the imperial college surgical assessment device for spinal anesthesia.
      The outcomes of time to complete a task and total path length can differentiate between novices and experts.
      • Genovese B.
      • Yin S.
      • Sareh S.
      • et al.
      Surgical hand tracking in open surgery using a versatile motion sensing system: are we there yet?.
      • Corvetto M.A.
      • Fuentes C.
      • Araneda A.
      • et al.
      Validation of the imperial college surgical assessment device for spinal anesthesia.
      • Wanzel K.R.
      • Hamstra S.J.
      • Caminiti M.F.
      • Anastakis D.J.
      • Grober E.D.
      • Reznick R.K.
      Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
      Additionally, the effect of feedback in relation to visual-spatial ability, as another determining factor for technical skills development, is unrecognized. Visual-spatial ability is defined as the ability that allows individuals to construct visual-spatial (ie, 3-dimensional) mental representations of 2D images and to mentally manipulate these representations.
      • Gordon H.W.
      The cognitive laterality battery: tests of specialized cognitive function.
      ,
      • Kozhevnikov M.
      • Hegarty M.
      A dissociation between object manipulation spatial ability and spatial orientation ability.
      This ability determines how well individuals are able to translate the acquired anatomical knowledge into clinical and surgical practice. Consequently, visual-spatial ability determines how well surgical residents are able to understand and perform spatially complex procedures. The positive association between visual-spatial ability and acquisition of surgical skills, including quality of hand motion, has been observed especially in the early phases of surgical training.
      • Wanzel K.R.
      • Hamstra S.J.
      • Caminiti M.F.
      • Anastakis D.J.
      • Grober E.D.
      • Reznick R.K.
      Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
      ,
      • Vajsbaher T.
      • Schultheis H.
      • Francis N.K.
      Spatial cognition in minimally invasive surgery: a systematic review.
      • Langlois J.
      • Bellemare C.
      • Toulouse J.
      • Wells G.A.
      Spatial abilities and technical skills performance in health care: a systematic review.
      • Roach V.A.
      • Brandt M.G.
      • Moore C.C.
      • Wilson T.D.
      Is three-dimensional videography the cutting edge of surgical skill acquisition?.
      Moreover, visual-spatial ability can have a modifying effect on outcomes. Individuals with lower visual-spatial abilities tend to perform worse than individuals with high visual-spatial abilities on acquisition of anatomical knowledge and surgical skills. However, with supportive instructional methods and deliberate practice and feedback they are able to achieve a comparable level of competency.
      • Wanzel K.R.
      • Hamstra S.J.
      • Caminiti M.F.
      • Anastakis D.J.
      • Grober E.D.
      • Reznick R.K.
      Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
      ,
      • Bogomolova K.
      • van der Ham I.J.M.
      • Dankbaar M.E.W.
      • et al.
      The effect of stereoscopic augmented reality visualization on learning anatomy and the modifying effect of visual-spatial abilities: a double-center randomized controlled trial.
      • Cui D.
      • Wilson T.D.
      • Rockhold R.W.
      • Lehman M.N.
      • Lynch J.C.
      Evaluation of the effectiveness of 3D vascular stereoscopic models in anatomy instruction for first year medical students.
      • Roach V.A.
      • Mistry M.R.
      • Wilson T.D.
      Spatial visualization ability and laparoscopic skills in novice learners: evaluating stereoscopic versus monoscopic visualizations.
      The aim of this study was to investigate whether a task-specific, stepwise feedback checklist (OCHRA) leads to a greater improvement in performance of a surgical procedure compared to a global rating scale method (OSATS) in terms of improvement of overall performance score, time to complete task, and total path length. These outcomes were also evaluated in relation to learners’ visual-spatial ability.

      Methods

      Study design and population

      A randomized controlled trial was conducted at the Leiden University Medical Center, The Netherlands. Participants were medical students and novices to almost any type of surgical procedures. Only right-handed students were included because left-handed novice students may have difficulties with the surgical instruments.
      • Tchantchaleishvili V.
      • Myers P.O.
      Left-handedness—a handicap for training in surgery?.
      Participation was voluntary, and written consent was obtained from all participants. The study protocol was approved by the Netherlands Association for Medical Education Ethical Review Board (NERB dossier number: 1013) (Fig 1).

      Randomization

      Participants were randomly allocated to either the OCHRA feedback (n = 25) or OSATS feedback group (n = 25) using an Excel random group generator.

      Surgical procedure

      The Lichtenstein open inguinal hernia repair was chosen as a procedure containing multiple surgical steps and because of its spatial complexity, which requires a certain level of surgical anatomical knowledge and visual-spatial ability of the learner. The first part of the surgery, until resecting the hernia sac, requires solely basic surgical skills such as incising, dissecting, and ligating. The second part, the placement and fixation of the mesh, is more complex. Each participant performed the Lichtenstein open inguinal hernia repair 2 times on a validated simulation model.
      • Nazari T.
      • Simons M.
      • Zeb M.
      • et al.
      Validity of a low-cost Lichtenstein open inguinal hernia repair simulation model for surgical training.
      Participants were given access to the online course 1 week before the experiment to prepare for the experiment. The course consisted of 3 components: an introductory description that included text and figures regarding the surgical anatomy, a stepwise textual description, and a video demonstration of the procedure on the identical model used during the experiment (Supplementary Table S1).
      • Nazari T.
      • Vlieger E.
      • Dankbaar M.
      • van Merriënboer J.
      • Lange J.
      • Wiggers T.
      Creation of a universal language for surgical procedures using the step-by-step framework.
      The video demonstration depicted all important steps that need to be undertaken during surgery. Video was accompanied by auditory explanation. Participants were able to retrieve the materials as many times as they wanted and were able to do it on their own pace.
      On the day of experiment, participants were given 30 minutes to complete each procedure.
      • Nazari T.
      • van de Graaf F.W.
      • Dankbaar M.E.W.
      • Lange J.F.
      • van Merriënboer J.J.G.
      • Wiggers T.
      One step at a time: step by step versus continuous video-based learning to prepare medical students for performing surgical procedures.
      The second procedure was performed directly after the provided feedback on the first procedure. Both procedures were recorded on video for blinded assessment. Participants were wearing a right-hand glove for the recording of motion by a motion tracking device (PST Base, PS-Tech B.V., Amsterdam, The Netherlands).

      Demographic questionnaire

      The questionnaire was administered before the experiment to account for factors that could possibly influence the performance (Supplementary Table S1). In a previous study, the time students studied for the open inguinal hernia repair with the use of a video demonstration had a significant modifying effect on surgical performance.
      • Nazari T.
      • van de Graaf F.W.
      • Dankbaar M.E.W.
      • Lange J.F.
      • van Merriënboer J.J.G.
      • Wiggers T.
      One step at a time: step by step versus continuous video-based learning to prepare medical students for performing surgical procedures.
      Therefore, study time was included in the questionnaire and was accounted for in the data analysis.

      Visual-spatial ability

      Visual-spatial ability was measured by the mental rotation test (MRT) before the experiment. The MRT is a validated 24-item psychometric test and is the gold standard in assessing visual-spatial ability in anatomical and surgical education.
      • Langlois J.
      • Bellemare C.
      • Toulouse J.
      • Wells G.A.
      Spatial abilities and technical skills performance in health care: a systematic review.
      ,
      • Langlois J.
      • Bellemare C.
      • Toulouse J.
      • Wells G.A.
      Spatial abilities and anatomy knowledge assessment: a systematic review.
      • Vandenberg S.G.
      • Kuse A.R.
      Mental rotations, a group test of three-dimensional spatial visualization.
      • Wanzel K.R.
      • Hamstra S.J.
      • Anastakis D.J.
      • Matsumoto E.D.
      • Cusimano M.D.
      Effect of visual-spatial ability on learning of spatially-complex surgical skills.
      Participants were given 10 minutes to complete the test. The maximum possible score for the test was 24 points.

      Interventions

      In the OCHRA feedback group, postoperative feedback was provided using OCHRA. The OCHRA checklist is a reliable and valid instrument that has been successfully used in assessment of performance in various surgical procedures.
      • Tang B.
      • Hanna G.B.
      • Joice P.
      • Cuschieri A.
      Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy.
      ,
      • Foster J.
      • Miskovic D.
      • Allison A.
      • et al.
      Application of objective clinical human reliability analysis (OCHRA) in assessment of technical performance in laparoscopic rectal cancer surgery.
      • Miskovic D.
      • Ni M.
      • Wyles S.M.
      • Parvaiz A.
      • Hanna G.B.
      Observational clinical human reliability analysis (OCHRA) for competency assessment in laparoscopic colorectal surgery at the specialist level.
      • Talebpour M.
      • Alijani A.
      • Hanna G.
      • Moosa Z.
      • Tang B.
      • Cuschieri A.
      Proficiency-gain curve for an advanced laparoscopic procedure defined by observation clinical human reliability assessment (OCHRA).
      It is a procedure-specific step-by-step skills assessment checklist that is characterized by a breakdown of a procedure into tasks.
      • Nazari T.
      • Vlieger E.
      • Dankbaar M.
      • van Merriënboer J.
      • Lange J.
      • Wiggers T.
      Creation of a universal language for surgical procedures using the step-by-step framework.
      Each step is assessed for being performed correctly and if errors are being made during the particular step. Provided feedback was based on the evaluation of each performed procedural step (Supplementary Table S2). If a particular step was performed incorrectly, the error was discussed and a proper execution of the step was explained. No points or final scores were awarded for the performance.
      In the OSATS feedback group, postoperative feedback was provided using the OSATS assessment tool (Supplementary Table S3). OSATS is a 7-item global rating scale that focuses on the following overall competencies: (1) respect for tissue, (2) time and motion, (3) instrument handling, (4) knowledge of instruments, (5) use of assistance, (6) flow of operation, and (7) knowledge of procedure.
      • Martin J.A.
      • Regehr G.
      • Reznick R.
      • et al.
      Objective structured assessment of technical skill (OSATS) for surgical residents.
      The tool has been previously validated in a wide range of surgical procedures and disciplines with reasonable index of reliability.
      • Martin J.A.
      • Regehr G.
      • Reznick R.
      • et al.
      Objective structured assessment of technical skill (OSATS) for surgical residents.
      ,
      • Faulkner H.
      • Regehr G.
      • Martin J.
      • Reznick R.
      Validation of an objective structured assessment of technical skill for surgical residents.
      ,
      • Vaidya A.
      • Aydin A.
      • Ridgley J.
      • Raison N.
      • Dasgupta P.
      • Ahmed K.
      Current status of technical skills assessment tools in surgery: a systematic review.
      Provided feedback was based on the evaluation of each of the 7 competencies in the exact order of OSATS. Suboptimal performance and errors made within a competence were discussed based on an example followed by an explanation for the improvement. No points or final scores were awarded for the performance to avoid any bias that could be introduced by grading the performance during the feedback phase.
      In both groups, feedback was provided immediately after performing the first procedure. The total feedback time was held constant in both conditions and was approximately 10 minutes. Feedback was provided by 1 of the 2 researchers who were trained in providing both types of feedback in the context of this experiment. Care was taken to ensure that the feedback was complete and that participants were able to ask questions and verify whether they understood the information properly.

      Performance score

      Video-recorded procedures were assessed blindly by 2 independent researchers using OSATS, as the most common assessment tool for surgical performance. A minimum of 1 and a maximum of 5 points could be awarded for each of the 7 competences. A maximum possible performance score for each procedure was 35 points. Both researchers were trained in assessment of recorded procedures. Training was facilitated by a surgeon who is an expert in this field. It included a comprehensive study of the procedure using the provided study material followed by execution of the procedure on the model themselves. After that, researchers were trained in assessment until they got sufficiently familiar with all aspects of OSATS. The actual assessment of recorded procedures was performed independently. In case of discrepancies, consensus was reached by re-evaluating the procedure. Additionally, 5% of procedures were randomly selected and assessed by the expert to detect any discrepancies in scoring. No differences in ratings were identified.

      Motion tracking

      Motion tracking analysis was performed using a combination of a commercially available optical tracker system (PST Base, PS-Tech B.V., Amsterdam, The Netherlands) and a customized glove for the dominant right hand. This could track 6 degrees of freedom position in Cartesian coordinates (X, Y, and Z axis) at a rate of 30 samples per second. Time to complete the task and path length were measured. These have shown to be excellent markers of surgical performance.
      • Datta V.
      • Mackay S.
      • Mandalia M.
      • Darzi A.
      The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model.
      ,
      • Atesok K.
      • Satava R.M.
      • Marsh J.L.
      • Hurwitz S.R.
      Measuring surgical skills in simulation-based training.
      • Binkley J.
      • Bukoski A.D.
      • Doty J.
      • Crane M.
      • Barnes S.L.
      • Quick J.A.
      Surgical simulation: markers of proficiency.
      • Datta V.
      • Chang A.
      • Mackay S.
      • Darzi A.
      The relationship between motion analysis and surgical technical assessments.
      Because not all participants were able to complete the procedure within 30 minutes, the completion of the step of hernia sac removal was chosen as the endpoint for the outcomes of motion tracking analysis.

      Outcomes

      The study outcomes were defined as the differences in mean improvement in performance score (as measured by the OSATS assessment tool; time (in seconds) and path length (in meters) between the first and the second procedure between 2 groups. Outcomes were stratified by MRT scores. Individuals who scored below the mean were assigned to the MRT-low group (n = 22). Students who scored above the mean were assigned to the MRT-high group (n = 28).

      Statistical analysis

      Because of the novelty of this study, no previous data were available to calculate the sample size. A sample size of 50 participants was assumed to be appropriate. Participants’ baseline characteristics were summarized using descriptive statistics. Differences in baseline measurements were assessed with an independent t test for differences in means and χ2 test for differences in proportions. The differences in mean performance scores of the first procedure between groups were assessed with an independent t test. The improvement between second and first procedure within a group was assessed with a paired t test. The difference in mean improvement (Δ) in performance scores between second and first procedure between groups were assessed with a 1-way ANCOVA. ΔPerformance score was included as dependent variable, intervention group and study time as fixed factor (0–1 vs 1–2 vs 2–3 hours), and performance score on the first procedure and MRT score as covariates. Additionally, the outcomes were stratified by MRT score to evaluate the effect of intervention for different levels of visual-spatial ability. The analyses were repeated for mean improvement in time (Δ time) and path length (Δ path length). Partial eta squared was calculated and used as an effect size (0.2 = small effect, 0.5 = moderate effect 0.8 = large effect). Analyses were performed using SPSS statistical software package version 25.0 for Windows (IBM Corp, Armonk, NY). Statistical significance was determined at the level of P < .05.

      Results

      A total of 50 medical students was included. There were no significant differences between groups on baseline characteristics, as shown in Table I.
      Table IBaseline characteristics of the included participants
      CharacteristicOCHRA feedback (N = 25)OSATS feedback (N = 25)P value
      Sex, n (%)
       Male10 (40)13 (52).571
       Female15 (60)12 (48)
      Age, mean (± SD), in years21.5 (2.2)21.2 (1.9).537
      Study phase, n (%)
       Bachelor students15 (60)14 (56).302
       Master students10 (40)11 (44)
      Time spent studying online course, n (%)
       0–1 hours8 (32)5 (20).288
       1–2 hours16 (64)16 (64)
       2–3 hours1 (4)4 (16)
      I liked the way the hernia repair was taught, median [IQR]
      Rated on a 10-point scale from “not at all” to “completely.”
      8.0 [7.0–9.0]7.0 [6.5–8.7].104
      I felt prepared after completing the online course, mean (± SD)
      Rated on a 10-point scale from “not at all” to “completely.”
      6.3 (1.2)6.1 (2.1).679
      Times seen open inguinal hernia repair surgery in real life, median [IQR]0.0 [0.0–0.5]0.0 [0.0–0.5].984
      Other sources used to study, n (%)
       Not used16 (64)11 (44).256
       Yes9 (36)14 (56)
      Time spent studying other sources, n (%)
       0–1 hours8 (88.9)13 (92.9)1.00
       1–2 hours1 (11.1)1 (7.1)
      Hours of sleep last night, median [IQR]7.0 [6.0–8.0]8.0 [7.0–8.0].471
      Alcohol consumption last night, median [IQR]0.0 [0.0–0.8]0.0 [0.0–0.0].402
      Coffee consumption before surgical performance, median [IQR]1.0 [0.0–1.0]0.5 [0.0–1.0].879
      Other circumstances that could have affected the surgical performance, n (%)
       Not used18 (72)22 (88).289
       Yes7 (28)3 (12)
      Mental Rotation Test score, mean (± SD)16.4 (5.5)16.7 (4.9).872
      OCHRA, Observational Clinical Human Reliability Analysis; OSATS, Objective Structured Assessment of Technical Skills; n, number of participants; SD, standard deviation; IQR, interquartile range.
      Rated on a 10-point scale from “not at all” to “completely.”
      Both groups improved significantly in terms of total OSATS score, time, and path length between the first and second time of performing the procedure (Table II). Since not all participants were able to complete the procedure within 30 minutes, the completion of the step of hernia sac removal was chosen as the endpoint for the outcome measures time (s) and path length (m). This step was performed by 42 (84%) of participants. Path length data of 5 of the participants was lacking due to technical issues.
      Table IIDifferences in performance scores, time, and path length between 2 interventions
      OCHRA feedbackOSATS feedbackP value
      Performance scoren = 25n = 25
      First procedure17.4 ± 3.117.4 ± 3.8.935
      Second procedure23.5 ± 5.421.8 ± 4.9.100
      Differences assessed with ANCOVA.
      Δ6.2 ± 3.5
      P < 0.001 paired t test.
      4.4 ± 4.7
      P < 0.001 paired t test.
      Time (seconds)n = 20n = 22
      First procedure1239.6 ± 274.81300.4 ± 382.3.561
      Second procedure868.6 ± 151.61025.7 ± 286.4.027
      Differences assessed with ANCOVA.
      Δ371.0 ± 223.4
      P < 0.001 paired t test.
      274.6 ± 341.6
      P < 0.001 paired t test.
      Path length (m)n = 19n = 18
      First procedure168.4 ± 61.5168.9 ± 39.6.977
      Second procedure112.4 ± 36.2134.2 ± 36.3.046
      Differences assessed with ANCOVA.
      Δ53.5 ± 42.4
      P < 0.001 paired t test.
      34.7 ± 39.0
      P < 0.001 paired t test.
      Δ = delta, difference between second and first procedure; m, meters.
      OCHRA, Observational Clinical Human Reliability Analysis; OSATS, Objective Structured Assessment of Technical Skills.
      P < 0.001 paired t test.
      Differences assessed with ANCOVA.
      The mean improvement in performance scores was not significantly different between the 2 groups (β = 2.1; 95% IC [–0.41 to –4.5]; η2 = 0.06; P = .100). However, the mean improvement in time (β = –139.4; 95% CI [–.262.5 to –16.5]; η2 = 0.13; P = .027) and in path length (β = –21.2; 95% CI [–41.9 to –0.5]; η2 = 0.13; P = .046) was significantly greater in the OCHRA feedback group.

      Effect of visual-spatial ability

      When outcomes were stratified by MRT scores, the greater improvement in time in the OCHRA feedback group was observed only among individuals with lower visual-spatial abilities (β = –220.2; 95% CI [–418.4 to –22,1]; η2 = 0.26; P = .032) (Fig 2). As shown in Figure 3, a similar trajectory was observed for the improvement in path length. However, this difference did not reach the significance level (β = –28.2; 95% CI [–56.8 to 0.42]; η2 = 0.24; P = .053). Regardless of intervention, MRT scores were significantly associated with mean improvement in time (β = –14.17; 95% CI [–26.9 to –2.6]; η2 = 0.14; P = .019), but not in path length (β = –0.74; 95% CI [–2.8 to 1.3]; η2 = 0.01; P = .469) and OSATS scores (β = 0.05; 95% CI [–0.2 to 0.3]; η2 = 0.004; P = .670).
      Figure thumbnail gr2
      Fig 2Differences in Δtime (s) between OCHRA feedback and OSATS feedback groups: (a) overall; (b) MRT-low group, and (c) MRT-high group; P < .05.
      Figure thumbnail gr3
      Fig 3Differences in Δpath length (mm) between OCHRA feedback and OSATS feedback group: (a) overall; (b) MRT-low group, and (c) MRT-high group; P < .05.

      Discussion

      The aim of this study was to investigate whether a task-specific, stepwise feedback checklist (OCHRA) leads to a greater improvement in surgical performance compared to a global rating scale feedback method (OSATS). The outcomes were evaluated in relation to visual-spatial ability. The mean improvement in performance scores was not significantly different between the OCHRA and OSATS feedback groups. However, the OCHRA feedback showed a significant improvement on performance in terms of time and path length, as measured by the hand-motion analysis system. The effects of OCHRA feedback were present mainly among individuals with lower visual-spatial abilities.
      The observed effectiveness of OCHRA feedback on surgical performance in a simplified hernia repair model, as a more precise and concise approach, is supported by the instructional alignment theory.
      • Cohen S.A.
      Instructional alignment: searching for a magic bullet.
      When training and assessment methods are aligned, the effects of instruction are up to 4 times greater than in nonaligned methods.
      • Cohen S.A.
      Instructional alignment: searching for a magic bullet.
      In the current study, participants prepared for the open inguinal hernia repair procedure using a stepwise video demonstration. As OCHRA feedback was based on the evaluation of the subsequent surgical steps instead of competencies as part of the OSATS feedback, a greater alignment between learning and feedback could be achieved. Although this did not result in a difference in outcome in terms of the surgical scores, differences were found for the time and path length. In this study, most participants could not finish the entire surgical procedure within the 30-minute timeframe. Possibly, differences in surgical scores would have been found if students did complete the entire surgical procedure. Additionally, the value of a checklist (OCHRA) and global rating scale (OSATS) assessments may depend on the level of learners’ experience.
      • Ilgen J.S.
      • Ma I.W.
      • Hatala R.
      • Cook D.A.
      A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment.
      Global rating scales have been reported to be more useful for learners with higher levels of expertise, whereas checklists may be more useful for novice learners, such as the participants in this study.
      • Ilgen J.S.
      • Ma I.W.
      • Hatala R.
      • Cook D.A.
      A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment.
      ,
      • Dankbaar M.E.
      • Stegers-Jager K.M.
      • Baarveld F.
      • et al.
      Assessing the assessment in emergency care training.
      The observed modifying effect of visual-spatial ability on time and path length leads to important considerations. First, the findings are in line with previous research reporting positive association between visual-spatial ability and hand motion.
      • Wanzel K.R.
      • Hamstra S.J.
      • Caminiti M.F.
      • Anastakis D.J.
      • Grober E.D.
      • Reznick R.K.
      Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
      ,
      • Groenier M.
      • Schraagen J.M.
      • Miedema H.A.
      • Broeders I.A.
      The role of cognitive abilities in laparoscopic simulator training.
      • Hedman L.
      • Ström P.
      • Andersson P.
      • Kjellin A.
      • Wredmark T.
      • Felländer-Tsai L.
      High-level visual-spatial ability for novices correlates with performance in a visual-spatial complex surgical simulator task.
      • Schlickum M.
      • Hedman L.
      • Enochsson L.
      • Henningsohn L.
      • Kjellin A.
      • Felländer-Tsai L.
      Surgical simulation tasks challenge visual working memory and visual-spatial ability differently.
      • Hedman L.
      • Klingberg T.
      • Enochsson L.
      • Kjellin A.
      • Felländer-Tsai L.
      Visual working memory influences the performance in virtual image-guided surgical intervention.
      However, by treating visual-spatial ability as a possible effect modifier, this study showed that this association was present only for individuals with lower levels of visual-spatial ability. This effect, also referred to as the aptitude-treatment effect,
      • Cook D.A.
      The research we still are not doing: an agenda for the study of computer-based learning.
      ,
      • Corraini P.
      • Olsen M.
      • Pedersen L.
      • Dekkers O.M.
      • Vandenbroucke J.P.
      Effect modification, interaction and mediation: an overview of theoretical insights for clinical investigators.
      has been repeatedly observed in the research field of anatomical education.
      • Bogomolova K.
      • van der Ham I.J.M.
      • Dankbaar M.E.W.
      • et al.
      The effect of stereoscopic augmented reality visualization on learning anatomy and the modifying effect of visual-spatial abilities: a double-center randomized controlled trial.
      • Cui D.
      • Wilson T.D.
      • Rockhold R.W.
      • Lehman M.N.
      • Lynch J.C.
      Evaluation of the effectiveness of 3D vascular stereoscopic models in anatomy instruction for first year medical students.
      • Roach V.A.
      • Mistry M.R.
      • Wilson T.D.
      Spatial visualization ability and laparoscopic skills in novice learners: evaluating stereoscopic versus monoscopic visualizations.
      ,
      • Cook D.A.
      The research we still are not doing: an agenda for the study of computer-based learning.
      Therefore, it is instrumental to consider possible modifying effects of visual-spatial ability on outcomes when designing new research. Second, the observed differences could be explained by the cognitive load theory.
      • van Merrienboer J.J.
      • Sweller J.
      Cognitive load theory in health professional education: design principles and strategies.
      Students with lower visual-spatial abilities are in general less effective in processing new spatial information in their working memory than students with higher visual-spatial abilities. However, in contrast to a global approach, the information from a task-specific stepwise feedback, building up on an already existing stepwise schema of a surgical procedure, could have decreased the cognitive load.
      • van Merrienboer J.J.
      • Sweller J.
      Cognitive load theory in health professional education: design principles and strategies.
      Subsequently, more working memory capacity could be created to process new procedural skills among low-performing individuals. This emphasizes the importance of an aptitude-based approach in learning and teaching surgical technical skills to novices. Lastly, the effect of visual-spatial ability on OSATS scores was found to be not significant. This could be because of the inability of most participants to complete the entire procedure within the given timeframe.
      OSATS was used both as an intervention and assessment scoring tool in this study. The rationale behind the choice to use the OSATS as an assessment scoring tool is that OSATS is considered to be the “gold standard” assessment tool for surgical performance and one of the few actually used in residency training and research.
      • Van Hove P.
      • Tuijthof G.
      • Verdaasdonk E.
      • Stassen L.
      • Dankelman J.
      Objective assessment of technical surgical skills.
      ,
      • Wanzel K.R.
      • Hamstra S.J.
      • Caminiti M.F.
      • Anastakis D.J.
      • Grober E.D.
      • Reznick R.K.
      Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
      In The Netherlands, OSATS is incorporated within the surgical residency training.
      • Hopmans C.J.
      • den Hoed P.T.
      • van der Laan L.
      • et al.
      Assessment of surgery residents’ operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study.
      Second, a systematic review comparing checklists with global rating scales as assessment tools reported that global rating scales might be better in capturing nuanced elements of expertise.
      • Ilgen J.S.
      • Ma I.W.
      • Hatala R.
      • Cook D.A.
      A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment.
      Other assessment tools for surgical performance, such as the recently reported Surgical Quality Assurance (SQA), could have been an option, and perhaps would have found differences in surgical performance.
      • Tsai A.Y.
      • Mavroveli S.
      • Miskovic D.
      • et al.
      Surgical quality assurance in COLOR III: standardization and competency assessment in a randomized controlled trial.
      The timing of feedback is still debated. Xeroulis et al distinguished feedback provided during the task (concurrent feedback) and feedback upon completing the task (summary feedback).
      • Xeroulis G.J.
      • Park J.
      • Moulton C.-A.
      • Reznick R.K.
      • LeBlanc V.
      • Dubrowski A.
      Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback.
      The latter was found to be superior for learning basic surgical skills; however, Al Fayyad et al found the opposite. In their study, concurrent (immediate) feedback was perceived as superior in learning basic surgical skills compared to summary (delayed) feedback.
      • Al Fayyadh M.J.
      • Hassan R.A.
      • Tran Z.K.
      • et al.
      Immediate auditory feedback is superior to other types of feedback for basic surgical skills acquisition.
      In our study, summary feedback was chosen because the students operated on a simulation model without the risk of doing any harm. With an actual patient, a trainee needs guidance from a surgeon using concurrent feedback to avoid harmful errors.

      Limitations

      This study has several limitations. First, the sample size could not be calculated beforehand due to the novelty of the study aim and design. Although it was sufficient to reveal significant differences in terms of time and path length, the sample size could have been too small to detect significant differences in OSATS scores. Second, not all participants were able to complete the procedure within given 30 minutes. As the step of hernia sac removal was reached by most participants, it was used as the endpoint to ensure a justified comparison in terms of time and path length. Allowing participants to complete the entire procedure would have provided a better display of their performance. Third, the participants were medical students with low and slightly various levels of anatomical knowledge and technical skills, including suturing. Due to random allocation, these differences are expected to have little to no effect on outcomes. Additionally, the mean improvement in outcome measures was chosen instead of the absolute scores to account for those differences. Another limitation is the possible inability to generalize the conclusions to left-handed students, because this study only included right-handed students. Furthermore, these findings cannot be generalized to other procedures outside of inguinal hernia repair. Last, the effect of OCHRA feedback was evaluated in a simulated environment. This study should be repeated among surgical residents with higher levels of anatomical knowledge and technical skills in a clinical setting on multiple procedures.
      The findings of this study have implications for both practice and research. In this study, the open inguinal hernia repair was chosen as an exemplary procedure. It is unknown whether an inguinal hernia repair simulation is ideally suited to detect the effects of different types of feedback on study outcomes. The implementation of structured, stepwise feedback that is aligned with the learning activities should be considered especially in the early phases of surgical training. The aligned stepwise instruction using stepwise video demonstrations and procedure-specific OCHRA checklist assessment can be transferred to other surgical procedures. The stepwise segmentation of a surgical procedure can be made using the step-by-step framework.
      • Nazari T.
      • Vlieger E.
      • Dankbaar M.
      • van Merriënboer J.
      • Lange J.
      • Wiggers T.
      Creation of a universal language for surgical procedures using the step-by-step framework.
      This stepwise description of a surgical procedure can then be used to create a procedure-specific OCHRA checklist. Moreover, an aptitude-based approach in teaching and learning of surgical procedural skills could be of benefit for individuals with lower visual-spatial abilities. As demonstrated, it is crucial to consider the modifying effect of visual-spatial ability on surgical outcomes when setting up new research. In fact, when overall outcomes are not evaluated for different levels of visual-spatial abilities, the real differences may remain unrevealed.
      In conclusion, a task-specific, stepwise feedback checklist (OCHRA) proves to be more effective in improving surgical skills, in terms of time and path length, among surgical novices compared to a global rating scale feedback (OSATS). The effects of a task-specific feedback are present mostly in individuals with lower visual-spatial abilities.

      Funding/Support

      None.

      Conflict of interest/Disclosure

      The authors declare no conflict of interest.

      Acknowledgment

      The authors sincerely thank Michel Dieben for his technical support in the use of the motion tracking device and interpretation of the data and Jan van Schaik for his support in recruitment of participants.

      Supplementary Materials

      References

        • El Boghdady M.
        • Alijani A.
        Feedback in surgical education.
        Surgeon. 2017; 15: 98-103
        • Porte M.C.
        • Xeroulis G.
        • Reznick R.K.
        • Dubrowski A.
        Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills.
        Amer J Surg. 2007; 193: 105-110
        • Xeroulis G.J.
        • Park J.
        • Moulton C.-A.
        • Reznick R.K.
        • LeBlanc V.
        • Dubrowski A.
        Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback.
        Surgery. 2007; 141: 442-449
        • Ahmed K.
        • Miskovic D.
        • Darzi A.
        • Athanasiou T.
        • Hanna G.B.
        Observational tools for assessment of procedural skills: a systematic review.
        Amer J Surg. 2011; 202: 469-480
        • Van Hove P.
        • Tuijthof G.
        • Verdaasdonk E.
        • Stassen L.
        • Dankelman J.
        Objective assessment of technical surgical skills.
        Br J Surg. 2010; 97: 972-987
        • Martin J.A.
        • Regehr G.
        • Reznick R.
        • et al.
        Objective structured assessment of technical skill (OSATS) for surgical residents.
        Br J Surg. 1997; 84: 273-278
        • Niitsu H.
        • Hirabayashi N.
        • Yoshimitsu M.
        • et al.
        Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room.
        Surgery Today. 2013; 43: 271-275
        • Tang B.
        • Hanna G.B.
        • Joice P.
        • Cuschieri A.
        Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy.
        Arch Surg. 2004; 139: 1215-1220
        • Biggs J.
        Enhancing teaching through constructive alignment.
        Higher Educ. 1996; 32: 347-364
        • Levin M.
        • McKechnie T.
        • Khalid S.
        • Grantcharov T.P.
        • Goldenberg M.
        Automated methods of technical skill assessment in surgery: a systematic review.
        J Surg Educ. 2019; 76: 1629-1639
        • Datta V.
        • Mackay S.
        • Mandalia M.
        • Darzi A.
        The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model.
        J Am Coll Surg. 2001; 193: 479-485
        • Shaharan S.
        • Nugent E.
        • Ryan D.M.
        • Traynor O.
        • Neary P.
        • Buckley D.
        Basic surgical skill retention: can patriot motion tracking system provide an objective measurement for it?.
        J Surg Educ. 2016; 73: 245-249
        • Genovese B.
        • Yin S.
        • Sareh S.
        • et al.
        Surgical hand tracking in open surgery using a versatile motion sensing system: are we there yet?.
        Amer Surg. 2016; 82: 872-875
        • Corvetto M.A.
        • Fuentes C.
        • Araneda A.
        • et al.
        Validation of the imperial college surgical assessment device for spinal anesthesia.
        BMC Anesthesiol. 2017; 17: 131
        • Wanzel K.R.
        • Hamstra S.J.
        • Caminiti M.F.
        • Anastakis D.J.
        • Grober E.D.
        • Reznick R.K.
        Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance.
        Surgery. 2003; 134: 750-757
        • Gordon H.W.
        The cognitive laterality battery: tests of specialized cognitive function.
        Int J Neurosci. 1986; 29: 223-244
        • Kozhevnikov M.
        • Hegarty M.
        A dissociation between object manipulation spatial ability and spatial orientation ability.
        Mem Cognit. 2001; 29: 745-756
        • Vajsbaher T.
        • Schultheis H.
        • Francis N.K.
        Spatial cognition in minimally invasive surgery: a systematic review.
        BMC Surg. 2018; 18: 94
        • Langlois J.
        • Bellemare C.
        • Toulouse J.
        • Wells G.A.
        Spatial abilities and technical skills performance in health care: a systematic review.
        Med Educ. 2015; 49: 1065-1085
        • Roach V.A.
        • Brandt M.G.
        • Moore C.C.
        • Wilson T.D.
        Is three-dimensional videography the cutting edge of surgical skill acquisition?.
        Anat Sci Educ. 2012; 5: 138-145
        • Bogomolova K.
        • van der Ham I.J.M.
        • Dankbaar M.E.W.
        • et al.
        The effect of stereoscopic augmented reality visualization on learning anatomy and the modifying effect of visual-spatial abilities: a double-center randomized controlled trial.
        Anat Sci Educ. 2019;
        • Cui D.
        • Wilson T.D.
        • Rockhold R.W.
        • Lehman M.N.
        • Lynch J.C.
        Evaluation of the effectiveness of 3D vascular stereoscopic models in anatomy instruction for first year medical students.
        Anat Sci Educ. 2017; 10: 34-45
        • Roach V.A.
        • Mistry M.R.
        • Wilson T.D.
        Spatial visualization ability and laparoscopic skills in novice learners: evaluating stereoscopic versus monoscopic visualizations.
        Anat Sci Educ. 2014; 7: 295-301
        • Tchantchaleishvili V.
        • Myers P.O.
        Left-handedness—a handicap for training in surgery?.
        J Surg Educ. 2010; 67: 233-236
        • Nazari T.
        • Simons M.
        • Zeb M.
        • et al.
        Validity of a low-cost Lichtenstein open inguinal hernia repair simulation model for surgical training.
        Hernia. 2019; 1–7
        • Nazari T.
        • Vlieger E.
        • Dankbaar M.
        • van Merriënboer J.
        • Lange J.
        • Wiggers T.
        Creation of a universal language for surgical procedures using the step-by-step framework.
        BJS Open. 2018; 2: 151-157
        • Nazari T.
        • van de Graaf F.W.
        • Dankbaar M.E.W.
        • Lange J.F.
        • van Merriënboer J.J.G.
        • Wiggers T.
        One step at a time: step by step versus continuous video-based learning to prepare medical students for performing surgical procedures.
        J Surg Educ. 2020; 77: 779-787
        • Langlois J.
        • Bellemare C.
        • Toulouse J.
        • Wells G.A.
        Spatial abilities and anatomy knowledge assessment: a systematic review.
        Anat Sci Educ. 2017; 10: 235-241
        • Vandenberg S.G.
        • Kuse A.R.
        Mental rotations, a group test of three-dimensional spatial visualization.
        Percept Mot Skills. 1978; 47: 599-604
        • Wanzel K.R.
        • Hamstra S.J.
        • Anastakis D.J.
        • Matsumoto E.D.
        • Cusimano M.D.
        Effect of visual-spatial ability on learning of spatially-complex surgical skills.
        Lancet (London, England). 2002; 359: 230-231
        • Foster J.
        • Miskovic D.
        • Allison A.
        • et al.
        Application of objective clinical human reliability analysis (OCHRA) in assessment of technical performance in laparoscopic rectal cancer surgery.
        Tech Coloproctol. 2016; 20: 361-367
        • Miskovic D.
        • Ni M.
        • Wyles S.M.
        • Parvaiz A.
        • Hanna G.B.
        Observational clinical human reliability analysis (OCHRA) for competency assessment in laparoscopic colorectal surgery at the specialist level.
        Surg Endosc. 2012; 26: 796-803
        • Talebpour M.
        • Alijani A.
        • Hanna G.
        • Moosa Z.
        • Tang B.
        • Cuschieri A.
        Proficiency-gain curve for an advanced laparoscopic procedure defined by observation clinical human reliability assessment (OCHRA).
        Surg Endosc. 2009; 23: 869-875
        • Faulkner H.
        • Regehr G.
        • Martin J.
        • Reznick R.
        Validation of an objective structured assessment of technical skill for surgical residents.
        Acad Med. 1996; 71: 1363-1365
        • Vaidya A.
        • Aydin A.
        • Ridgley J.
        • Raison N.
        • Dasgupta P.
        • Ahmed K.
        Current status of technical skills assessment tools in surgery: a systematic review.
        J Surg Res. 2020; 246: 342-378
        • Atesok K.
        • Satava R.M.
        • Marsh J.L.
        • Hurwitz S.R.
        Measuring surgical skills in simulation-based training.
        J Amer Acad Orthop Surg. 2017; 25: 665-672
        • Binkley J.
        • Bukoski A.D.
        • Doty J.
        • Crane M.
        • Barnes S.L.
        • Quick J.A.
        Surgical simulation: markers of proficiency.
        J Surg Educ. 2019; 76: 234-241
        • Datta V.
        • Chang A.
        • Mackay S.
        • Darzi A.
        The relationship between motion analysis and surgical technical assessments.
        Amer J Surg. 2002; 184: 70-73
        • Cohen S.A.
        Instructional alignment: searching for a magic bullet.
        Educ Res. 1987; 16: 16-20
        • Ilgen J.S.
        • Ma I.W.
        • Hatala R.
        • Cook D.A.
        A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment.
        Med Educ. 2015; 49: 161-173
        • Dankbaar M.E.
        • Stegers-Jager K.M.
        • Baarveld F.
        • et al.
        Assessing the assessment in emergency care training.
        PLoS One. 2014; 9e114663
        • Groenier M.
        • Schraagen J.M.
        • Miedema H.A.
        • Broeders I.A.
        The role of cognitive abilities in laparoscopic simulator training.
        Adv Health Sci Educ Theory Pract. 2014; 19: 203-217
        • Hedman L.
        • Ström P.
        • Andersson P.
        • Kjellin A.
        • Wredmark T.
        • Felländer-Tsai L.
        High-level visual-spatial ability for novices correlates with performance in a visual-spatial complex surgical simulator task.
        Surg Endosc. 2006; 20: 1275-1280
        • Schlickum M.
        • Hedman L.
        • Enochsson L.
        • Henningsohn L.
        • Kjellin A.
        • Felländer-Tsai L.
        Surgical simulation tasks challenge visual working memory and visual-spatial ability differently.
        World J Surg. 2011; 35: 710-715
        • Hedman L.
        • Klingberg T.
        • Enochsson L.
        • Kjellin A.
        • Felländer-Tsai L.
        Visual working memory influences the performance in virtual image-guided surgical intervention.
        Surg Endosc. 2007; 21: 2044-2050
        • Cook D.A.
        The research we still are not doing: an agenda for the study of computer-based learning.
        Acad Med. 2005; 80: 541-548
        • Corraini P.
        • Olsen M.
        • Pedersen L.
        • Dekkers O.M.
        • Vandenbroucke J.P.
        Effect modification, interaction and mediation: an overview of theoretical insights for clinical investigators.
        Clin Epidemiol. 2017; 9: 331-338
        • van Merrienboer J.J.
        • Sweller J.
        Cognitive load theory in health professional education: design principles and strategies.
        Med Educ. 2010; 44: 85-93
        • Hopmans C.J.
        • den Hoed P.T.
        • van der Laan L.
        • et al.
        Assessment of surgery residents’ operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study.
        Surgery. 2014; 156: 1078-1088
        • Tsai A.Y.
        • Mavroveli S.
        • Miskovic D.
        • et al.
        Surgical quality assurance in COLOR III: standardization and competency assessment in a randomized controlled trial.
        Ann Surg. 2019; 270: 768-774
        • Al Fayyadh M.J.
        • Hassan R.A.
        • Tran Z.K.
        • et al.
        Immediate auditory feedback is superior to other types of feedback for basic surgical skills acquisition.
        J Surg Educ. 2017; 74: e55-e61