Advertisement

Application of power analysis to determine the optimal reporting time frame for use in statewide trauma system quality reporting

      Abstract

      Background

      Meaningful reporting of quality metrics relies on detecting a statistical difference when a true difference in performance exists. Larger cohorts and longer time frames can produce higher rates of statistical differences. However, older data are less relevant when attempting to enact change in the clinical setting. The selection of time frames must reflect a balance between being too small (type II errors) and too long (stale data). We explored the use of power analysis to optimize time frame selection for trauma quality reporting.

      Methods

      Using data from 22 Level III trauma centers, we tested for differences in 4 outcomes within 4 cohorts of patients. With bootstrapping, we calculated the power for rejecting the null hypothesis that no difference exists amongst the centers for different time frames. From the entire sample for each site, we simulated randomly generated datasets. Each simulated dataset was tested for whether a difference was observed from the average. Power was calculated as the percentage of simulated datasets where a difference was observed. This process was repeated for each outcome.

      Results

      The power calculations for the 4 cohorts revealed that the optimal time frame for Level III trauma centers to assess whether a single site’s outcomes are different from the overall average was 2 years based on an 80% cutoff.

      Conclusion

      Power analysis with simulated datasets allows testing of different time frames to assess outcome differences. This type of analysis allows selection of an optimal time frame for benchmarking of Level III trauma center data.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Hashmi Z.G.
        • Schneider E.B.
        • Castillo R.
        • et al.
        Benchmarking trauma centers on mortality alone does not reflect quality of care: implications for pay-for-performance.
        J Trauma Acute Care Surg. 2014; 76: 1184-1191
        • Hashmi Z.G.
        • Dimick J.B.
        • Efron D.T.
        • et al.
        Reliability adjustment: a necessity for trauma center ranking and benchmarking.
        J Trauma Acute Care Surg. 2013; 75: 166-172
        • Austin P.C.
        • Ceyisakar I.E.
        • Steyerberg E.W.
        • Lingsma H.F.
        • Marang-van de Mheen P.J.
        Ranking hospital performance based on individual indicators: can we increase reliability by creating composite indicators?.
        BMC Med Res Methodol. 2019; 19: 131
        • Krell R.W.
        • Hozain A.
        • Kao L.S.
        • Dimick J.B.
        Reliability of risk-adjusted outcomes for profiling hospital surgical quality.
        JAMA Surg. 2014; 149: 467-474
        • Newgard C.D.
        • Fildes J.J.
        • Wu L.
        • et al.
        Methodology and analytic rationale for the American College of Surgeons Trauma Quality Improvement Program.
        J Am Coll Surg. 2013; 216: 147-157
        • Jakubus J.L.
        • Di Pasquo S.L.
        • Mikhail J.N.
        • Cain-Nielsen A.H.
        • Jenkins P.C.
        • Hemmila M.R.
        Pull back the curtain: external data validation is an essential element of quality improvement benchmark reporting.
        J Trauma Acute Care Surg. 2020; 89: 199-207
        • Machado-Aranda D.A.
        • Jakubus J.L.
        • Wahl W.L.
        • et al.
        Reduction in venous thromboembolism events: trauma performance improvement and loop closure through participation in a state-wide quality collaborative.
        J Am Coll Surg. 2015; 221: 661-668
        • Freiman J.A.
        • Chalmers T.C.
        • Smith Jr., H.
        • Kuebler R.R.
        The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 “negative” trials.
        N Engl J Med. 1978; 299: 690-694
        • Jaffe T.A.
        • Hasday S.J.
        • Dimick J.B.
        Power outage-inadequate surgeon performance measures leave patients in the dark.
        JAMA Surg. 2016; 151: 599-600
        • Dimick J.B.
        • Welch H.G.
        • Birkmeyer J.D.
        Surgical mortality as an indicator of hospital quality: the problem with small sample size.
        JAMA. 2004; 292: 847-851
        • Hemmila M.R.
        • Jakubus J.L.
        • Cain-Nielsen A.H.
        • et al.
        The Michigan Trauma Quality Improvement Program: results from a collaborative quality initiative.
        J Trauma Acute Care Surg. 2017; 82: 867-876
        • Hemmila M.R.
        • Cain-Nielsen A.H.
        • Wahl W.L.
        • et al.
        Regional collaborative quality improvement for trauma reduces complications and costs.
        J Trauma Acute Care Surg. 2015; 78 (discussion85–7): 78-85
        • Roberts J.K.
        • Fan X.
        Bootstrapping within the Multilevel/Hierarchical Linear Modeling Framework: A Primer for Use with SAS and SPLUS.
        Vol. 30. Mult Linear Regres Viewpoints, 2004: 23-34
        • Efron B.
        • Rogosa D.
        • Tibshirani R.
        Resampling Methods of Estimation.
        in: Wright J.D. International Encyclopedia of the Social & Behavioral Sciences. Elsevier, London2015: 492-495
        • American College of Surgeons (ACS)
        American College of Surgeons Trauma Quality Improvement Program (ACS TQIP).
        ACS TQIP Benchmark Report: Fall 2021. American College of Surgeons TQIP, 2021: 5
        • Hemmila M.R.
        • Jakubus J.L.
        Trauma quality improvement.
        Crit Care Clin. 2017; 33: 193-212
        • Hemmila M.R.
        • Cain-Nielsen A.H.
        • Jakubus J.L.
        • Mikhail J.N.
        • Dimick J.B.
        Association of hospital participation in a regional trauma quality improvement collaborative with patient outcomes.
        JAMA Surg. 2018; 153: 747-756
        • Ban K.A.
        • Cohen M.E.
        • Ko C.Y.
        • et al.
        Evaluation of the ProPublica Surgeon Scorecard “adjusted complication rate” measure specifications.
        Ann Surg. 2016; 264: 566-574
        • Friedberg M.W.
        • Pronovost P.J.
        • Shahian D.M.
        • et al.
        A methodological critique of the ProPublica Surgeon Scorecard.
        Rand Health Q. 2016; 5: 1
        • Auffenberg G.B.
        • Ghani K.R.
        • Ye Z.
        • et al.
        Comparing publicly reported surgical outcomes with quality measures from a statewide improvement collaborative.
        JAMA Surg. 2016; 151: 680-682
        • Jenkins P.C.
        • Timsina L.
        • Murphy P.
        • et al.
        Extending trauma quality improvement beyond trauma centers: hospital variation in outcomes among nontrauma hospitals.
        Ann Surg. 2022; 275: 406-413