Advertisement

Development and validation of models for detection of postoperative infections using structured electronic health records data and machine learning

  • Kathryn L. Colborn
    Correspondence
    Reprint requests: Kathryn Colborn, University of Colorado Anschutz Medical Campus, 12631 E. 17th Ave, C-305, Aurora, CO 80045.
    Affiliations
    Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO

    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO

    Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO

    Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • Yaxu Zhuang
    Affiliations
    Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
    Search for articles by this author
  • Adam R. Dyas
    Affiliations
    Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO

    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • William G. Henderson
    Affiliations
    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO

    Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
    Search for articles by this author
  • Helen J. Madsen
    Affiliations
    Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO

    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • Michael R. Bronsert
    Affiliations
    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO

    Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • Michael E. Matheny
    Affiliations
    Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN

    Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN

    Division of General Internal Medicine, Vanderbilt University Medical Center, Nashville, TN
    Search for articles by this author
  • Anne Lambert-Kerzner
    Affiliations
    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • Quintin W.O. Myers
    Affiliations
    Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO

    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
  • Robert A. Meguid
    Affiliations
    Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO

    Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO

    Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO
    Search for articles by this author
Published:December 02, 2022DOI:https://doi.org/10.1016/j.surg.2022.10.026

      Abstract

      Background

      Postoperative infections constitute more than half of all postoperative complications. Surveillance of these complications is primarily done through manual chart review, which is time consuming, expensive, and typically only covers 10% to 15% of all operations. Automated surveillance would permit the timely evaluation of and reporting of all operations.

      Methods

      The goal of this study was to develop and validate parsimonious, interpretable models for conducting surveillance of postoperative infections using structured electronic health records data. This was a retrospective study using 30,639 unique operations from 5 major hospitals between 2013 and 2019. Structured electronic health records data were linked to postoperative outcomes data from the American College of Surgeons National Surgical Quality Improvement Program. Predictors from the electronic health records included diagnoses, procedures, and medications. Infectious complications included surgical site infection, urinary tract infection, sepsis, and pneumonia within 30 days of surgery. The knockoff filter, a penalized regression technique that controls type I error, was applied for variable selection. Models were validated in a chronological held-out dataset.

      Results

      Seven percent of patients experienced at least one type of postoperative infection. Models selected contained between 4 and 8 variables and achieved >0.91 area under the receiver operating characteristic curve, >81% specificity, >87% sensitivity, >99% negative predictive value, and 10% to 15% positive predictive value in a held-out test dataset.

      Conclusion

      Surveillance and reporting of postoperative infection rates can be implemented for all operations with high accuracy using electronic health records data and simple linear regression models.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Al-Tawfiq J.A.
        • Tambyah P.A.
        Healthcare associated infections (HAI) perspectives.
        J Infect Public Health. 2014; 7: 339-344
        • de Lissovoy G.
        • Fraeman K.
        • Hutchins V.
        • Murphy D.
        • Song D.
        • Vaughn B.B.
        Surgical site infection: incidence and impact on hospital utilization and treatment costs.
        Am J Infect Control. 2009; 37: 387-397
        • Zimlichman E.
        • Henderson D.
        • Tamir O.
        • et al.
        Health care-associated infections: a meta-analysis of costs and financial impact on the US health care system.
        JAMA Intern Med. 2013; 173: 2039-2046
        • Stone P.W.
        • Dick A.
        • Pogorzelska M.
        • Horan T.C.
        • Furuya E.Y.
        • Larson E.
        Staffing and structure of infection prevention and control programs.
        Am J Infect Control. 2009; 37: 351-357
        • ACS NSQIP
        American College of Surgeons.
        https://www.facs.org/quality-programs/acs-nsqip/about
        Date: 2019
        Date accessed: February 24, 2019
        • Aasen D.M.
        • Bronsert M.R.
        • Rozeboom P.D.
        • et al.
        Relationships between predischarge and postdischarge infectious complications, length of stay, and unplanned readmissions in the.
        ACS NSQIP database. Surgery. 2021; 169: 325-332
        • Hammermeister K.E.
        • Henderson W.G.
        • Bronsert M.R.
        • Juarez-Colunga E.
        • Meguid R.A.
        Bringing quantitative risk assessment closer to the patient and surgeon: a novel approach to improve outcomes.
        Ann Surg. 2016; 263: 1039-1041
        • Centers for Disease Control and Prevention
        National Healthcare Safety Network.
        https://www.cdc.gov/nhsn/about-nhsn/index.html
        Date: 2021
        Date accessed: May 24, 2021
        • Branch-Elliman W.
        • Strymish J.
        • Itani K.M.
        • Gupta K.
        Using clinical variables to guide surgical site infection detection: a novel surveillance strategy.
        Am J Infect Control. 2014; 42: 1291-1295
        • Colborn K.L.
        • Bronsert M.
        • Amioka E.
        • Hammermeister K.
        • Henderson W.G.
        • Meguid R.
        Identification of surgical site infections using electronic health record data.
        Am J Infect Control. 2018; 46: 1230-1235
        • Goto M.
        • Ohl M.E.
        • Schweizer M.L.
        • Perencevich E.N.
        Accuracy of administrative code data for the surveillance of healthcare-associated infections: a systematic review and meta-analysis.
        Clin Infect Dis. 2014; 58: 688-696
        • Hu Z.
        • Simon G.J.
        • Arsoniadis E.G.
        • Wang Y.
        • Kwaan M.R.
        • Melton G.B.
        Automated detection of postoperative surgical site infections using supervised methods with electronic health record data.
        Stud Health Technol Inform. 2015; 216: 706-710
        • Ju M.H.
        • Ko C.Y.
        • Hall B.L.
        • Bosk C.L.
        • Bilimoria K.Y.
        • Wick E.C.
        A comparison of 2 surgical site infection monitoring systems.
        JAMA Surg. 2015; 150: 51-57
        • Branch-Elliman W.
        • Strymish J.
        • Kudesia V.
        • Rosen A.K.
        • Gupta K.
        Natural language processing for real-time catheter-associated urinary tract infection surveillance: results of a pilot implementation trial.
        Infect Control Hosp Epidemiol. 2015; 36: 1004-1010
        • Choudhuri J.A.
        • Pergamit R.F.
        • Chan J.D.
        • et al.
        An electronic catheter-associated urinary tract infection surveillance tool.
        Infect Control Hosp Epidemiol. 2011; 32: 757-762
        • Colborn K.L.
        • Bronsert M.
        • Hammermeister K.
        • Henderson W.G.
        • Singh A.B.
        • Meguid R.A.
        Identification of urinary tract infections using electronic health record data.
        Am J Infect Control. 2018; 47: 371-375
        • Gundlapalli A.V.
        • Divita G.
        • Redd A.
        • et al.
        Detecting the presence of an indwelling urinary catheter and urinary symptoms in hospitalized patients using natural language processing.
        J Biomed Inform. 2017; 71s: S39-S45
        • Hsu H.E.
        • Shenoy E.S.
        • Kelbaugh D.
        • et al.
        An electronic surveillance tool for catheter-associated urinary tract infection in intensive care units.
        Am J Infect Control. 2015; 43: 592-599
        • Landers T.
        • Apte M.
        • Hyman S.
        • Furuya Y.
        • Glied S.
        • Larson E.
        A comparison of methods to detect urinary tract infections using electronic data.
        Jt Comm J Qual Patient Saf. 2010; 36: 411-417
        • Sanger P.C.
        • Granich M.
        • Olsen-Scribner R.
        • et al.
        Electronic surveillance for catheter-associated urinary tract infection using natural language processing.
        AMIA Annu Symp Proc. 2018; 2017: 1507-1516
        • Shepard J.
        • Hadhazy E.
        • Frederick J.
        • et al.
        Using electronic medical records to increase the efficiency of catheter-associated urinary tract infection surveillance for National Health and Safety Network reporting.
        Am J Infect Control. 2014; 42: e33-e36
        • Sopirala M.M.
        • Syed A.
        • Jandarov R.
        • Lewis M.
        Impact of a change in surveillance definition on performance assessment of a catheter-associated urinary tract infection prevention program at a tertiary care medical center.
        Am J Infect Control. 2018; 46: 743-746
        • Wald H.L.
        • Bandle B.
        • Richard A.
        • Min S.
        Accuracy of electronic surveillance of catheter-associated urinary tract infection at an academic medical center.
        Infect Control Hosp Epidemiol. 2014; 35: 685-691
        • Zhan C.
        • Elixhauser A.
        • Richards Jr., C.L.
        • et al.
        Identification of hospital-acquired catheter-associated urinary tract infections from Medicare claims: sensitivity and positive predictive value.
        Med Care. 2009; 47: 364-369
        • Thottakkara P.
        • Ozrazgat-Baslanti T.
        • Hupf B.B.
        • et al.
        Application of machine learning techniques to high-dimensional clinical data to forecast postoperative complications.
        PLoS One. 2016; 11e0155705
        • Hu Z.
        • Melton G.B.
        • Moeller N.D.
        • et al.
        Accelerating chart review using automated methods on electronic health record data for postoperative complications.
        AMIA Annu Symp Proc. 2017; 2016: 1822-1831
        • Leth R.A.
        • Moller J.K.
        Surveillance of hospital-acquired infections based on electronic hospital registries.
        J Hosp Infect. 2006; 62: 71-79
        • FitzHenry F.
        • Murff H.J.
        • Matheny M.E.
        • et al.
        Exploring the frontier of electronic health record surveillance: the case of postoperative complications.
        Med Care. 2013; 51: 509-516
        • Murff H.J.
        • FitzHenry F.
        • Matheny M.E.
        • et al.
        Automated identification of postoperative complications within an electronic medical record using natural language processing.
        JAMA. 2011; 306: 848-855
        • Selby L.V.
        • Narain W.R.
        • Russo A.
        • Strong V.E.
        • Stetson P.
        Autonomous detection, grading, and reporting of postoperative complications using natural language processing.
        Surgery. 2018; 164: 1300-1305
        • Knepper B.C.
        • Young H.
        • Jenkins T.C.
        • Price C.S.
        Time-saving impact of an algorithm to identify potential surgical site infections.
        Infect Control Hosp Epidemiol. 2013; 34: 1094-1098
        • Wei W.Q.
        • Bastarache L.A.
        • Carroll R.J.
        • et al.
        Evaluating phecodes, clinical classification software, and ICD-9-CM codes for phenome-wide association studies in the electronic health record.
        PLoS One. 2017; 12e0175508
        • Wu P.
        • Gifford A.
        • Meng X.
        • et al.
        Mapping ICD-10 and ICD-10-CM Codes to Phecodes: workflow development and initial evaluation.
        JMIR Med Inform. 2019; 7e14325
        • Barber R.F.
        • Candes E.J.
        Controlling the false discovery rate via knockoffs.
        Ann Stat. 2015; 43: 2055-2085
        • Candes E.
        • Fan Y.
        • Janson L.
        • Lv J.
        Panning for gold: ‘model-X’ knockoffs for high dimensional controlled variable selection.
        J R Stat Soc Series B Stat Methodol. 2018; 80: 551-577
        • Bronsert M.
        • Singh A.B.
        • Henderson W.G.
        • Hammermeister K.
        • Meguid R.A.
        • Colborn K.L.
        Identification of postoperative complications using electronic health record data and machine learning.
        Am J Surg. 2020; 220: 114-119
        • Youden W.J.
        Index for rating diagnostic tests.
        Cancer. 1950; 3: 32-35
        • Patterson E.
        • Sesia M.
        knockoff: The Knockoff Filter for Controlled Variable Selection. R package version 0.3.6.
        https://CRAN.R-project.org/package=knockoff
        Date: 2022
        Date accessed: January 1, 2022
        • Friedman J.
        • Hastie T.
        • Tibshirani R.
        Regularization paths for generalized linear models via coordinate descent.
        J Stat Softw. 2010; 33: 1-22
        • Robin X.
        • Turck N.
        • Hainard A.
        • et al.
        pROC: an open-source package for R and S+ to analyze and compare ROC curves.
        BMC Bioinformatics. 2011; 12: 77
        • Colborn K.L.
        • Zhuang Y.
        Automated surveillance of postoperative infections (ASPIN).
        https://github.com/katiecolborn/ASPIN
        Date: 2022
        Date accessed: January 10, 2022
        • Moons K.G.
        • Altman D.G.
        • Reitsma J.B.
        • et al.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration.
        Ann Inter Med. 2015; 162: W1-W73
        • Ozkan I.A.
        • Koklu M.
        • Sert I.U.
        Diagnosis of urinary tract infection based on artificial intelligence methods.
        Comput Methods Programs Biomed. 2018; 166: 51-59
        • Zhu Y.
        • Simon G.J.
        • Wick E.C.
        • et al.
        Applying machine learning across sites: external validation of a surgical site infection detection algorithm.
        J Am Coll Surg. 2021; 232: 963-971.e961
        • Davis S.E.
        • Greevy R.A.
        • Lasko T.A.
        • Walsh C.G.
        • Matheny M.E.
        Comparison of prediction model performance updating protocols: using a data-driven testing procedure to guide updating.
        AMIA Annu Symp Proc. 2019; 2019: 1002-1010
        • Davis S.E.
        • Lasko T.A.
        • Chen G.
        • Matheny M.E.
        Calibration drift among regression and machine learning models for hospital mortality.
        AMIA Annu Symp Proc. 2017; 2017: 625-634