A novel machine learning approach to identify social risk factors associated with textbook outcomes after surgery

  • J. Madison Hyer
    Department of Surgery, The Ohio State University Wexner Medical Center and James Comprehensive Cancer Center, Columbus, OH
    Search for articles by this author
  • Adrian Diaz
    Department of Surgery, The Ohio State University Wexner Medical Center and James Comprehensive Cancer Center, Columbus, OH

    National Clinician Scholars Program at the Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, MI

    Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
    Search for articles by this author
  • Diamantis Tsilimigras
    Department of Surgery, The Ohio State University Wexner Medical Center and James Comprehensive Cancer Center, Columbus, OH
    Search for articles by this author
  • Timothy M. Pawlik
    Reprint requests: Timothy M. Pawlik, MD, MPH, PhD, Professor and Chair, Department of Surgery, The Urban Meyer III and Shelley Meyer Chair for Cancer Research, The Ohio State University, Wexner Medical Center, 395 W. 12th Ave., Suite 670, Columbus, OH 43210.
    Department of Surgery, The Ohio State University Wexner Medical Center and James Comprehensive Cancer Center, Columbus, OH
    Search for articles by this author



      Identifying social determinants of health has become a priority for many researchers, health care providers, and payers. The vast amount of patient and population-level data available on social determinants creates, however, both an opportunity and a challenge as these data can be difficult to synthesize and analyze.


      Medicare beneficiaries who underwent 1 of 4 common operations between 2013 and 2017 were identified. Using a machine learning algorithm, the primary independent variable, surgery social determinants of health index, was derived from 15 common, publicly available social determents of health measures. After development of a surgery social determinants of health index, multivariable logistic regression was used to estimate the association of this index with textbook outcomes, as well as the component metrics of textbook outcomes.


      A novel surgery social determinants of health index was developed with factor component weights that varied relative to their impact on postoperative outcomes. Factors with the highest weight in the algorithm relative to postoperative outcomes were the proportion of noninstitutionalized civilians with a disability and persons without high school diploma, while components with the lowest weights were the proportion of households with more people than rooms and persons below poverty. Overall, an increase in surgery social determinants of health index was associated with 6% decreased odds (95% confidence interval: 0.93–0.94) of achieving a textbook outcome. In addition, an increase in surgery social determinants of health index was associated with increased odds of each of the individual components of textbook outcome; ranging from 3% increased odds (95% confidence interval: 1.03–1.04) for 90-day readmission to 10% increased odds (95% confidence interval: 1.09–1.11) for 90-day mortality. Further, there was 6% increased odds (95% confidence interval: 1.05–1.07) of experiencing a complication and 7% increased odds (95% confidence interval: 1.06–1.07) of having an extended length of stay. Minority patients from a high surgery social determinants of health index had 38% lower odds (95% confidence interval: 0.60–0.65) of achieving a textbook outcome compared with White/non-Hispanic patients from a low surgery social determinants of health index area.


      Using a machine learning approach, we developed a novel social determents of health index to predict the probability of achieving a textbook outcome after surgery.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Hood C.M.
        • Gennuso K.P.
        • Swain G.R.
        • et al.
        County health rankings: relationships between determinant factors and health outcomes.
        Am J Prev Med. 2016; 50: 129-135
        • McGinnis J.M.
        • Williams-Russo P.
        • Knickman J.R.
        The case for more active policy attention to health promotion.
        Health Aff (Millwood). 2002; 21: 78-93
        • Schroeder S.A.
        We can do better: improving the health of the American people.
        N Engl J Med. 2007; 357: 1221-1228
        • World Health Organization
        Social determinants of health.
      1. Wong JH, Irish WD, DeMaria EJ, et al. Development and assessment of a systematic approach for detecting disparities in surgical access. JAMA Surg. Epub ahead of print December 16, 2020.

        • Pitt S.C.
        • Merkow R.P.
        Disparities research: mitigating inequities in surgical care.
        JAMA Surg. 2020; 155: 1012-1014
        • Levine A.A.
        • de Jager E.
        • Britt L.D.
        Perspective: identifying and addressing disparities in surgical access: a health systems call to action.
        Ann Surg. 2020; 271: 427-430
        • Davidson K.W.
        • McGinn T.
        Screening for social determinants of health: the known and unknown.
        JAMA. 2019; 322: 1037-1038
        • CDC
        Social Vulnerability Index..
        Date accessed: March 14, 2020
        • Azap R.A.
        • Paredes A.Z.
        • Diaz A.
        • et al.
        The association of neighborhood social vulnerability with surgical textbook outcomes among patients undergoing hepatopancreatic surgery.
        Surgery. 2020; 168: 868-875
        • Diaz A.
        • Barmash E.
        • Azap R.
        • et al.
        Association of county-level social vulnerability with elective versus non-elective colorectal surgery.
        J Gastrointest Surg. 2020; : 1-9
      2. Diaz A, Hyer JM, Barmash E, et al. County-level social vulnerability is associated with worse surgical outcomes especially among minority patients. Ann Surg. Epub ahead of print July 16, 2021.

        • Hyer J.M.
        • Tsilimigras D.I.
        • Diaz A.
        • et al.
        High social vulnerability and “textbook outcomes” after cancer surgery.
        J Am Coll Surg. 2021; 232: 351-359
        • Deo R.C.
        Machine learning in medicine.
        Circulation. 2015; 132: 1920-1930
      3. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. New York, NY: Springer. Epub ahead of print 2009.

        • Corey K.M.
        • Kashyap S.
        • Lorenzi E.
        • et al.
        Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): a retrospective, single-site study.
        PLOS Med. 2018; 15e1002701
        • Bertsimas D.
        • Kung J.
        • Trichakis N.
        • et al.
        Development and validation of an optimized prediction of mortality for candidates awaiting liver transplantation.
        Am J Transplant. 2019; 19: 1109-1118
        • Bertsimas D.
        • Dunn J.
        • Velmahos G.C.
        • et al.
        Surgical risk is not linear: derivation and validation of a novel, user-friendly, and machine-learning-based Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) calculator.
        Ann Surg. 2018; 268: 574-583
        • Hyer J.M.
        • Ejaz A.
        • Tsilimigras D.I.
        • et al.
        Novel machine learning approach to identify preoperative risk factors associated with super-utilization of Medicare expenditure following surgery.
        JAMA Surg. 2019; 154: 1014-1021
        • Hyer J.M.
        • White S.
        • Cloyd J.
        • et al.
        Can we improve prediction of adverse surgical outcomes? Development of a surgical complexity score using a novel machine learning technique.
        J Am Coll Surg. 2020; 230: 43-52.e1
        • Merath K.
        • Chen Q.
        • Bagante F.
        • et al.
        Textbook outcomes among Medicare patients undergoing hepatopancreatic surgery.
        Ann Surg. 2020; 271: 1116-1123
        • Merath K.
        • Chen Q.
        • Bagante F.
        • et al.
        A multi-institutional international analysis of textbook outcomes among patients undergoing curative-intent resection of intrahepatic cholangiocarcinoma.
        JAMA Surg. 2019; 154e190571
        • Lawthers A.G.
        • McCarthy E.P.
        • Davis R.B.
        • et al.
        Identification of in-hospital complications from claims data: is it valid?.
        Med Care. 2000; 38: 785-795
        • Weingart S.N.
        • Iezzoni L.I.
        • Davis R.B.
        • et al.
        Use of administrative data to find substandard care: validation of the complications screening program.
        Med Care. 2000; 38: 796-806
        • Muñoz-Price L.S.
        • Nattinger A.B.
        • Rivera F.
        • et al.
        Racial disparities in incidence and outcomes among patients with COVID-19.
        JAMA Netw Open. 2020; 3e2021892
        • National Institute on Minority Heaoth and Health Disparites
        Surgical disparities research. NIMHD.
        • American College of Surgeons
        Committee on Health Care Disparities.
        Date accessed: January 26, 2021
        • University of Wisconsin School of Medicine and Public Health
        Area Deprivation Index v.2.0.
        Date accessed: September 28, 2020
        • Copeland G.P.
        • Jones D.
        • Walters M.
        POSSUM: a scoring system for surgical audit.
        BJS Br J Surg. 1991; 78: 355-360
        • Murtaugh P.A.
        • Dickson E.R.
        • Dam G.M.V.
        • et al.
        Primary biliary cirrhosis: prediction of short-term survival based on repeated patient visits.
        Hepatology. 1994; 20: 126-134
        • Haskins I.N.
        • Maluso P.J.
        • Schroeder M.E.
        • et al.
        A calculator for mortality following emergency general surgery based on the American College of Surgeons National Surgical Quality Improvement Program database.
        J Trauma Acute Care Surg. 2017; 82: 1094-1099
        • Protopapa K.L.
        • Simpson J.C.
        • Smith N.C.E.
        • et al.
        Development and validation of the Surgical Outcome Risk Tool (SORT).
        Br J Surg. 2014; 101: 1774-1783
        • Bilimoria K.Y.
        • Liu Y.
        • Paruch J.L.
        • et al.
        Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons.
        J Am Coll Surg. 2013; 217: 833-842.e3
        • Mitka M.
        Data-based risk calculators becoming more sophisticated—and more popular.
        JAMA. 2009; 302: 730-731
      4. ACS Risk Calculator.
        • Fischer J.P.
        • Wink J.D.
        • Tuggle C.T.
        • et al.
        Wound risk assessment in ventral hernia repair: generation and internal validation of a risk stratification system using the ACS-NSQIP.
        Hernia. 2015; 19: 103-111
        • Osborne N.H.
        • Nicholas L.H.
        • Ryan A.M.
        • et al.
        Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries.
        JAMA. 2015; 313: 496-504
        • Cohen M.E.
        • Liu Y.
        • Ko C.Y.
        • et al.
        An examination of American College of Surgeons NSQIP surgical risk calculator accuracy.
        J Am Coll Surg. 2017; 224: 787-795.e1
        • Armstrong E.A.
        • Beal E.W.
        • Lopez-Aguiar A.G.
        • et al.
        Evaluating the ACS-NSQIP risk calculator in primary GI neuroendocrine tumor: results from the United States Neuroendocrine Tumor Study Group.
        Am Surg. 2019; 85: 1334-1340
        • Azap R.A.
        • Hyer J.M.
        • Diaz A.
        • et al.
        Association of county-level vulnerability, patient-level race/ethnicity, and receipt of surgery for early-stage hepatocellular carcinoma.
        JAMA Surg. 2021; 156: 197-199
        • Walker R.J.
        • Strom Williams J.
        • Egede L.E.
        Influence of race, ethnicity and social determinants of health on diabetes outcomes.
        Am J Med Sci. 2016; 351: 366-373
        • Morenoff J.D.
        • House J.S.
        • Hansen B.B.
        • et al.
        Understanding social disparities in hypertension prevalence, awareness, treatment, and control: the role of neighborhood context.
        Soc Sci Med. 2007; 65: 1853-1866
        • Herrera-Escobar J.P.
        • Seshadri A.J.
        • Rivero R.
        • et al.
        Lower education and income predict worse long-term outcomes after injury.
        J Trauma Acute Care Surg. 2019; 87: 104-110
        • Chetty R.
        • Stepner M.
        • Abraham S.
        • et al.
        The association between income and life expectancy in the United States, 2001–2014.
        JAMA. 2016; 315: 1750
        • Braveman P.A.
        • Cubbin C.
        • Egerter S.
        • et al.
        Socioeconomic disparities in health in the United States: what the patterns tell us.
        Am J Public Health. 2010; 100: S186-S196
        • Mudrazija S.
        • López-Ortega M.
        • Vega W.A.
        • et al.
        Household composition and longitudinal health outcomes for older Mexican return migrants.
        Res Aging. 2016; 38: 346-373
        • Turagabeci A.R.
        • Nakamura K.
        • Kizuki M.
        • et al.
        Family structure and health, how companionship acts as a buffer against ill health.
        Health Qual Life Outcomes. 2007; 5: 61
        • ATSDR
        CDC SVI Documentation 2018: place and health.
        • Vart P.
        • Powe N.R.
        • McCulloch C.E.
        • et al.
        National trends in the prevalence of chronic kidney disease among racial/ethnic and socioeconomic status groups, 1988–2016.
        JAMA Netw Open. 2020; 3e207932
        • Kurani S.S.
        • McCoy R.G.
        • Lampman M.A.
        • et al.
        Association of neighborhood measures of social determinants of health with breast, cervical, and colorectal cancer screening rates in the US Midwest.
        JAMA Netw Open. 2020; 3e200618
        • Mullings L.
        • Schulz A.J.
        Intersectionality and health: an introduction.
        in: Gender, Race, Class, and Health: Intersectional Approaches. Jossey-Bass/Wiley, Hoboken, NJ2006: 3-17
        • Caiola C.
        • Docherty S.
        • Relf M.
        • et al.
        Using an intersectional approach to study the impact of social determinants of health for African-American mothers living with HIV.
        ANS Adv Nurs Sci. 2014; 37: 287-298
        • Iezzoni L.I.
        • Daley J.
        • Heeren T.
        • et al.
        Identifying complications of care using administrative data.
        Med Care. 1994; 32: 700-715
        • Iezzoni L.I.
        Assessing quality using administrative data.
        Ann Intern Med. 1997; 127: 666-674