Advertisement
Healthcare| Volume 168, ISSUE 2, P253-266, August 2020

Decision analysis and reinforcement learning in surgical decision-making

      Abstract

      Background

      Surgical patients incur preventable harm from cognitive and judgment errors made under time constraints and uncertainty regarding patients’ diagnoses and predicted response to treatment. Decision analysis and techniques of reinforcement learning theoretically can mitigate these challenges but are poorly understood and rarely used clinically. This review seeks to promote an understanding of decision analysis and reinforcement learning by describing their use in the context of surgical decision-making.

      Methods

      Cochrane, EMBASE, and PubMed databases were searched from their inception to June 2019. Included were 41 articles about cognitive and diagnostic errors, decision-making, decision analysis, and machine-learning. The articles were assimilated into relevant categories according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

      Results

      Requirements for time-consuming manual data entry and crude representations of individual patients and clinical context compromise many traditional decision-support tools. Decision analysis methods for calculating probability thresholds can inform population-based recommendations that jointly consider risks, benefits, costs, and patient values but lack precision for individual patient-centered decisions. Reinforcement learning, a machine-learning method that mimics human learning, can use a large set of patient-specific input data to identify actions yielding the greatest probability of achieving a goal. This methodology follows a sequence of events with uncertain conditions, offering potential advantages for personalized, patient-centered decision-making. Clinical application would require secure integration of multiple data sources and attention to ethical considerations regarding liability for errors and individual patient preferences.

      Conclusion

      Traditional decision-support tools are ill-equipped to accommodate time constraints and uncertainty regarding diagnoses and the predicted response to treatment, both of which often impair surgical decision-making. Decision analysis and reinforcement learning have the potential to play complementary roles in delivering high-value surgical care through sound judgment and optimal decision-making.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Hadley J.
        • Steinberg E.P.
        • Feder J.
        Comparison of uninsured and privately insured hospital patients. Condition on admission, resource use, and outcome.
        JAMA. 1991; 265: 374-379
        • Goldenson R.M.
        The encyclopedia of human behavior; psychology, psychiatry, and mental health.
        Doubleday, Garden City, NY1970
        • Dijksterhuis A.
        • Bos M.W.
        • Nordgren L.F.
        • van Baaren R.B.
        On making the right choice: the deliberation-without-attention effect.
        Science. 2006; 311: 1005-1007
        • Bekker H.L.
        Making choices without deliberating.
        Science. 2006; 312 (author reply 1472): 1472
        • Wolf F.M.
        • Gruppen L.D.
        • Billi J.E.
        Differential diagnosis and the competing-hypotheses heuristic. A practical approach to judgment under uncertainty and Bayesian probability.
        JAMA. 1985; 253: 2858-2862
        • Graber M.L.
        • Franklin N.
        • Gordon R.
        Diagnostic error in internal medicine.
        Arch Intern Med. 2005; 165: 1493-1499
        • Kirch W.
        • Schafii C.
        Misdiagnosis at a university hospital in 4 medical eras.
        Medicine (Baltimore). 1996; 75: 29-40
        • Sonderegger-Iseli K.
        • Burger S.
        • Muntwyler J.
        • Salomon F.
        Diagnostic errors in three medical eras: a necropsy study.
        Lancet. 2000; 355: 2027-2031
        • Healey M.A.
        • Shackford S.R.
        • Osler T.M.
        • Rogers F.B.
        • Burns E.
        Complications in surgical patients.
        Arch Surg. 2002; 137 (discussion 617-618): 611-617
        • Shanafelt T.D.
        • Balch C.M.
        • Bechamps G.
        • et al.
        Burnout and medical errors among American surgeons.
        Ann Surg. 2010; 251: 995-1000
        • Leeds I.L.
        • Rosenblum A.J.
        • Wise P.E.
        • et al.
        Eye of the beholder: risk calculators and barriers to adoption in surgical trainees.
        Surgery. 2018; 164: 1117-1123
        • Brotman D.J.
        • Walker E.
        • Lauer M.S.
        • O'Brien R.G.
        In search of fewer independent risk factors.
        Arch Intern Med. 2005; 165: 138-145
        • Bertsimas D.
        • Dunn J.
        • Velmahos G.C.
        • Kaafarani H.M.A.
        Surgical risk is not linear: Derivation and validation of a novel, user-friendly, and machine-learning-based Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) calculator.
        Ann Surg. 2018; 268: 574-583
        • Bergquist S.L.
        • Brooks G.A.
        • Keating N.L.
        • Landrum M.B.
        • Rose S.
        Classifying lung cancer severity with ensemble machine learning in health care claims data.
        Proc Mach Learn Res. 2017; 68: 25-38
        • Groopman J.E.
        How doctors think.
        Houghton Mifflin, Boston2007
        • Kopecky K.E.
        • Urbach D.
        • Schwarze M.L.
        Risk calculators and decision aids are not enough for shared decision making.
        JAMA Surg. 2019; 154: 3-4
        • Weinstein J.N.
        • Clay K.
        • Morgan T.S.
        Informed patient choice: patient-centered valuing of surgical risks and benefits.
        Health Aff (Millwood). 2007; 26: 726-730
        • Legare F.
        • Ratte S.
        • Gravel K.
        • Graham I.D.
        Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals’ perceptions.
        Patient Educ Couns. 2008; 73: 526-535
        • Bertrand P.M.
        • Pereira B.
        • Adda M.
        • et al.
        Disagreement between clinicians and score in decision-making capacity of critically ill patients.
        Crit Care Med. 2019; 47: 337-344
        • de Mik S.M.L.
        • Stubenrouch F.E.
        • Balm R.
        • Ubbink D.T.
        Systematic review of shared decision-making in surgery.
        Br J Surg. 2018; 105: 1721-1730
        • Wilson A.
        • Ronnekleiv-Kelly S.M.
        • Pawlik T.M.
        Regret in surgical decision making: a systematic review of patient and physician perspectives.
        World J Surg. 2017; 41: 1454-1465
        • Acuna S.A.
        • Chesney T.R.
        • Baxter N.N.
        Incorporating patient preferences in noninferiority trials.
        JAMA. 2019; https://doi.org/10.1001/jama.2019.7059
        • Giuliano A.E.
        • Hunt K.K.
        • Ballman K.V.
        • et al.
        Axillary dissection vs no axillary dissection in women with invasive breast cancer and sentinel node metastasis: a randomized clinical trial.
        JAMA. 2011; 305: 569-575
        • Gage B.F.
        • Waterman A.D.
        • Shannon W.
        • Boechler M.
        • Rich M.W.
        • Radford M.J.
        Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation.
        JAMA. 2001; 285: 2864-2870
        • McGee T.M.
        Acute appendicitis in pregnancy.
        Aust N Z J Obstet Gynaecol. 1989; 29: 378-385
        • Kahneman D.
        • Slovic P.
        • Tversky A.
        Judgment under uncertainty: heuristics and biases.
        Cambridge University Press, Cambridge1982
        • Grootendorst P.
        • Feeny D.
        • Furlong W.
        Health Utilities Index Mark 3: Evidence of construct validity for stroke and arthritis in a population health survey.
        Med Care. 2000; 38: 290-299
        • Glasziou P.P.
        • Bromwich S.
        • Simes R.J.
        Quality of life six months after myocardial infarction treated with thrombolytic therapy. AUS-TASK Group. Australian arm of International tPA/SK Mortality Trial.
        Med J Aust. 1994; 161: 532-536
        • Solomon N.A.
        • Glick H.A.
        • Russo C.J.
        • Lee J.
        • Schulman K.A.
        Patient preferences for stroke outcomes.
        Stroke. 1994; 25: 1721-1725
        • Man-Son-Hing M.
        • Laupacis A.
        • O’Connor A.
        • et al.
        Warfarin for atrial fibrillation. The patient’s perspective.
        Arch Intern Med. 1996; 156: 1841-1848
        • Guyatt G.H.
        • Sinclair J.
        • Cook D.J.
        • Glasziou P.
        Users’ guides to the medical literature: XVI. How to use a treatment recommendation. Evidence-Based Medicine Working Group and the Cochrane Applicability Methods Working Group.
        JAMA. 1999; 281: 1836-1843
      1. Risk factors for stroke and efficacy of antithrombotic therapy in atrial fibrillation. Analysis of pooled data from five randomized controlled trials.
        Arch Intern Med. 1994; 154: 1449-1457
        • Pauker S.G.
        • Kassirer J.P.
        The threshold approach to clinical decision making.
        N Engl J Med. 1980; 302: 1109-1117
        • Carr J.A.
        • Walls J.
        • Bryan L.J.
        • Snider D.L.
        The treatment of gallbladder dyskinesia based upon symptoms: results of a 2-year, prospective, nonrandomized, concurrent cohort study.
        Surg Laparosc Endosc Percutan Tech. 2009; 19: 222-226
        • Wybourn C.A.
        • Kitsis R.M.
        • Baker T.A.
        • Degner B.
        • Sarker S.
        • Luchette F.A.
        Laparoscopic cholecystectomy for biliary dyskinesia: which patients have long term benefit?.
        Surgery. 2013; 154 (discussion 767-768): 761-767
        • Djulbegovic B.
        • Hozo I.
        • Lyman G.H.
        Linking evidence-based medicine therapeutic summary measures to clinical decision analysis.
        MedGenMed. 2000; 2: E6
        • Vickers A.J.
        • Elkin E.B.
        Decision curve analysis: a novel method for evaluating prediction models.
        Med Decis Making. 2006; 26: 565-574
        • Lee V.S.
        • Kawamoto K.
        • Hess R.
        • et al.
        Implementation of a value-driven outcomes program to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality.
        JAMA. 2016; 316: 1061-1072
        • O’Brien B.J.
        • Heyland D.
        • Richardson W.S.
        • Levine M.
        • Drummond M.F.
        Users’ guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group.
        JAMA. 1997; 277: 1802-1806
        • Gage B.F.
        • Cardinalli A.B.
        • Albers G.W.
        • Owens D.K.
        Cost-effectiveness of warfarin and aspirin for prophylaxis of stroke in patients with nonvalvular atrial fibrillation.
        JAMA. 1995; 274: 1839-1845
        • Robbins J.M.
        • Tilford J.M.
        • Jacobs R.F.
        • Wheeler J.G.
        • Gillaspy S.R.
        • Schutze G.E.
        A number-needed-to-treat analysis of the use of respiratory syncytial virus immune globulin to prevent hospitalization.
        Arch Pediatr Adolesc Med. 1998; 152: 358-366
        • Sutton R.S.
        • Barto A.G.
        Reinforcement learning: an introduction.
        2nd ed. The MIT Press, Cambridge, MA2018
        • Chaiyachati K.H.
        • Shea J.A.
        • Asch D.A.
        • et al.
        Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations.
        JAMA Intern Med. 2019; 179: 760-767
        • Guarisco S.
        • Oddone E.
        • Simel D.
        Time analysis of a general medicine service: results from a random work sampling study.
        J Gen Intern Med. 1994; 9: 272-277
        • Komorowski M.
        • Celi L.A.
        • Badawi O.
        • Gordon A.C.
        • Faisal A.A.
        The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care.
        Nat Med. 2018; 24: 1716-1720
        • Silver D.
        • Huang A.
        • Maddison C.J.
        • et al.
        Mastering the game of Go with deep neural networks and tree search.
        Nature. 2016; 529: 484-489
        • Mnih V.
        • Kavukcuoglu K.
        • Silver D.
        • et al.
        Human-level control through deep reinforcement learning.
        Nature. 2015; 518: 529-533
        • Hinton G.E.
        • Salakhutdinov R.R.
        Reducing the dimensionality of data with neural networks.
        Science. 2006; 313: 504-507
        • Shickel B.
        • Loftus T.J.
        • Adhikari L.
        • Ozrazgat-Baslanti T.
        • Bihorac A.
        • Rashidi P.
        DeepSOFA: a continuous acuity score for critically ill patients using clinically interpretable deep learning.
        Sci Rep. 2019; 9: 1879
        • Sundaram L.
        • Gao H.
        • Padigepati S.R.
        • et al.
        Predicting the clinical impact of human mutation with deep neural networks.
        Nat Genet. 2018; 50: 1161-1170
        • Li Y.
        • Kang H.
        • Ye K.
        • Yin S.
        • Li X.
        FoldingZero: Protein folding from scratch in hydrophobic-polar model.
        https://arxiv.org/abs/1812.00967
        Date accessed: September 15, 2019
        • Rajpurkar P.
        • Irvin J.
        • Ball R.L.
        • et al.
        Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists.
        PLoS Med. 2018; 15e1002686
        • Davoudi A.
        • Malhotra K.R.
        • Shickel B.
        • et al.
        Intelligent ICU for autonomous patient monitoring using pervasive sensing and deep learning.
        Sci Rep. 2019; 9: 8020
        • Hashimoto D.A.
        • Rosman G.
        • Witkowski E.R.
        • et al.
        Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy.
        Ann Surg. 2019; 270: 414-421
        • Topol E.J.
        A decade of digital medicine innovation.
        Sci Transl Med. 2019; 11https://doi.org/10.1126/scitranslmed.aaw7610
        • Silver D.
        • Schrittwieser J.
        • Simonyan K.
        • et al.
        Mastering the game of Go without human knowledge.
        Nature. 2017; 550: 354-359
        • Rivers E.
        • Nguyen B.
        • Havstad S.
        • Ressler J.
        • Muzzin A.
        • Knoblich B.
        • et al.
        Early goal-directed therapy in the treatment of severe sepsis and septic shock.
        N Engl J Med. 2001; 345: 1368-1377
        • Jones A.E.
        • Shapiro N.I.
        • Trzeciak S.
        • et al.
        Lactate clearance vs central venous oxygen saturation as goals of early sepsis therapy: a randomized clinical trial.
        JAMA. 2010; 303: 739-746
        • Marik P.
        • Bellomo R.
        A rational approach to fluid therapy in sepsis.
        Br J Anaesth. 2016; 116: 339-349
        • Lammi M.R.
        • Aiello B.
        • Burg G.T.
        • et al.
        Response to fluid boluses in the fluid and catheter treatment trial.
        Chest. 2015; 148: 919-926
        • Biais M.
        • Ehrmann S.
        • Mari A.
        • et al.
        Clinical relevance of pulse pressure variations for predicting fluid responsiveness in mechanically ventilated intensive care unit patients: the grey zone approach.
        Crit Care. 2014; 18: 587
        • Chowdhury A.H.
        • Cox E.F.
        • Francis S.T.
        • Lobo D.N.
        A randomized, controlled, double-blind crossover study on the effects of 2-L infusions of 0.9% saline and plasma-lyte(R) 148 on renal blood flow velocity and renal cortical tissue perfusion in healthy volunteers.
        Ann Surg. 2012; 256: 18-24
        • Boyd J.H.
        • Forbes J.
        • Nakada T.A.
        • Walley K.R.
        • Russell J.A.
        Fluid resuscitation in septic shock: a positive fluid balance and elevated central venous pressure are associated with increased mortality.
        Crit Care Med. 2011; 39: 259-265
        • Maitland K.
        • Kiguli S.
        • Opoka R.O.
        • et al.
        Mortality after fluid bolus in African children with severe infection.
        N Engl J Med. 2011; 364: 2483-2495
        • Pineau J.
        • Guez A.
        • Vincent R.
        • Panuccio G.
        • Avoli M.
        Treating epilepsy via adaptive neurostimulation: a reinforcement learning approach.
        Int J Neural Syst. 2009; 19: 227-240
        • Van Calster B.
        • Wynants L.
        • Verbeek J.F.M.
        • et al.
        Reporting and interpreting decision curve analysis: a guide for investigators.
        Eur Urol. 2018; 74: 796-804
        • Tinetti M.E.
        • Bogardus Jr., S.T.
        • Agostini J.V.
        Potential pitfalls of disease-specific guidelines for patients with multiple conditions.
        N Engl J Med. 2004; 351: 2870-2874
        • Boyd C.M.
        • Darer J.
        • Boult C.
        • Fried L.P.
        • Boult L.
        • Wu A.W.
        Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.
        JAMA. 2005; 294: 716-724
        • Gottesman O.
        • Johansson F.
        • Komorowski M.
        • et al.
        Guidelines for reinforcement learning in healthcare.
        Nat Med. 2019; 25: 16-18
        • Che Z.
        • Purushotham S.
        • Khemani R.
        • Liu Y.
        Interpretable deep models for ICU outcome prediction.
        AMIA Annu Symp Proc. 2016; 2016: 371-380
        • Gal Y.
        Uncertainty in deep learning: Dissertation 2016. University of Cambridge Web site.
        • Guo C.
        • Pleiss G.
        • Sun Y.
        • Weinberger K.
        On calibration of modern neural networks.
        https://arxiv.org/pdf/1706.04599.pdf
        Date accessed: July 13, 2019
        • Vergouwe Y.
        • Moons K.G.
        • Steyerberg E.W.
        External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients.
        Am J Epidemiol. 2010; 172: 971-980
        • Van Calster B.
        • Vickers A.J.
        Calibration of risk prediction models: impact on decision-analytic performance.
        Med Decis Making. 2015; 35: 162-169
        • Goldstein B.A.
        • Pomann G.M.
        • Winkelmayer W.C.
        • Pencina M.J.
        A comparison of risk prediction methods using repeated observations: an application to electronic health records for hemodialysis.
        Stat Med. 2017; 36: 2750-2763
        • McGlynn E.A.
        • Asch S.M.
        • Adams J.
        • et al.
        The quality of health care delivered to adults in the United States.
        N Engl J Med. 2003; 348: 2635-2645
        • Awad E.
        • Dsouza S.
        • Kim R.
        • et al.
        The Moral Machine experiment.
        Nature. 2018; 563: 59-64