If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author. Department of Surgery, Division of Acute Care Surgery, University of Virginia School of Medicine, 1300 Jefferson Park Avenue, Charlottesville, VA 22908, USA. (J.F. Calland).
Continuous predictive monitoring has been employed successfully to predict subclinical adverse events. Should low values on these models, however, reassure us that a patient will not have an adverse outcome? Negative predictive values of such models could help predict safe patient discharge. The goal of this study was to validate the negative predictive value of an ensemble model for critical illness (using previously developed models for respiratory instability, hemorrhage, and sepsis) based on bedside monitoring data in the intensive care units and intermediate care unit.
Methods
We calculated the relative risk of 3 critical illnesses for all patients every 15 minutes (n= 124,588) for 2,924 patients downgraded from the surgical intensive care units and intermediate care unit between May 2014 to May 2016. We constructed an ensemble model to estimate at the time of intensive care units or intermediate care unit discharge the probability of favorable outcome after downgrade.
Results
Outputs form the ensemble model stratified patients by risk of favorable and bad outcomes in both intensive care units/intermediate care unit; area under the receiver operating characteristic curve = .639/.629 respectively for favorable outcomes and .645/.641 for adverse events. These performance characteristics are commensurate with published models for predicting readmission. The ensemble model remained a statistically significant predictor after adjusting for hospital duration of stay and admitting service. The rate of favorable outcome in the highest and lowest deciles in the intensive care units were 76.2% and 27.3% (2.8-fold decrease) and 88.3% and 33.2% in the intermediate care unit (2.7-fold decrease), respectively.
Conclusion
An ensemble model for critical illness predicts favorable outcome after downgrade and safe patient discharge (hospital stay <7 days, no readmission, upgrade, or death).
Intensive care units (ICU) are facing increased numbers of patients, increased occupancy rates, greater acuity illnesses, and ultimately, increased costs.
This trend of increased ICU demand will continue to grow as the mean age of the population increases. Combined with the nationwide stagnating and even decreasing number of new board certifications in critical care medicine, surgery, and anesthesiology, the strain on intermediate and intensive care units is increasing.
Some well-known examples of aggregate, weighted, predictive models include the National Early Warning Score and Modified Early Warning Score for early identification of deterioration in an acute care ward, and Acute Physiology and Chronic Health Evaluation IV for prediction of ICU mortality.
Such systems may predict deterioration through evaluation of intermittent vital signs (which may be static for hours), laboratory results (such as serum electrolytes and blood counts), and/or demographics.
To better assess patient dynamics, predictive models have been developed in the past 10 years that use data from continuous bedside monitoring instead of intermittent, manually recorded values.
The only randomized controlled trial to date in the published scientific literature of implementation of a continuous predictive monitoring system demonstrated a dramatic impact.
developed a respiratory decompensation model in adult ICUs to predict urgent intubations, which was validated later in the intermediate care setting and for predicting a need for upgrade in level of care.
updated this respiratory instability model with improved performance and developed models that predict adult sepsis and hemorrhage in ICU patients.
Models using vital signs can be developed as prediction tools, because these adverse physiologic events (i.e., respiratory decompensation, sepsis, and hemorrhage) have identifiable signatures.
For example, prior to clinically apparent hemorrhage in adults, there is a characteristic decrease in diastolic blood pressure, and subclinical respiratory failure demonstrates an increased respiratory rate.
But, does a lack of signature suggest a lack of illness? Analysis of the heart rate characteristics score (to predict infant sepsis) showed that low values reclassified nonevent infants into a further low-risk stratum and may increase reassurance in the provider about low-risk infants.
The question then becomes should low values of predictive monitoring tools increase our reassurance that a patient will have no adverse outcome?
The benefits of predicting adverse events with the intention of intervening to prevent such events are inherently obvious decreased morbidity, decreased mortality, decreased durations of stay.
But equally important is the ability to predict favorable outcomes. Which patients can be transferred safely out of an ICU? Which patients can be discharged safely with a low risk of readmission? These are questions that affect both the patient and the health care system.
we developed and validated models to predict high impact events in the surgical ICU. In the present study, we wish to predict the opposite, i.e., the absence of critical illness such that a patient may be discharged safely. Therefore, we constructed an ensemble model based on the previously developed and validated models for respiratory instability, hemorrhage, and sepsis. We hypothesize that this ensemble model will have sufficient negative predictive value to predict favorable outcomes of patient populations in the ICU and intermediate care unit (IMU) at time of downgrade. The IMU often hosts patients discharged from the surgical ICU, or whose status is not severe enough to warrant ICU admission. Therefore, we also included the IMU due to the similarity of patient population and primary diagnosis between the ICU and IMU.
Methods
Study design
This retrospective study spanning May 2014 through May 2016 was conducted in a tertiary-care, level 1 trauma center with a 15-bed SICU and 12-bed surgical IMU (SIMU). All patients were managed by teams of board-certified surgical intensivists, residents, and nurse practitioners. All SICU and SIMU beds were connected to continuous physiologic monitoring systems and an electronic data warehouse that archives the complete medical record.
Primary outcome measures were duration of hospital stay after downgrade from an ICU/IMU, readmissions, and adverse events (upgrades in level of care and mortality). These data were identified automatically from records derived from the University of Virginia Data Warehouse.
We used the statement checklist of the Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis in analyzing and reporting this study.
The University of Virginia Institutional Review Board approved this study with a waiver of informed consent.
Study populations and outcome definitions
Inclusion criteria were defined as any patient >17 years old who was admitted to the SICU or SIMU for at least 6 h and downgraded from the unit within the study period noted above. Multiple admissions from the same patient in either unit were included and considered as independent for the analysis. Admissions without archived data of physiologic monitoring due to technical complications were excluded.
We classified each patient downgrade based on the patient's status within 7 days of downgrade. Downgrades were classified as (1) adverse event (i.e., IMU or ICU admission or death within 7 days), (2) readmission or prolonged hospital stay (i.e., hospital readmission within 7 days or hospital duration of stay >7 days, excluding downgrades with adverse event), or (3) favorable outcome (i.e., discharge from hospital within 7 days, without adverse event or hospital readmission within 7 days). Each downgrade falls into exactly one classification. We selected 7 days rather than the common 30-day definition by the Centers for Medicare and Medicaid Services, because we wanted to focus on patients where readmission is due to unnoticed physiologic derangement at the time of discharge. In addition, recent work indicates that shorter term readmission, i.e., within 7 days, may be a better indicator of quality of care.
A predictive analytics approach to reducing 30-day avoidable readmissions among patients with heart failure, acute myocardial infarction, pneumonia, or COPD.
All outcomes were identified via the data warehouse, and sampling of individual patient records were then reviewed systematically through the electronic medical records, which allowed us to confirm data warehouse information.
Acquisition of histologic data and predictors
All SICU and SIMU beds are connected to a GE Carescape Gateway (GE Healthcare 9900 W. Innovation Drive Wauwatosa, WI) continuous bedside monitoring system for data collection, which is then captured by software developed by Bernouilli (Bernoulli Enterprise, Inc, Milford, CT). Measurements reported by the bedside monitor every 2 seconds included HR, RR, SpO2 and both invasive and noninvasive blood pressure. These data were downloaded and warehoused daily on a custom grid-computing cluster. Stored data included a timestamp and bed assignment, and data were linked to medical record information stored in the Clinical Data Repository to reunite physiologic data with clinical information.
We calculated in 30-minute windows with 50% overlap the following measures: the mean and standard deviation of every 2 s heart rate (HR), respiratory rate (RR), pulse oximetry (SO2), and noninvasive blood pressure or invasive blood pressure in its absence (BP); 3 pairwise, cross-correlations between HR, RR, and SO2; mean and standard deviation of RR intervals (HRV); local dynamics score
and all calculations were performed using CoMET (AMP3D Inc., Charlottesville, VA).
Statistical analysis
We calculated every 15 min looking back 30 min the relative risk of sepsis, respiratory failure, and hemorrhage using the features described above and the models of Moss et al.
We then estimated the absolute risk of each patient for any critical illness after downgrade by averaging the 3 relative risk estimates; we refer to this as the ensemble model. Finally, we calculated the average ensemble model output over the 12 h prior to downgrade. These scores underwent analysis of negative predictive value and thus, validation for predicting favorable outcomes in both an ICU and IMU. All statistical analyses were performed in MatLab (The MathWorks Inc., Natick, MA) and R
The confidence intervals for area under the receiver operating characteristic curve (AUC) were obtained by bootstrapping.
Results
Study population
From May 1, 2014, through May 4, 2016, there were 5,355 consecutive patient admissions to the SICU and SIMU combined, of which 3,999 (2,924 patients) met the above inclusion criteria. Of those downgrades from the SICU, 1,392 (62%) were favorable outcomes, 619 (28%) were readmissions/prolonged stays, and 218 (10%) were adverse events. Of those downgrades from the SIMU, 1,302 (74%) were favorable outcomes, 321 (18%) were readmissions/prolonged stays, and 147 (8%) were adverse events. Baseline demographics, including admitting service and duration monitored, are listed in Table 1 (SICU) and Table 2 (SIMU). While readmissions and prolonged stays affect the patient and health care system differently, the model outputs were similar (P = .040 in SICU, P = .048 in SIMU), and therefore, we combined these groups for simplicity. Additionally, continuous predictive monitoring for respiratory instability and hemorrhage was displayed in the SICU beginning June 22, 2015; however, favorable outcome rates were not different before (0.629) and after (0.620; P = .67).
Table 1SICU population baseline patient demographics.
Across the SICU and SIMU, we analyzed 124,588 observations of data at 15-minute intervals, which represents 847.3 patient-days (81,343 observations) for favorable outcomes, 304.1 patient-days (29,196 observations) for readmissions/prolonged stays, and 146.3 patient-days (14,049 observations) for adverse events. Hence, the overall incidence rate (SICU and SIMU) for favorable outcomes was 65.3% (81,343 of 124,588 observations), for readmissions/prolonged stays was 23.4% (29,196 of 124,588), and for adverse events was 11.3% (14,049 of 124,588).
Development of the model, specification, and performance
Continuous monitoring data that contribute most to the models include: for the respiratory instability model—RR, SpO2, variability in SpO2, and the cross-correlation coefficient of HR and SpO2; for the hemorrhage model—HR, diastolic blood pressure (DBP), SpO2, and variability in SpO2; for the sepsis model—HR, DBP; and for the ensemble model—all listed previously. Patients experiencing favorable outcomes demonstrated decreased HR and decreased RR compared with all other outcome categories (Fig 1). Average HR at downgrade from the SICU was 83.8 for favorable outcome versus 89.3 (P < .001) and 91.0 (P < .001) for readmission/prolonged stay and adverse event, respectively. Average HR at downgrade from the SIMU was 83.8 for favorable outcome vs 87.7 (P < .001) and 87.3 (P < .001) for readmission/prolonged stay and adverse event, respectively. Similarly, average RR at downgrade from the SICU was 20.2 for favorable outcome versus 21.0 (P = .006) and 22.1 (P < .001) for readmission/prolonged stay and adverse event, respectively. Average RR in the SIMU was 20.8 for favorable outcome vs 21.7 (P < .001) and 21.3 (P < .001) for readmission/prolonged stay and adverse event, respectively. Note that the magnitude of these differences is likely too small to be of use clinically, hence the value of statistical models that calibrate differences in multiple variables and create a more clinically useful risk estimate.
Fig. 1The average value of some vital sign statistics used in the model are shown in the 24 hours prior to downgrade. Compared with other outcome categories, the average HR for favorable outcomes is less in the SICU (A) and SIMU (B). Similarly, the average RR for favorable outcomes is less in the SICU (C) and SIMU (D). The SpO2 is not significantly different across categories in neither the SICU (E) nor SIMU (F). The gray lines represent the overall average of each vital sign for all patients at all times in the unit. FO, favorable outcome; RA + PS, readmission or prolonged stay; AE, adverse event.
The time series of the ensemble model output averaged across patients in each outcome classification is shown for the 24 hours prior to downgrade from the SICU and SIMU in Fig 2, A and B, respectively. The model outputs are consistently less for the favorable outcome group than the other 2 groups, and decreases during the 24 hours prior to downgrade. A visual depiction of the distribution of model outputs across outcomes categories is shown in Fig 3. These curves (like a histogram) show where the model outputs lie for each patient group; lesser values are to the left and indicate patients with lesser estimated risk of sepsis, respiratory instability, and hemorrhage. The distribution of model outputs is less (i.e., shifted to the left) in patients with favorable outcome. Model outputs are less in the SIMU than the SICU, as one might expect due to the lesser-risk nature of IMU patients. The average ensemble model output in the last 12 hours prior to downgrade was significantly different across outcome categories as shown in Table 3. Model outputs are least for the favorable outcome group, greatest for adverse event group, and intermediate for readmission/prolonged stay. The distributions of the 3 groups are statistically different, and favorable outcome is significantly lesser than the combination of readmission/prolonged stay and adverse events.
Fig. 2The average value of each model in the SICU and SIMU (respiratory instability, A and B; hemorrhage, C and D; sepsis, E and F; and ensemble, G and H, respectively) are shown in the 24 hours prior to downgrade. The gray lines represent the overall average of each model output for all patients at all times in the unit. FO, favorable outcome; RA + PS, readmission or prolonged stay; AE, adverse event.
Fig. 3Distribution of outputs (log scale) of the ensemble model outputs across outcome categories in the SICU (A) and SIMU (B) are shown. As a comparison to the present patient population of downgraded patients, the gray dotted lines represent patients who experienced an adverse event during the index unit admission (were not downgraded, and thus not included in this study of favorable outcomes). In the SICU, AE without downgrades represent deaths. This population in the SIMU (previously studied by Blackburn et al
P value of favorable outcome group versus other groups combined as one.
SICU
0.70 ± 0.36
0.87 ± 0.47
1.00 ± 0.68
<.0001
<.0001
SIMU
0.65 ± 0.33
0.77 ± 0.33
0.89 ± 0.60
<.0001
<.0001
Significance was determined using the nonparametric Kruskal-Wallis test for analysis of variance and the Wilcoxon rank-sum test for continuous variables.
* P value of favorable outcome group versus other groups combined as one.
To quantify the performance of the ensemble model as a function of time horizon for outcomes, we evaluated the model on sliding cumulative windows of available data in 1-day increments over 30 days (Fig 4). The 3 models were developed originally to predict subclinical illness within 4 − 8 hours
; thus, expectedly, the predictive value of the ensemble model is limited as time increases. The SIMU population displayed a pronounced decrease in AUC at 7 days, while the model in the SICU population maintained predictability for a much greater duration. At 7 days, the ensemble model maintained a significant predictive value with an AUC of 0.639 and 0.629 for favorable outcomes in the SICU and SIMU, respectively; and an AUC of 0.645 and 0.641 for adverse events, respectively. Duration of any-unit hospital stay prior to downgrade, age, and surgical service also were highly significant predictors of outcome category (Table 1); however, the ensemble model remained significant after adjusting for these variables. Note that this is the performance for existing models (designed to predict critical illness) when evaluated for predicting favorable outcome; one expects better performance by training a model specifically for this task or by including other data elements, such as nursing vital signs and laboratory measurements.
Fig. 4Area under the receiver operating characteristic curve (AUC) for outcomes as a function of time after downgrade from the SICU (A) and SIMU (B). The gray line indicates 7 days, at which point the model in the SIMU population displays a decrease in predictability.
The probability of a favorable outcome was, not surprisingly, greater in the SIMU population (78.6%) than the SICU population (58.4%; Fig 5). In the SICU population, the rate of favorable outcome in patients with the least decile ensemble scores was 76.2% while favorable outcome in patients with the greatest decile scores occurred only 27.3% of the time (a 2.8-fold decrease). That is, a patient with an ensemble score of 1.8 is 2.8-times more likely to have readmission, prolonged stay, or adverse event than a patient with a score of 0.3. In the SIMU, the rate of favorable outcome in patients with the least decile scores was 88.3% while favorable outcome in patients with the greatest decile scores occurred only 33.2% of the time (a 2.7-fold decrease).
Fig. 5Probability of a favorable outcome in the SICU (A) and SIMU (B) as a function of the outputs of the ensemble model is shown. The gray lines represent the average likelihood of favorable outcome across all patients and model outputs. Each dot represents the deciles of model output with the first dot being the highest decile and the last dot being the lowest decile.
described statistical models for early detection of respiratory failure leading to urgent, unplanned intubation, hemorrhage leading to large, unplanned transfusion, and sepsis in a surgical ICU. The goal of our study was to validate the negative predictive value of these models in both the surgical ICU and IMU. Patients in the surgical IMU can be thought of as a subset of the less critically ill in the SICU, because patients in the IMU either come from the ICU or are admitted because their status approaches that of an ICU patient. We found that all models that were tested predicted favorable outcomes successfully via low model output scores. An ensemble created by averaging the 3 model predictions was the most predictive strategy and remained predictive when demographics and electronic medical records characteristics were removed, leaving only vital signs.
Why identify the low risk patient? Maintaining constant vigilance is expensive both fiscally and in resources. At >$1,500 per day, the costs of admissions requiring ICU stays quickly add up.
Readmissions in themselves are likewise costly; postoperative complications resulting in readmission (such as delayed postoperative ileus, anastomotic leak, or surgical site infection) can cost >$6,000.
Thus, there is inherent benefit in identifying and preventing readmissions and ICU vigilance when feasible. The European Society of Cardiology pointed out recently that 3 continues to be a definitive need for “an objective and reliable definition of low-risk characteristics to identify early discharge candidates” in ER patients with symptoms of heart failure.
European Society of Cardiology—Acute Cardiovascular Care Association position paper on safe discharge of acute heart failure patients from the emergency department.
There are many studies in the scientific literature examining safe patient discharge, particularly for heart failure. Two such examples studied the association between instability of vital signs at discharge and post-hospital outcomes.
Increased heart rates at discharge are associated with increased all-cause mortality in patients admitted for heart failure. The risk of 1-year mortality in this population increases 18% for every 10 beats per minute >75 at discharge.
found that ≈20% of patients admitted to any medicine service were discharged from the hospital with at least one unstable vital sign, and of patients discharged with 1 to 3 vital sign instabilities, ≈ 17% to 26%, respectively, died or were readmitted within 30 days.
Interestingly in the present study, we also found that greater heart rates and respiratory rates indicated decreased likelihood of favorable outcomes. The differences, though, were very small—about 7 beats per minute and 2 breaths per minute—and, thus, discerning predictions via clinician observation in this population would be very difficult.
Other studies using a multivariate approach have evaluated the predictive capacity of a variety of inputs—from electronic health records, to clinical data, to administrative data—and of a variety of statistics—simple risk scores, neural networks, and machine-learning models.
Validated, electronic health record deployable prediction models for assessing patient risk of 30-day rehospitalization and mortality in older heart failure patients.
Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches.
No predictive strategy is believed to be sufficient for clinical practice; however, the AUCs we found—0.629 to 0.645—compare favorably with those reported in a review of hospital readmission models by Kansagara et al.
It is important to note that this performance is for existing models used to predict events for which these were not trained. Training models to predict specifically favorable outcomes is expected to yield better performance, but is beyond the scope of the current study; we focused instead on clinically implemented models for critical illness that are likely to be used to inform discharge decisions. In addition, the predictions presented here used only the bedside monitoring data that are available and updated continuously for every ICU patient. Decision support tools such as these are intended to assist clinical personnel in identifying patients at risk for bad outcome after discharge and should be evaluated in the complete clinical context for an individual patient of interest.
This work was conducted at a single center via a retrospective analysis. Additionally, the display of this model does not guarantee improved outcomes, because prospective analysis and displays in adults have not yet been tested in clinical practice. We used logistic regression which makes assumptions about monotonic relationships between predictor variables and outcomes. We note, though, a recent study found no benefit of adding any of a large number of modern techniques of machine learning to the statistical modeling.
Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches.
We also note that these models were developed and evaluated at a single institution; it remains to understand the performance of such models across other care units (e.g., the medical ICU), as well as other institutions with different patient populations and underlying illness.
Taken together our studies and those of others point to a fundamental difficulty in estimating the risk of adverse event at the time of downgrade. In our case, we used models trained to detect imminent catastrophe in an ICU as opposed to more subacute complications that might arise outside the ICU. In addition, nonphysiologic parameters, such as home situation, chronic medical comorbidities, socioeconomic status, insurance, and others, all play very important roles to the outcome of the downgraded or discharged patient.
In conclusion, a low value of predictive statistical models of respiratory instability, hemorrhage, and sepsis is reassuring that downgrade from the surgical ICU or IMU and discharge from the hospital is low risk. Thus, display of continuous predictive analytics may aid clinicians in both anticipating adverse events when the values are high and conversely, predicting safe downgrade when the values are low.
Supplementary data
The following is the supplementary data to this article:
Duke E.M. The Critical Care Workforce: A Study of the Supply and Demand for Critical Care Physicians. 2006 (US Department of Health & Human Services, Health Resources & Services Administration)
A predictive analytics approach to reducing 30-day avoidable readmissions among patients with heart failure, acute myocardial infarction, pneumonia, or COPD.
European Society of Cardiology—Acute Cardiovascular Care Association position paper on safe discharge of acute heart failure patients from the emergency department.
Validated, electronic health record deployable prediction models for assessing patient risk of 30-day rehospitalization and mortality in older heart failure patients.
Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches.
30656-6&id=ymsy5253-fig-0001.jpg){kind=link}
30656-6&id=ymsy5253-fig-0002.jpg){kind=link}
30656-6&id=ymsy5253-fig-0003.jpg){kind=link}
30656-6&id=ymsy5253-fig-0004.jpg){kind=link}
30656-6&id=ymsy5253-fig-0005.jpg){kind=link}