If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Center for Healthcare Outcomes and Policy, Department of Surgery, University of Michigan, Ann Arbor, MIGeneral Surgery Residency Program, Department of Surgery, University of Michigan, Ann Arbor, MI
In 2011, the Accreditation Council for Graduate Medical Education (ACGME) implemented additional restrictions on resident work hours. Although the impact of these restrictions on the education of surgical trainees has been examined, the effect on patient safety remains poorly understood.
Methods
We used national Medicare Claims data for patients undergoing general (n = 1,223,815) and vascular (n = 475,262) surgery procedures in the 3 years preceding the duty hour changes (January, 2009–June, 2011) and the 18 months thereafter (July, 2011–December, 2012). Hospitals were stratified into quintiles by teaching intensity using a resident to bed ratio. We utilized a difference-in-differences analytic technique, using nonteaching hospitals as a control group, to compare risk-adjusted 30-day mortality, serious morbidity, readmission, and failure to rescue (FTR) rates before and after the duty hour changes.
Results
After duty hour reform, no changes were seen in the measured outcomes when comparing teaching with nonteaching hospitals. Even when stratifying by teaching intensity, there were no differences. For example, at the highest intensity teaching hospitals (resident/bed ratio of ≥0.6), mortality rates before and after the duty hour changes were 4.2% and 4.0%, compared with 4.7% and 4.4% for nonteaching hospitals (relative risk [RR], 0.98; 95% CI, 0.89–1.07). Similarly, serious complication (RR, 1.02; 95% CI, 0.98–1.06), FTR (RR, 0.95; 95% CI, 0.87–1.04), and readmission (odds ratio, 1.00; 95% CI, 0.96–1.03) rates were unchanged.
Conclusion
In Medicare beneficiaries undergoing surgery at teaching hospitals, outcomes have not improved since the 2011 ACGME duty hour regulations.
Concerns for patient safety have led to several landmark changes in the structure of resident training over the past decade. In July 2003, the Accreditation Council for Graduate Medical Education (ACGME) enacted the first restrictions to resident duty hours.
These restrictions limited residents to an average of ≤80 hours per week in the hospital, limited the frequency of on-call shifts to no more than 1 in 3 nights, and mandated ≥1 day off in 7. In 2008, the Institute of Medicine provided further recommendations for resident training, stressing the impact of prolonged and variable shift lengths on trainees' fatigue, as well as the potential link between provider fatigue and patient safety.
These restrictions limit postgraduate year 1 (PGY-1) trainees to a 16-hour maximum shift, require a minimum rest period between shifts of 8 hours, and increase the specificity of supervisory requirements.
Resident duty hours in the learning and working environment: comparison of 2003 and 2011 standards. Accreditation Council for Graduate Medical Education,
Chicago2011
Whether these most recent changes have had the intended effect on patient outcomes remains unclear. There is evidence to suggest that these changes may have benefits, largely by reducing trainee fatigue. A number of studies have shown increased medical error rates by fatigued trainees,
However, with the 2011 regulations, there is concern that the shortened shifts have necessitated an increase in transitions of care and patient handoffs as hospitals move to a shift-based care system.
The increase in handoffs and potential for communication errors may offset any potential gains from decreased provider fatigue. Although a recent study analyzed the impact of these changes on patient outcomes, this study was limited in that it focused on only a small proportion of surgical residencies, and focused only on residency programs participating in the National Surgical Quality Improvement Program (NSQIP).
; as such, these hospitals may have robust mechanisms in place that limit their sensitivity to small variations in resident care as a result of the duty hour regulations.
In this context, we designed a study to evaluate patient outcomes in the period immediately after the most recent duty hour reforms in a national surgery population. To do this, we used Medicare claims data and incorporated a broad range of surgical conditions. This analysis includes essentially every hospital participating in general surgery training. We used a control group of nonteaching hospitals and compared outcomes among hospitals of varying teaching intensity to this nonteaching control.
Methods
Data source and study population
We used complete Medicare claims data for patients undergoing 11 different common general and vascular surgery procedures (cholecystectomy, colectomy, incisional hernia repair, appendectomy, gastrectomy, pancreatic resection, liver resection, esophagectomy, abdominal aortic aneurysm repair, lower extremity bypass, carotid endarterectomy) during 2 time periods: January 2009–June 2011 and July 2011–December 2012. Patients who were <65 years of age or >99 years of age were excluded from the study, as were patients not enrolled in Medicare parts A and B, and patients enrolled in Medicare Advantage. This selection method for Medicare patients has been described previously by our group.
We excluded patients whose hospitalization spanned from June 30, 2011, to July 1, 2011, because they would have been hospitalized both before and after the institution of the new regulations. We also included only those patients whose index hospitalization occurred during the study period; that is, we did not include a patient in the post-reform cohort if they had previously been admitted for the same condition before the reform, because this would represent a crossover between the 2 study periods. Patients were thus assigned to the pre-reform or post-reform cohorts based on their index hospitalization. For the small proportion of patients admitted in June of 2011, any outcome measure occurring after July 1 was attributed to the pre-reform period. Further, at the hospital level we excluded hospitals that either opened or closed during the study period, and very small hospitals with <350 admissions per year. The rationale for excluding these small hospitals is that they may not be true acute care facilities, and their outcomes would be less reliable owing to their small size.
Patient outcomes were 30-day mortality, serious morbidity, failure to rescue (FTR), and readmission in each time period. To identify morbidity rates, we utilized the Complication Screening Project, a validated method for identifying postoperative complications, which identifies 8 different types of complications from administrative data.
The categories of these complications are pulmonary failure, pneumonia, myocardial infarction, deep venous thrombosis or embolism, acute renal failure, postoperative hemorrhage, surgical site infection, and gastrointestinal bleeding. We focused on serious complications, which we defined as the presence of a postoperative complication combined with an extended length of stay (>75th percentile). This method has been used previously, and attempts to increase the specificity of complication measurement by ignoring coded complications that do not have meaningful clinical impact. FTR was measured using methods previously described,
defining FTR as mortality within 30 days after a recognized complication.
Teaching intensity was measured at each hospital using a ratio of residents to patient beds; this ratio is a commonly used measure of teaching intensity and is linked to Medicare's indirect Medical Education payment adjustments.
The number of residents at each hospital is obtained from the Center for Medicare and Medicaid Services' Medicare Cost Reports; the accuracy of this number is considered very high owing to its linkage to graduate medical education funding. We then stratified hospitals into quintiles of teaching intensity using this ratio, in a fashion that has been previously used in studies evaluating teaching hospitals.
The bottom quintile was all nonteaching hospitals (resident to bed ratio = 0); the remaining quintiles were stratified from “very minor” to “very major” degrees of teaching (very minor teaching > 0–0.05; minor > 0.05–0.25; major > 0.25–0.60; very major > 0.60).
Statistical analysis
Our study aimed to evaluate changes in patient outcomes after the institution of the new duty hour regulations in July 2011. We used an econometric technique for evaluating the impact of public policy, the difference-in-differences approach, for our analysis. This method attempts to isolate the effects associated with a policy change and exclude any other changes over the same time period. The difference-in-differences technique compares the hospitals with themselves in the periods before and after duty hour reforms, adjusting for any time invariant differences between the groups of hospitals between the 2 time periods. It also compares the hospitals against a control group, in this case nonteaching hospitals, to prevent any temporal changes that affected outcomes in all hospitals over the same time period from being attributed to the change in resident duty hours.
Before performing the difference-in-differences analysis, we first assessed the trends in outcomes before the duty hour regulation changes. If there were differing trends in our measured outcomes between the hospital groups in the period leading up to the duty hour change and these differences persisted after the change, these differences could be mistakenly attributed to the duty hour regulations themselves. Over the 2 years before the 2011 regulations, the trends in performance were parallel between the teaching hospitals and nonteaching control group for the majority of our measured outcomes. However, for mortality, for the major and very major teaching intensity quintiles, mortality rates were decreasing at a faster rate than at less teaching intense hospitals and nonteaching hospitals. Therefore, for mortality, we compared the post-reform mortality rate to only the most recent year pre-reform rather than the entire pre-reform period to mitigate this differing trend. It is notable that in the difference-in-differences model this would only exaggerate any significant decrease in mortality among major teaching hospitals, if present.
To perform the difference-in-differences analysis, we used logistic regression models to evaluate the relative risk (RR) of each of our outcome measures (mortality, serious morbidity, readmission, FTR) before and after the implementation of the 2011 duty hour changes, in each quintile of teaching intensity compared with the nonteaching control group. The RR of each outcome was measured in each hospital quintile comparing the rates before and after the reform and against the nonteaching hospital control. In our models, each outcome measure was analyzed separately and served as the dependent variable. A dummy variable (year of surgery) was used to indicate whether the patient had surgery in the pre- or post-duty hour reform time period. A continuous time variable was included, to account for secular trends over time. Our exposure variable was the teaching status of the hospitals, using teaching status as a dummy variable. The coefficient of this interaction term between the exposure (teaching status) and the pre–post duty hour reform variable indicates the independent effect of the duty hours regulation changes on each outcome measure (the difference-in-differences estimate).
Further, our regression models were risk adjusted using the Elixhauser comorbidity index.
This commonly used method of risk adjustment has been validated externally; it uses 29 different comorbidities abstracted from administrative data to account for potential differences in the patient populations of our different hospital cohorts. The models were also adjusted for hospital characteristics, including annual volume and case mix. We used RRs rather than odds ratios as our main analysis owing to the limitations in accuracy of odds ratios when examining a common outcome variable.
All statistical analyses were conducted using STATA 13.0 (College Station, TX). All P values are 2-sided. This study was considered exempt from review by the University of Michigan Institutional Review Board.
Results
Patient characteristics before and after the duty hour changes are shown in Table I. We included 1,223,815 general surgery and 475,262 vascular surgery patients in the study. There were small differences in patient demographics across the 2 periods that reached statistical significance (Table I). The post-reform cohort had a slightly higher incidence of comorbid diseases, with higher rates of obesity, diabetes, hypertension, peripheral vascular disease, and congestive heart failure. Table I also demonstrates the characteristics of the hospitals included in the analysis, with mean bed numbers and the number of teaching versus nonteaching hospitals.
Table IPatient and hospital characteristics before and after institution of 2011 ACGME duty hour changes
The Figure shows the change over time in all outcome variables, before and after the reform, for each quintile of teaching intensity. The rates of mortality, serious morbidity, readmission, and FTR were relatively stable over time for each group of hospital, although the baseline rates of adverse events vary with teaching intensity. Table II shows the rates of mortality, serious morbidity, readmission, and FTR in both the pre-reform and post-reform periods, across each quintile of teaching intensity. The RRs of a simple before-and-after comparison for each outcome are also shown in Table II.
FigureTrends in adverse outcomes over time, before and after the institution of the 2011 ACGME duty hour reforms, stratified by level of hospital teaching intensity. A, Mortality. B, Serious complications. C, Readmissions. D, Failure to rescue.
The independent effect is derived from the Difference-in-differences model and represents changes in outcome before and after the duty hour reform in the quantiles of teaching intensity, compared with nonteaching hospitals (the control group).
Mortality
Nonteaching
4.7
4.4
0.96 (0.91–1.01)
1 (Reference)
Very minor teaching
4.7
4.4
0.91 (0.82–0.99)
1.00 (0.95–1.05)
Minor teaching
4.6
4.4
0.99 (0.92–1.06)
1.01 (0.96–1.06)
Major teaching
4.7
4.6
1.15 (1.06–1.23)
1.03 (0.97–1.08)
Very major teaching
4.2
4.0
0.95 (0.79–1.11)
0.99 (0.90–1.08)
Serious complications
Nonteaching
11.3
10.9
0.95 (0.92–0.98)
1 (Reference)
Very minor teaching
12.3
12.1
0.93 (0.88–0.98)
1.01 (0.98–1.04)
Minor teaching
12.9
12.4
0.95 (0.91–0.99)
0.99 (0.96–1.01)
Major teaching
14.6
14.9
0.98 (0.93–1.03)
1.02 (0.99–1.05)
Very major teaching
14.7
14.6
0.94 (0.88–1.01)
1.02 (0.98–1.06)
Readmissions
Nonteaching
12.6
12.2
1.02 (0.99–1.04)
1 (Reference)
Very minor teaching
12.9
12.4
1.03 (0.98–1.07)
0.99 (0.96–1.01)
Minor teaching
12.9
12.5
0.98 (0.95–1.02)
0.99 (0.97–1.02)
Major teaching
15.7
15.0
0.96 (0.91–1.02)
0.98 (0.95–1.01)
Very major teaching
16.6
16.0
1.02 (0.95–1.09)
1.00 (0.96–1.03)
Failure to rescue
Nonteaching
17.0
15.7
0.97 (0.92–1.02)
1 (Reference)
Very minor teaching
16.5
15.4
0.95 (0.87–1.04)
1.00 (0.95–1.05)
Minor teaching
16.1
15.5
1.01 (0.94–1.07)
1.03 (0.99–1.07)
Major teaching
15.7
15.0
1.12 (1.03–1.20)
1.01 (0.95–1.07)
Very major teaching
14.3
13.3
0.94 (0.78–1.10)
0.95 (0.87–1.04)
∗ Adjusted for patient factors, hospital volume, procedure type.
† The independent effect is derived from the Difference-in-differences model and represents changes in outcome before and after the duty hour reform in the quantiles of teaching intensity, compared with nonteaching hospitals (the control group).
The difference-in-differences analysis that compares teaching hospitals versus nonteaching shows no significant changes in the RR of any of our outcome measures after the duty hour changes (Table II). For example, when comparing very major teaching hospitals with nonteaching hospitals, the RR for mortality was 0.99 (95% CI, 0.90–1.08). Similarly, no change was seen in the RR of serious complication (RR, 1.02; 95% CI, 0.98–1.06), readmission (RR, 1.00; 95% CI, 0.96–1.03), or FTR (RR, 0.95; 95% CI, 0.87–1.04) after the 2011 duty hour reforms.
We also evaluated the change in outcomes within each strata of teaching intensity relative to nonteaching hospitals and found similar results (Table II). We then evaluated outcomes for general and vascular procedures separately, with similar findings. For example, in the very major hospital quintile, the RR of mortality when compared with the nonteaching controls was 0.99 (95% CI, 0.90–1.09) for general surgery procedures and 0.93 (95% CI, 0.72–1.14) for vascular procedures.
Discussion
After 20 years without significant change in the structure of resident training programs, concerns for patient safety led the ACGME to increase their oversight and limitations on resident duty hours beginning with the 2003 regulations. In this study, we conducted a large-scale analysis of the most recent 2011 ACGME duty hour restriction on outcomes in a surgery population. We found that a broad range of outcome measures were unchanged after enactment of these new regulations. In this analysis, we compared teaching with nonteaching hospitals; even when evaluating these changes across the spectrum of teaching intensity, we found no effects on patient outcomes.
Most prior research evaluating the ACGME's duty hour policies has focused on the initial changes in 2003. Several early studies, largely from single institutions, attempted to identify changes in outcomes in the surgery population and showed conflicting results. A recent systematic review of duty hour changes identified 35 such studies examining patient safety that were deemed sufficiently high quality; 48% of these demonstrated no change in patient outcomes and 19% identified any improvement.
This review demonstrates the ongoing uncertainty regarding the impact of the duty hour reforms on patient outcomes. Additionally, several studies have attempted to evaluate the impact of these regulations on residents themselves. Overall, a number of studies have demonstrated significant improvement in quality of life,
There is much less literature evaluating the recent 2011 duty hour changes. Two recent studies evaluated the impact of these changes on patient outcomes. A study published by the FIRST trial group containing NSQIP participating hospitals found no impact of the duty hour regulations.
However, their analysis only included a subset of teaching hospitals representing a minority of the surgery residency programs in the United States. Participation in the NSQIP may have acted as a confounder, given that these hospitals are engaged actively in high-level quality improvement initiatives that may mask changes resulting from resident quality of care. A similarly structured analysis was published in a Medicare population; however, this study included only mortality and readmissions data as outcome measures, included a mixed medical and surgical population, and had limited postintervention follow-up.
Our study builds on these existing studies by including a broader set of patient outcome measures in a national patient population encompassing essentially all surgery residency programs. In addition to studying mortality rates and a composite measure of complication rates, we also assessed readmissions, because hospital discharges require careful communication and coordination of care. We also included FTR as an outcome measure, because rescue itself requires timely diagnosis and management, typically performed by residents at the bedside. In our analysis, we felt that FTR would be a particularly sensitive metric to identify any changes as a result of the duty hour regulations.
There have been additional studies focusing on intermediate outcomes, such as handoffs or medical errors. A recent study demonstrated that trainees receive inadequate training on giving and receiving effective patient sign-out, and that a number of practical barriers exist to effective handoffs in surgical residencies. In 1 preliminary study of night-float–based shift systems in a medical residency program, the night-float model was actually abandoned after nursing staff reported increased medical errors and communication lapses.
Effect of the 2011 vs 2003 duty hour regulation-compliant models on sleep duration, trainee education, and continuity of patient care among internal medicine house staff: a randomized trial.
Given these potential unintended negative consequences, it is important to understand whether these changes are having their intended impact on patient outcomes. Our study adds to the understanding of this policy change by demonstrating in a national surgical population the impact on patient outcomes.
Our study has several important limitations. First, this was a retrospective study of Medicare data. The use of administrative data has several limitations pertinent to our findings. However, given our goals of assessing the national impact of the duty hour regulations, this is clearly the most appropriate dataset and the advantages afforded by Medicare data outweigh these limitations. In addition to providing a large national sample, the Medicare population is an inherently high-risk one, and may have greater sensitivity to small variations in the quality of patient care. There are well-known limitations in coding of complications within administrative datasets. Therefore, we limited our assessment to a recognized and validated subset of complications using a composite measure.
Further, to increase the specificity of the complication codes, we combined them with an extended length of stay criterion, which we labeled serious complications.
Administrative data also have limitations in the accuracy of coded comorbidities. In our dataset the post-reform cohort seemed to have higher rates of obesity and obesity-related comorbidities, including diabetes and hypertension, likely reflecting national trends over time. To minimize the impact of this change in comorbidities over time, we used the best available measures of risk adjustment,
and included a time variable in our regression model. The use of the difference-in-differences analytic method and the longitudinal nature of the study mitigate bias owing to unmeasured comorbidities as well. Another limitation is the limited postintervention follow-up period. We have used the most recently available Medicare data to maximize the duration of the study period. In addition, the 2011 duty hours most significantly impact PGY-1 trainees, suggesting that any impact on outcomes should be seen in the short term rather than the long term. Our study used a common measure to assess teaching intensity; however, the accuracy of this for surgery training programs may be limited. Within the Medicare data, we were not able to specifically assess the quantity of surgical trainees at a given hospital, only the overall number of residents in all disciplines. However, general surgery programs are among the most common residencies at teaching hospitals. Further, at the highest quintile of teaching intensity, these hospitals are typically large, academic medical centers that would universally have surgery training programs. The resident to bed ratio in Medicare data may not reflect fully the resident complement of a hospital, because a number of large hospitals have additional unfunded trainees over their Medicare cap. However, this would bias our measure of teaching intensity toward underestimating intensity, thereby strengthening the comparison with the nonteaching control group. The use of only a surgery patient population may represent a limitation as well, given that the duty hour regulation changes have impacted all disciplines of resident trainees. However, postoperative patients are particularly high risk for an in-hospital error and require intensive floor care. The timely diagnosis and rescue from surgical complications is a surveillance-dependent process in which residents play a key role.
These findings have important implications for resident work hour regulations set by the ACGME. The intention of the 2003 regulations was primarily to improve patient outcomes through a strategy of fatigue mitigation by duty hour regulation. Importantly, these same regulations were also meant to improve trainees' work–life balance and quality of training. Despite much initial controversy, the 2003 duty hour regulations have now come to be largely regarded as salutary for trainees' quality of life and safe for patient care. The impact of the 2011 regulations remains much less clear. There has been concern that these policy changes have not had their intended impact on trainees—that they have not succeeded in reducing fatigue and may have had unintended negative consequences by increasing work compression and increasing patient handoffs. The regulation of resident work hours has affected hospitals significantly as well, forcing changes in staffing patterns, supervision, and increased hiring of mid-level practitioners to offset the reduced hours of trainees.
Although our data demonstrate no impact on patient outcomes, this does not reflect the degree of effort undertaken by hospitals to maintain the same level of care. Further, the implementation of the 16-hour shift limit and the required 8-hour break between shifts have limited individual residency programs' flexibility to the extent that they may not be able to meet the educational needs of their trainees. Residents themselves echo a number of these concerns, particularly regarding the impact on their quality of life outside the hospital. In a recent national survey, Drolet et al
found that the majority of surgery residents believed the 2011 regulations had worsened both quality of life for senior trainees and preparation for their role as senior residents and attendings.
In acknowledgement of this growing level of concern, the ACGME (working with the American Board of Surgery and the American College of Surgeons) has authorized the first prospective, randomized trial comparing different duty hour paradigms, the Flexibility in Surgery Resident Duty Hours Trial (FIRST Trial).
This trial has randomized hospitals/residency programs to 2 study arms: a usual care, or control arm continuing the current 2011 duty hours, and an intervention arm that liberalized programs to a more flexible set of requirements closely resembling the 2003 regulations. Our study is especially relevant in light of this landmark trial, because our early findings demonstrate no improvement in patient outcomes after the onset of this policy, thereby supporting equipoise between the flexible and usual care arms of the FIRST trial. With regard to duty hour reform, there is significant concern that the scales have been tipped too far in favor of restrictions without clear benefit to patients and with significant drawbacks for trainees. If this is indeed true, policy changes to pursue less restrictive and more flexible regulations may help to strike the ideal balance between optimal patient outcomes and optimal education in surgery.
References
Philibert I.
Friedmann P.
Williams W.T.
ACGME Work Group on Resident Duty Hours. Accreditation Council for Graduate Medical Education
Resident duty hours in the learning and working environment: comparison of 2003 and 2011 standards. Accreditation Council for Graduate Medical Education,
Chicago2011
Effect of the 2011 vs 2003 duty hour regulation-compliant models on sleep duration, trainee education, and continuity of patient care among internal medicine house staff: a randomized trial.
Dr Scally is supported by a grant from the National Cancer Institute (5T32CA009672-23). This study was supported by a grant to Dr Dimick from the National Institute of Aging (R01AG039434). The views expressed herein do not necessarily represent the views of the United States Government.
Dr Dimick is a consultant and has an equity interest in ArborMetrix, Inc, which provides software and analytics for measuring hospital quality and efficiency. The company had no role in the study herein.
00364-5&id=gr1.jpg){kind=link}