Residency program models, implications, and evaluation: Results of a think tank consortium on resident work hours☆☆☆★★★

Critical components of the ideal residency program 

The teams identified a number of components of residency programs thought to be crucial from the 5 assigned stakeholder perspectives (Table I).

Table I. Key attributes of an ideal general surgery residency program from various stakeholder perspectives

	Stakeholder	Key attributes
	Patient	Opportunity to get to know resident involved in their care
		Clear communications between patient and those involved with care
		Warm, compassionate, high quality care
		Understand role of resident
		Clear that resident is being adequately supervised by attending physician
		Motivated and talented residents
	Faculty member	Well-read residents fully prepared for patient care responsibilities
		Residents who know and care about their patients
		Involved in preoperative, intraoperative, and postoperative care of a patient
		Available for important learning opportunities
		Program in complete compliance with RRC regulations
		Truly graded and appropriately supervised independence
	Resident	Dedicated and talented teachers and role models
		Adequate salaries
		Scheduling flexibility
		Time to exercise, access to healthy foods, and some balance in life
		Clear roles, expectations, and responsibilities
		Opportunities for established relationships with patients and faculty
		Availability of residents when help was needed
	Other health care providers (RNs, etc)	Clear communication systems
		Well-understood roles within multidisciplinary care team
		Continuity of care
		Increased nurse autonomy
		Sense of team with clear responsibilities and mutual respect
		Attract the best and brightest applicants
		Availability of residents for around-the-clock care
	Hospital administrator	Clear communications among team members
		Warm, compassionate, high quality care
		Appropriate supervision of residents
		Residents well prepared to care for patients
		Resident education should be cost neutral

Taking the perspective of other health professionals who work with residents (nurses, physician assistants, pharmacists, social workers), group 2 thought that an ideal residency program had ready availability of residents to provide appropriate help when needed, as well as a clear system for knowing whom to contact for patient care needs. Ideally, there should be a well-understood chain of command for people to be able to work together effectively, with each team member having clearly defined roles and expectations. Other identified elements of an ideal program included patient continuity of care, residents as patient advocates, and a sense of team. In the ideal program, the group suggested that a multidisciplinary team would make rounds. Face-to-face conversations on the plan of care would occur on a regular basis. In the ideal program, nurses would have more autonomy to make the decisions they were trained to make, so that residents would be relieved of duties that could be carried out effectively by nurses. In such a system, nurses would have well-defined roles and expectations. In general, an ideal program valued a culture of professionalism in which each member of the team was respected and had clearly defined responsibilities and would be characterized by coordinated and consistent communication between disciplines.

The group taking the perspective of hospital administrators believed an ideal program would attract the best and brightest resident applicants. The availability of residents was believed to be an advantage of teaching hospitals in the marketplace, so around-the-clock, in-house care by residents was a desirable feature of the ideal program. Also from the hospital administrators' point of view, changes in residency education would be cost neutral. An ideal program would include a true team approach to patient care that effectively and efficiently involved others in the care of the patient. Administrators would value the use of critical pathways that improve system efficiencies and standardize care.

The ideal program from a resident's perspective would include truly graded independence, dedicated and talented teachers and role models, adequate salaries, and some flexibility in scheduling. Noneducational activities would be minimized if not eradicated. This group highlighted the need for continuity and coordination of patient care, because relationships with patients were too compromised without it. In the ideal program, residents would have time to exercise, have access to healthy foods throughout the day and night, and have some semblance of a balanced lifestyle. The program would have an effective system for transferring care and the opportunity for residents to operate independently when ready but with assistance as needed. The ideal program would avoid conflicts associated with unclear roles and expectations and pager abuse. The program would promote opportunities for established relationships with patients and help to develop comfort and confidence when treating patients.

The ideal program from a patients' perspective required that they have the opportunity to know the resident involved in their care, preferably from the preoperative through postoperative stages. It was believed they wanted efficient care that also included clear communications among those involved with no missed signals or surprises. This team thought that patients would want warm, compassionate, high quality care with their resident and attending physician available. They would want to understand the role of their resident, and that he or she would be appropriately supervised and adequately prepared to participate in their care.

Continuity of care, efficiency, clearly understood roles and expectations, and systematic communications were common components among all stakeholders. Meaningful interpersonal relationships between residents and other health professionals, faculty, patients, and peers were also highly valued and thought to be critical both to the integrity of the residency program and for the satisfaction of all stakeholders.

Re-engineering resident responsibilities 

When asked which resident activities should be preserved, eliminated, replaced, done earlier, or otherwise re-engineered, most teams were in agreement (Table II).

Table II. Suggested disposition of various resident tasks

	Eliminate or replace	Pre-rounding
		Moonlighting
		First-assisting or observing in the operating room
		Routine admission and discharge paperwork and summaries
		Routine preoperative histories and physicals, particularly when the resident will not be scrubbed on the operation
		Most ward procedures (eg, blood draws)
		Most administrative activities
		“Scut” work (eg, patient transport)
	Re-engineering	Night call (increased home call, decreased cross-coverage)
		Nurses' authority/autonomy (increased use of patient care protocols)
		Patient handovers
		RRC defined category minimums, as well as time in secondary components
		Teaching conferences
		Pager use guidelines
	Do earlier	Some ward procedures (eg, intravenous insertions, nasogastric tubes)
		Basic surgical skills
	Do later	Routine consultations
	Maintain	Morning rounds
		Operative cases as surgeon and teaching assistant
		Clinic/office attendance for new and selected established patients
		Research blocks for 1 year or more
		Ward-based teaching of junior residents and medical students
		Effective and efficient transfer of care between resident teams
		Personal time (time for adequate sleep, exercise, etc)

Those activities thought to be unnecessary because they had little or no educational value included pre-rounding, moonlighting, research blocks of less than 1 year, first-assisting or observing in the operating room on procedures, routine admission and preoperative histories and physicals, discharge paperwork including discharge summaries, most ward procedures (blood draws), and most administrative activities. All “scut” work (ie, patient transport) was considered expendable.

It was suggested that 2 activities could be done earlier to be more efficient. Several ward procedures (intravenous insertions, nasogastric tubes) could become the responsibility of medical students when these skills are part of their clerkship curriculum. Second, it was believed that residents' learning and mastering basic surgical skills could be accomplished more efficiently if done earlier in the residency within a skills laboratory environment. Given the cost of the operating room and patient and surgeon expectations, residents could initiate learning of suturing, knot tying, instrument selection and handling, and other skills in a skills laboratory. Although models only exist for certain procedures and development of more models is needed, the group hypothesized that a skills laboratory could become an important component of a competency-based program of graduated experience in training in some areas. The evolution of new models and technology would play an even greater role in acquisition of motor skills. It was thought that these activities could effectively replace first-assisting and scrubbing as an observer in operative cases in the early years of training.

Several activities fell into the category of needing reconsideration and re-engineering. These included night call (most groups advocated an increase in home call, a decrease in cross-coverage, and an increase in nurse autonomy). There was agreement that the process of patient handovers needs to be formalized. The groups believed that the defined category minimums of the residency review committee (RRC) should be reexamined, as well as time spent in secondary component areas and subspecialties (cardiothoracic, orthopedics, urology, anesthesia, etc). Participants agreed that pager abuse occurs because residents are always available and easily accessible. It was recommended that nursing staff, if given clearer patient protocols and guidelines for when it is appropriate and not appropriate to page, would reduce substantially the number of unnecessary pages that disrupt sleep or other work. Teaching conferences could also be re-engineered by translating current lecture style conferences into Web-based or other electronic learning mediums so residents could have the flexibility that distance learning affords. This approach would reduce the need for all residents to attend day lectures, given that not all residents will be in the hospital at the same time.

Activities that could be accomplished later included routine consultations. Many consults called to surgical residents in the evening and at night are not urgent and could be seen the next day or even after discharge in the ambulatory setting.

Ideas for enhancing resident activities 

Several ideas were generated on how residency education could be enhanced either now or in the future by using developments from other fields or professions. The results of this advanced brainstorming exercise elicited a few ideas that might be considered farfetched. But it is not uncommon for this type of metaphoric thinking to give way to new inventions or ideas.

Residency activities considered included documentation, communication, learning and supervision, and performance feedback and evaluation. Ideas to enhance documentation included the use of “swipe”-able bar codes bracelets for patients so that residents could immediately identify the patient's location in the hospital by computer rather than having to use the telephone. Some also suggested residents have bar-coded badges to document duty hours by “clocking” in or out. Software for electronic patient records could be modified to generate medication lists and prescriptions at the time of discharge. Voice-activated software, already used in the military and other fields, would enhance efficiencies in charting write-ups, postoperative notes, and clinic notes.

Communication was an area thought to need enhancement, and analogies were drawn from coaching in which coaches wearing headsets and players using earphones could have verbal exchanges more readily. Attendees discussed the idea of residents wearing headsets, so that they would not have to use the telephone to communicate with other physicians and nurses, communicate with various locations in the hospital, and answer consults. Advanced wireless technology support could make rounds more efficient with portable computers for order entry; each patient's room should have a computer that can generate customized real-time displays of vital signs, laboratory results, and other pertinent information for rounds. Technology seemed to be the most common solution to resolve communication problems; clearly there is room for development in this area.

Topics on learning and supervision yielded several ideas. On-line “homework” with progressive documentation of successfully completed units could replace some traditional conferences or lectures. Surgical faculty, as in most other university departments, would undergo instruction on how to maximally utilize this technology. The Advanced Trauma Life Support instructional program administered by the American College of Surgeons was thought to be a good example of how we could teach residents various skills more efficiently. Stations could be developed, and residents could progress through these instructional experiences for both learning and assessment. Having “black box” technology in the operating room and wards similar to what is available in airliners would be helpful for recording data, events, and problems. Faculty and residents could then review and reflect on the procedures with the aim to either learn from any errors or to discuss how the procedure could have been done better, even if no errors were made. Regional skills centers, just as there are various Centers of Excellence, could conserve resources, given the cost of simulators and other models, and yet provide access for medical students, residents, and faculty to learn and to hone various skill sets. Feedback and evaluation were an area also believed in need of enhancement. Scorecards, similar to those that reflect players' statistics and as used widely in industry, were suggested as an improved means to provide feedback and evaluative information to learners so they know where they stand in comparison to others at their level throughout their residency program. A scorecard approach would be particularly valuable in documenting progress toward the ACGME-mandated competencies.

This exercise resulted in several other discussions that highlighted the inefficiencies in our current system of patient care and residency education. Technology was the most frequently cited solution and will likely evolve as we continue to strive to educate our residents while ensuring high quality patient outcomes.

Residency models 

The teams described 4 models of residency training that we ultimately named the Apprentice Model, the Mastery or Case-Based Model, the Night Float Model, and the Stretch Model (Table III).

Table III. Key features of 4 residency models

	Apprenticeship Model	Residents work side by side with mentor(s)
	One resident works with 1 or 2 faculty members over a sustained period of time	Faculty carefully selected for mentor role
		Take home call with mentor(s)
		Requires some (minimal) in-house presence
		Early intensive technical skill training
	Mastery (Case-Based) Model
	Residents' proficiency knowledge and skills associated with predefined diseases/operations paced by personal progress	Cases assigned according to learning needs
		Proficiency verified through formal assessment and then allowed to move on
		Mentor paired to residents to oversee progress
		Weekly meeting to assign residents to patients and attending staff based on learning needs
		Specific expectations are clear at start and mastery-based but broken down by years for planning purposes
	Night Float Model	Several teams work day shift on weekdays
	More traditional team system with residents designated to work the night shift	One team works night shift on weekdays
		Weekday day shift rotate weekend coverage
		Chiefs do not night float but cover from home
		Separate ICU rotations
		Team-specific clinic/office day
		Case distribution overseer
	Stretch Model	Traditional teams
	Traditional model with q4 and out the door	Junior residents on call every 4th night
		Residents leave hospital after night call completed
		Senior residents rotate call every x night depending on number of senior residents

The Apprentice Model is defined as 1 resident working exclusively with 1 or 2 faculty members during a sustained period (months). In this model, residents work side by side with their assigned mentors in the operating room and outpatient office. Residents would take home call when their mentor is on call and return to the hospital when their mentors are called for a consult or emergency. Residents are involved only in the care of their mentors' patients. Faculty members would need to be selected carefully on the basis of dedication to education and an appropriate practice profile. If junior residents participate in an apprentice model, early intensive technical skill training would be required, preferably in a skills laboratory, so they are equipped to work effectively with their mentors in the operating room. It would be a clear expectation that mentors allow residents to perform cases suitable to their level of training with the mentor assisting.

The Apprentice Model would require some investment of resources to provide assistance to staff physicians who are not assigned a resident. It would require some in-house presence of other health professionals or a resident team at night so that the resident would not always be on call for his/her patients but rather could take call in rotation with the attending mentor. The advantages of this model are that the model provides continuity of care for patients and allows the resident to participate in care across the continuum from preoperative evaluation through surgery to postoperative follow-up. This model has the strong advantage of mimicking the real life of a surgeon. It promotes a depth of interaction between the resident and their mentors as well as with the patients. Learning can be tailored to the resident's learning needs, and careful evaluation and feedback could be given, because the extent and scope of direct observations would be maximized. It reduces the need for cross-coverage and provides residents with early experiences in the operating room. One strong advantage of this model is that it gives program directors the opportunity to assign residents according to the quality of the teaching experience.

The disadvantages of the Apprentice Model are that not all faculty members would be assigned a resident, leading to possible divisiveness. Faculty with limited clinical practices who are good teachers might not be afforded teaching opportunities because of their practice profile. There would be scheduling challenges that would need to be overcome. The system would need to be flexible in case there were serious personality clashes or other problems between a mentor and his/her mentee so reassignments could be accomplished. This model is somewhat resource intensive, because physician assistants or other health professionals would need to be available to assist those faculty members without mentees. Last, a sense of a resident team (camaraderie) could be compromised with this model.

The Mastery (Case-Based) Model was defined as a system in which patient cases are assigned to residents on the basis of the learning needs of the individual residents irrespective of the attending staff or team assignments. Proficiency, knowledge, and skills associated with diseases and operations are measured by personal progress, not by time. Proficiency is verified through formal assessment, after which residents are allowed to move on to other areas and, just as importantly, not required to scrub on operations they have mastered unless they feel the need to refresh their knowledge. An adviser would need to be paired with each resident to oversee the resident's progress by using performance portfolios. In such a system, participating residents would meet each week to receive their final patient/attending assignments for the coming week based on preoperative, intraoperative, and postoperative needs of that specific resident. Residents would be responsible for making arrangements to review the cases with the appropriate attending physician. Residents would round on their own patients in the morning and go to the clinic or operating room depending on their assignments for that week. Residents would then follow all of their assigned/operated patients, irrespective of attending or service staff. There would be an outpatient clinic block, which would probably have to be attending-based, because it would be difficult for residents to follow up on their patients in multiple ambulatory offices. It would be critical that learning expectations are made clear at the start to be mastery-based but broken down by year of training for planning purposes.

The advantage of the Mastery (Case-based) Model is that it reflects principles of adult learning in that residents assume more responsibility for their own learning and development. It is based on attainment of competence rather than an accumulation of weeks, months, or years of experience. It decreases time spent in noneducational activities such as participating in too many of any one operation. Once basic competencies are met, the clinical experiences can potentially be tailored to individual residents' needs and interests. If such a system were adopted widely, aggregate performance data could inform the RRC on the number of cases needed by residents to achieve proficiency. On the other hand, several disadvantages to this model must be kept in mind. This model would markedly complicate communications between attending staff and residents and would require sophisticated scheduling methods. The entire surgical care system would need to be less dependent on resident services than is currently the case, and faculty would undoubtedly have to assume more responsibilities for their patients' care. It would also require a more robust performance evaluation system than what currently exists.

The Night Float Model consists of a traditional resident team system, except that a percentage of the program's total residents are designated to work a permanent night shift, usually for a month at a time. Residents assigned to nights would leave in the early morning after handing over patients to the day team. Several teams would work the day shift, with those shifts having the same or staggered start times. Teams working during the day would leave in the evening and take no in-house night call. The “night float” team would work a night shift for a 1-month period 5 or 6 days per week. The chief residents would ordinarily not work on the night float team but rather cover services from home, although the exact configuration would depend on the hospital's requirement for emergency/trauma coverage. The day shift teams would include a mixture of junior, senior, and chief residents, and a combination of these from various day shift teams would share weekend call and cover the night or two not covered by the float pool. Night float members would not be allowed to schedule vacations during the 1-month periods of night float duty. The strengths of this model are that the model is not particularly disruptive compared to the current system, that workloads are shared, and that the resident team concept is maintained. Residents would still assume their teaching roles, and clear lines of responsibility are easily drawn. But this model may not work for smaller residency programs, and it could potentially dilute the operating room experiences for junior residents. It may require instituting nonteaching services, because the effective resident day pool is reduced, and could require eliminating some or all elective rotation experiences to free up persons to cover the float pool. Other surgery programs have described their programs' approaches to using a night float team.5, 6

The final model, the Stretch Model, is the most traditional model, which we nicknamed “q4 and out the door.” In this model, residents would take call every fourth night and leave the next morning after call (up to 6 hours are allowed for transition of care). The advantages of this model are that it is the easiest to implement and has the least amount of impact on faculty. It preserves the resident team concept and continues residents in their roles as teachers. Its biggest disadvantage is increased cross-coverage at night, potentially “stretching” the residents beyond reasonable patient loads. It can impact operative experience negatively, because one fourth of residents must go home during the day. It could eliminate some or all specialty experiences or require nonteaching services, because the pool of residents available during the day is reduced.

Although the extent of resources required for implementation varies from model to model, additional resources will be needed for all of them. Unless additional residents are added to surgical programs, other employees will be required to do the work typically done by residents that will no longer be done given the 80 duty hour maximums. Nurse autonomy, perhaps increased nurse staffing, and clinical pathways and protocols for common problems will need to be accentuated. Enhanced information technology and communication systems are needed to make resident work more efficient and patient information exchanges more consistent. Faculty will need to hone their skills as teachers, evaluators, and providers of performance feedback. Refined standards and measures for evaluating proficiency will become more important. Early intensive technical skills training will aid in helping residents function and learn more efficiently in the operating room. This concept of skills training will be important, because the operative experience for residents will have to be spread more evenly over the residency than currently happens in many programs. Plans for trauma and ICU coverage will need consideration, because the above models may not be ideal in those settings. Having an apartment or other housing accommodations near the hospital might make residents and faculty members more easily available for certain types of call but still “at home” and hopefully more comfortable than in a call room. Last, additional or alternative delivery systems to replace traditional conferences are needed so residents can study and learn in the evenings and commit their time during the day to learning from direct patient care activities.

Evaluation research 

It will be important for the surgical profession to be in a position to assess the impact of changes in resident work hours. A multilevel evaluation plan was suggested. The identified variables needing to be evaluated are organized according to Kirkpatrick's taxonomy (Reaction Evaluation, Evaluation of Learning, Behavioral Changes, Impact Evaluation) for evaluating educational programs plus 2 components (Input Evaluation and Process Evaluation) from Chronbach's model of program evaluation.7, 8

Reaction evaluation that focuses on perception and opinion data would be solicited from all stakeholders. It will be important to learn from resident and faculty about their overall satisfaction with the program and their role in it. Also critical will be studying their opinions of perceived adequacy of their program's curriculum, as well as their quality of life, their sense of confidence in their knowledge and skills, and the extent of compliance of both the individuals and the overall program. It would also be interesting to learn whether they perceive differences in residents' work ethic and commitment to patient care. Patients' reactions could be studied as to their understanding of the resident's role, satisfaction with care delivered by the resident, and the extent of relationship built between the patient and resident. Reactions of members of the health care team could be evaluated as to their perception of the residents' roles, their attitude toward working with the residents, and the strengths and weaknesses of the new program model as it relates to patient care. Other reactions that would be worth studying include those of medical students, peers, spouses and families, and hospital administrators.

Evaluation of learning could include tracking of the scores on the American Board of Surgery In-Training Examination (ABSITE) and departmentally administered exercises such as mock oral examinations or performance-based examinations. Other measures should also be tracked, including performance ratings from faculty or other sources, as well as scores from the American Board of Surgery Qualifying and Certifying Examinations. Reviewing scores of residents working in a new residency format and comparing those scores in the traditional system would help faculty determine whether the “product” (residents' ability levels) are altered positively or negatively by the changes in the residency program and perhaps identify areas of strengths and weakness that need to be addressed.

Studies are needed to explore the extent to which modifications in the structure of residency programs change people's behaviors. Variables identified as potentially worthy of study included reading and study habits, time spent in various patient care and educational activities, the extent of preparedness for operations, ward and clinic responsibilities, and resident efficiency. The frequency of moonlighting, the use of clinical protocols, attendance at conferences or skill laboratories, and the number of people sleeping during conferences were also mentioned as possible outcome variables for study. The method and frequency of resident communication with attending staff, patients, etc were suggested as an important behavior to evaluate.

Research variables associated with Impact Evaluation identified for possible study included faculty and resident retention rates, number and gender of applicants, number of residents completing research years or advanced degrees, number of residency programs closed, amount and types of new technology introduced, and changes in patient care systems. Other variables mentioned included data on divorce rates, marriage and birth rates, and lifestyle variables hypothesized to be impacted by infrastructure changes.

Input Evaluation reflects resources used to implement and to maintain the changes in an educational program. Variables considered important included dollars invested, number of new staff hired to administer the program, types and nature of technologies introduced to the new program, and facilities or equipment required for implementation. The number and type of health care professionals hired and associated costs would also be an important specific variable to track.

Process Evaluation involves collecting descriptive data that outline specific changes in the educational program. Variables included specific modifications made to schedules, conferences, and other resident activities. Documenting who was involved in planning and implementing the changes, how the changes were implemented, and how the plan for change was communicated would be part of process evaluation. Indications of what would be done the same or differently after implementation of a new model would be included as well as descriptions of actual versus intended changes. Process evaluation is critical so that programs considering similar changes can learn from others' experiences.

Several sources for the type of outcome evaluation data listed above already exist. Sources include data from the RRC, Joint Commission on Accreditation of Healthcare Organizations, Association of American Medical Colleges (ie, Graduation Questionnaire), morbidity and mortality conference, ABSITE scores, and other performance and program evaluation measures already collected by programs as well as data collected by the various governing bodies. Each program will likely want to have their own evaluation plan, but regional, national, and international societies should also develop blueprint plans for evaluating the impact of changes in residency education programs so large scale studies can be completed that are generalizable.

Attendees emphasized that evaluation studies should be done with scientific rigor. Dissemination of results should be widespread when findings can aid other surgical specialties or medical disciplines.