Simulation-based Training Improves Physicians' Performance in Patient Care in High-stakes Clinical Setting of Cardiac Surgery

• ❖ Simulation training improves skills and management when crises are subsequently presented in a simulation environment, but whether this translates to real life situations is not known

• ❖ Senior anesthesia trainees who received simulation training for weaning from cardiopulmonary bypass performed better in this situation in real life than those who received traditional interactive seminars

AT some stage in their educational process, trainees apply their knowledge to treat patients in real-life clinical settings, before achieving full clinical competency. Patient safety may be compromised during this period.1Simulation technology has been advocated as a safer method for students to learn and practice skills in high-acuity scenarios without exposing real patients to the possibility of adverse events.2,3However, investigations into the effectiveness of simulation-based training for dynamic domains involving high stakes and invasive interventions such as in the fields of anesthesiology, surgery, critical care, and emergency medicine are limited. The unpredictable occurrence of medical crises in real patients makes many outcome studies unfeasible. To provide stronger evidence of support, a randomized controlled trial with assessment in the clinical setting is required to support simulation-based training for dynamic domains.

We have used weaning from cardiopulmonary bypass (CPB) as a model to translate the effects of simulation-based training into clinical practice and to evaluate the efficiency of simulation training for residency education. Weaning from CPB, despite occurring routinely at scheduled times, shares many similarities with “crisis management” scenarios. For example, it involves rapid task delegation, resource use, decision making, leadership, and communication with multiple players in a high-stakes environment.4 

During CPB, the heart is intentionally arrested. Thus, weaning from CPB requires routine management of and resuscitation from controlled cardiac arrests. Decisions made throughout this period are based on information from clinical assessment of hemodynamic monitors and interactions with the cardiac team. These sensory inputs can be realistically reproduced with modifications to existing high-fidelity simulator technology. Routine weaning from CPB involves a series of predictable technical and nontechnical clinical actions. Therefore, valid comparisons can be made both within and between trainees.

Our hypothesis was that simulation-based training is superior to traditional interactive seminars for achieving the clinical skills required for successful weaning of patients from CPB in the cardiac operating room.

The aim of this study was to compare two CPB teaching modalities: high-fidelity simulation-based training versus  a traditional interactive seminar format for learning of CPB weaning. The main and secondary outcome measures were the transfer of nontechnical and technical skills to real-life clinical practice, as measured by the Anesthesiologists' Nontechnical Skills (ANTS) and checklist scores, respectively.

Before data acquisition, we received ethics approval from the Research Ethics Board of St. Michael's Hospital (Toronto, Ontario, Canada). Written informed consent was obtained from patients and trainees participating in the study.

Twenty residents and fellows in anesthesiology, postgraduate year 4 or higher, volunteered to participate in the study. The trainees were randomized to one of the two intervention groups using sealed envelopes: (1) simulation-based education or (2) interactive seminar-based education. The exclusion criterion for trainees was experience in cardiac anesthesia, defined as participation in more than 10 operations involving CPB.

The outcome measures took place at 2 and 5 weeks after the intervention as the trainees weaned patients from CPB. Participants' performance was measured using a previously validated global assessment scale5and a checklist based on the syllabus. The study design is illustrated in figure 1.

To familiarize the trainees with cardiac anesthesia, each trainee received the usual residency program syllabus for cardiac surgery and CPB weaning. One week later, trainees had a pretest session in the operation room.

All CPB weaning sessions were supervised by a staff anesthesiologist (attending), with a subspecialty in cardiac anesthesia, who was blinded to the trainees' randomization. A separate blinded rater, trained in cardiac anesthesia, rated all trainees in real time. To determine interrater reliability,6a second blinded rater attended the clinical scenarios in random order according to his availability. He independently rated 26 CPB weanings.

Patients scheduled for elective coronary artery bypass grafting were included in the study. Exclusion criteria were surgery including noncoronary artery bypass grafting or non-CPB procedures and previous coronary artery bypass grafting because of a presumed increase in the complexity of the weaning process. The surgeons performing the coronary artery bypass grafting procedure were informed of the purpose of the study before the tests but were blinded to trainees' randomization. Surgeons were allowed to stop the session at any time based on the patient's clinical state.

Standard monitoring included invasive arterial and pulmonary artery catheters, a 5-lead electrocardiogram, and pulse oximetry. Standard anesthetics were used. Ten minutes before CPB weaning, the trainees were brought into the operating room and familiarized with the anesthesia setup. Trainees were briefed on the patient's history, physical condition, patient's baseline hemodynamic parameters, and intraoperative course. Common resuscitation medications, inotropes, vasopressors, and vasodilators were premixed in usual concentrations, labeled, and made available.

Direction for the separation from CPB came exclusively from the trainee in communication with the perfusionist and surgeon. There were no active interventions by the surgical team, other than to respond to direct questions from the trainee.

During CPB weaning, the attending anesthesiologist was expected to intervene under specified criteria: heart rate less than 50 or more than 100 beats per minute for more than 10 s; mean arterial pressure less than 55 or more than 100 mmHg for more than 10 s; mean pulmonary arterial pressure more than 25% or less than 25% of the baseline value for more than 10 s; persistent inadequate cardiac output (cardiac index <2 l · min⁻¹· m⁻²) for more than 60 s; severe arrhythmia (ventricular fibrillation or asystole) for more than 5 s; inadequate oxygenation (saturation < 92%) or ventilation (end-tidal CO²< 22 or > 40 mmHg) for more than 30 s; and alkalosis (pH > 7.5) or acidosis (pH < 7.2) where treatment was not immediately initiated. After the allotted time for any intervention criterion, trainees were prompted as to whether they were planning an intervention. If no intervention was planned and immediately executed, the attending anesthesiologist took over that task, and the rating on the checklist (table 1) was “not performed.” Trainees then continued with other tasks as required. The decision for intervention was made by the attending anesthesiologist not by the raters. The 10 s for intervention was a guideline for intervention. A stopwatch was not used.

Twenty patients were not included in the study after having given informed consent. They were all excluded by the operating team because of intraoperative issues and concerns about possible difficulties weaning the patient. These patients were replaced with additional 20 patients for the study. In these cases, the trainee's test was postponed to the next possible date within 1 week.

Each trainee randomized to simulation-based training attended an individual simulation training for a 2-h session, including debriefing. The trainee was first oriented to the simulation center. This orientation and familiarization phase included a description of the role of the various participants (surgeon and perfusionist). The trainee was reminded to manage the patient as he or she would in a real operating room and to verbalize all the observations and clinical findings during patient assessment, the initial and potential alternative diagnoses, and the management plan and treatment choices.

The simulation room was set up to closely mimic a cardiac operating room (high-fidelity simulation). All medications required for CPB were available, and intravenous infusions were prepared with common inotropes, vasopressors, and vasodilators identical to in the real operating room. The simulation room had a CPB machine that was handled by an actor perfusionist. A flat 17-inch LCD monitor was placed on the chest of the simulated patient (SimMan Universal Simulator; Laerdal, Wappingers Falls, NY) to loop video recordings of real patient hearts in different phases of CPB removal. All sessions were videotaped to facilitate debriefing and reinforce the learning objectives of the session.

At the end of each session, the trainee was debriefed by the educator at the simulation center with regard to his or her performance in nontechnical and technical skills. The educator was a staff anesthesiologist with long-term experience in both cardiac anesthesia and simulation-based training of residents. The debriefing included the following points: understanding the anesthesiologist's role for CPB weaning; surgical and anesthesia checklists before CPB weaning; recognizing and treating various scenarios during CPB weaning; verbalizing the initial diagnosis, potential alternative diagnoses, management plan, and treatment choices.

To help enhance the trainee's communication with surgeons, perfusionists, and nurses and to illustrate the most common types of weaning difficulties encountered in the operating room, four scenarios were created: low preload state, low afterload state, low cardiac index, and arrhythmias.

In scenario one, the trainee had to treat a low preload, visually underfilled heart, low blood pressure, and low cardiac index requiring volume administration. In the second scenario, the patient's heart showed low contractility, resulting in low cardiac output and requiring correction with inotropes. The third scenario was complicated by low systemic vascular resistance and a low cardiac index, requiring vasopressors and inotropes after correction of low preload. The fourth scenario added arrhythmias (supraventricular and ventricular) and conduction abnormalities to the previous conditions. The trainee had to defibrillate for ventricular fibrillation and initiate cardiac pacing for bradycardia afterward.

Each trainee attended an individual 2-h interactive training seminar. A staff anesthesiologist, experienced in both cardiac anesthesiology and resident training, lectured the seminar. The aim was to provide best-practice teaching. The seminar included audiovisual aids such as PowerPoint (Microsoft, Redmond, WA) slides, handouts, and face-to-face discussion of four paper-based scenarios similar to those described for simulation training. The learning objectives of the seminar covered the same content domains as the debriefing for the simulation group. Trainees of both groups had the possibility to discuss the syllabus with regard to the tasks assessed by the checklist. Trainees in both groups may have finished before the full 2 h allocated for training. If no further questions were asked, the trainees were allowed to leave their respective sessions before the full 2 h allocated for training.

Within 2 weeks (posttest) and 5 weeks (retention test) of completing training, trainees were asked to wean a patient from CPB in a real-life clinical setting. Trainees were not exposed to any cardiac anesthesia in the time period between the training and the two tests (posttest and retention test). The protocol for the posttest and retention test phases was identical to that of the pretest phase.

Previous studies have found global rating assessments to be both valid and reliable.7Therefore, we chose this tool as our primary outcome measure. A previously validated global assessment tool, ANTS scale (table 2), has established construct validity and reliability, and it was used to assess trainees' cognitive and behavioral performance in CPB weaning.5 

A checklist for CPB weaning was developed specifically for the purpose of this study to assess the technical skills of trainees (table 1). We used a Delphi method,8including one cardiac surgeon and four cardiac staff anesthesiologists from our department. To include a task on the checklist, 80% agreement had to be obtained through an iterative process.

We hypothesized that there would be a difference in the ability to wean patients from CPB in favor of the simulation-based training group. In the field of psychology, an effect size of more than 1 SD is considered a large and acceptable difference for assessing teaching interventions.9We agreed that an effect size of 1 SD would be required to demonstrate a statistically significant and practical teaching advantage for simulation-based training over traditional seminar instruction. To calculate our sample size, we expected this intervention to have its largest effect in the posttest phase. From this, we estimated that we needed at least 17 trainees per arm based on a two-tailed alpha of 0.05 and a power of 0.8. An interim analysis was planned for when 10 trainees in each group had completed all clinical assessments.10 

An interclass correlation test was used to establish the level of interobserver correlation between the two expert evaluators for both the ANTS scale and the checklist. ANTS were analyzed according to the global rating score (the sum of the four rating category level subscores). Therefore, the maximum possible score was 16 and the minimum 4.

To analyze data from the checklist, tasks performed independently were scored 2 points, tasks performed after prompting were scored 1 point, and not-done tasks received 0 points. If the task did not have to be performed, then it was not counted. The percentage of total points possible from the checklist was then analyzed.

Separate mixed-design ANOVA, with the group (simulation or seminar) as the between-trainee variable and test phase (baseline [pretest], posttest, and retention test) as the within subject variable, were conducted on the ANTS and checklist scores. Post hoc  analyses were conducted using independent samples t  tests for each time, with Bonferroni corrections for multiple comparisons. Independent samples t  tests were conducted to measure the differences between the two groups in trainees' ages, years of clinical training in anesthesiology, number of previous simulation sessions, and amount of clinical experience in CPB weaning; and in patients' ages, left ventricular function, and number of grafts performed during surgery. For all statistical tests, a two-tailed P  value of < 0.05 was considered significant. Statistics were run using SPSS version 17.0 (SPSS Inc., Chicago IL). The data are reported as mean and standard error.

No significant differences were observed between the two groups in terms of any of the trainee or patient variables. ANTS and checklist scores at pretest did not correlate with trainees' age, training level, previous simulation sessions, or previous clinical experience in CPB weaning. The interim analysis after 20 trainees revealed highly significant differences in the performances of the two groups' in CPB weaning of real patients. After weighing the benefits of continuing the study, we decided to stop recruitment after a total of 20 trainees and 60 patients.10 

A total of 20 assessments had to be postponed because of excluded patients. In the seminar group 5 and 3, trainees were postponed for the posttest and retention test, respectively. In the simulation group 7 and 5, trainees had to be postponed for the posttest and retention test, respectively. Based on 26 tests scored by two raters, the intraclass correlation coefficient between the scores was 0.95 for the ANTS scale and 0.67 for the checklist.

Both groups improved from the pretest to posttest to retention test as indicated by the magnitude of the group means. The significant group by time interaction effect suggests that the trend for improvement differs in magnitude between the simulation-based training group and the seminar group (F(2,36) = 11.9, mean standard error [MSE]= 0.47, P < 0.001). Overall, the simulation group scored higher than the seminar group as indicated by the main effect of group (F(1,18) = 11.5, MSE = 4.55, P = 0.003). Although both groups improved, the simulation group showed significantly higher improvements compared with the seminar group, as indicated by significant group by time interaction (F(2,36) = 11.9, MSE = 0.47, P < 0.001). Post hoc  t  tests revealed similar pretest ANTS scores for the two groups (simulation group 10.6 ± 0.46 vs.  seminar group 10.0 ± 0.46; t(18) =−0.999, P = 0.331). After training, however, the simulation group significantly outperformed the seminar group in both the posttest (14.3 ± 0.41 vs.  11.8 ± 0.41; t(18) =−4.280, P < 0.001) and the retention test (14.1 ± 0.41 vs.  11.7 ± 0.41; t(18) =−4.249, P < 0.001) phases (fig. 2). All components of the ANTS, such as “situation awareness,”“team working,”“decision making,” and “task management” were significantly different between the two groups at the posttest and the retention test (table 3).

For the checklist data, the ANOVA revealed a main effect of time (F[2.36]= 33.38, MSE 0.011, P < 0.001) and group (F[1.18]= 9.94, MSE = 0.020, P = 0.005). Similar patterns of improvements were indicated by a nonsignificant time by group interaction (F[2.36]= 1.86, MSE = 0.011, P = 0.170). Post hoc  independent samples t  test revealed equivalent pretest checklist scores between the two groups (simulation group 62.6 ± 5.3% vs.  seminar group 58.3 ± 5.3%; t(18) =−0.578, P = 0.571). However, the simulation group scored significantly higher than the seminar group during both the posttest (89.9 ± 3.0% vs.  75.4 ± 3.0%; t(18) =−3.365, P = 0.003) and the retention test (93.2 ± 2.4% vs.  77.0 ± 2.4%; t(18) =−4.836, P < 0.001) phases (fig. 3).

Our study demonstrated that the skills required to wean a real patient from CPB, a highly standardized but potentially stressful situation for physician trainees, can be acquired through simulation-based training. In addition, compared with traditional interactive seminars, simulation-based training was superior in achieving clinical skills needed for weaning a patient from CPB.

We evaluated the following during the CPB weaning of 60 patients: performance in patient care based on the performance of (1) nontechnical skills, as measured by the ANTS global rating scale; and (2) technical or procedural skills, as measured by the checklist. Both interventions—simulation-based training and interactive seminar—educated trainees in the same medical content domains. Both groups received a syllabus, which the checklist was based on, before the pretest assessment. Therefore, we can be confident that the results were not because of differences in core medical knowledge achieved by either group. A few studies have shown the transfer of procedural technical skills acquired during simulation-based studies into real-life performance11–13; however, our study is the first to also show the effective transfer of nontechnical skills.

Simulation-based education has been studied and evaluated as a tool for resident training since the 1960s.14,15In the 1980s, a new strategy in simulation-based education, high-fidelity simulation (i.e. , the recreation of a full operating room), entered the educational literature.16The first study to demonstrate the improvements in performance as a result of simulation-based training, measured by improved ratings on a scoring system, was published in 1994.17However, as with most subsequent studies, outcomes were measured on the simulator and not in real life.18More recently, the transfer of technical skills to the operating room has been validated.12,13,19Two recent retrospective studies found that simulation-based education improved the quality of care during cardiac arrest team responses at an academic teaching hospital and during neonatal resuscitation.20,21The results of our study add further evidence in support of simulation-based education becoming a routine part of a health professional's education.22 

Human error has been shown to be a major contributor to adverse events.23–25Nontechnical skills have been identified that may reduce human error and allow the successful management of high-acuity events in health care.4,25–31Immersive training in nontechnical skills during the simulation scenario might be superior to passive discussions in interactive seminars. High-fidelity simulation that creates a surrounding mimicking reality may enhance behaviors during real-life scenarios and, therefore, tests clinical judgment. In the literature, it has been referred as “immersive simulation.”32 

Technical or in other words “procedural” skills—which mainly influence the transfer of best-practice adherence—are critical for patient care. The results described here show improved translation of the technical skills achieved during simulation-based training compared with those achieved via  an interactive seminar. This improvement in skills might be explained by the effect of immersive training. Recent literature suggests that for guidelines adherence, low-fidelity simulation might be as efficient as high-fidelity simulation.33However, lowest-fidelity simulation might be closer in nature to interactive seminars, and the educational literature states that the probability and efficiency of retrieving an item from memory depend on the similarity between the conditions of encoding and the conditions of retrieval (contextual learning).34 

Simulation-based training might also be influenced by a structured summary debriefing after the simulation-based training session. Previous studies support the importance of a structured debriefing for improving the learning effect of simulation-based training.35,36In addition, such debriefing has been found to be superior to concurrent feedback during the teaching of clinical technical skills.37Debriefing also allows the teaching contents to be adapted to the personal performance of each trainee by the educator. In contrast, the educator in an interactive seminar does not receive such information on personal performance. In the literature, the importance of professional personal feedback on practical performance to ensure quality in patient care is also referred as “deliberate practice.”38 

Neither group in our study improved from the posttest to retention test phases. However, during the real-life part of this study, we purposely attempted not to teach the trainees to maximize the effects of our intervention to be able to draw conclusions. The trainees only received concurrent feedback in the form of prompting or taking-over of tasks by supervising staff. Explanations were limited as far as possible. Because this is not best teaching practice, all trainees were debriefed after the retention test, with the aim of improving their skills.

Trainees were rated in the operating room. Although the second rater was instructed to rate independently, there may have been nonverbal communication between the raters during the session. There may have been communications between the two raters as they were physically in the same operating room.

Interim analyses have been criticized, especially if the trial is sponsored by the industry; however, if differences are high, ethical reasons can justify this practice.39,40An interim analysis was planned a priori  for two main reasons: first, to avoid depriving some of our trainees of an efficient educational intervention; and second, to minimize unnecessary patient exposure to potentially inadequately trained residents and fellows.10,40We, also, did not adjust our P  value to account for interim analysis. However, our P  values for our main scale, ANTS, were sufficiently small to justify our conclusions.

The ANTS scale was not specifically taught to either group, and the importance of nontechnical (behavioral) skills was addressed in the training of both groups. However, only the simulation group was able to reflect on their own performance, and this may have led to learning. Depending on one's point of view, this may be seen as biasing toward the simulation group or taking advantage of simulation to improve transfer of nontechnical skills to the clinical settings. However, this might also be a major advantage of simulation-based training.

We attempted to separate technical and nontechnical skills. However, our phrase of “technical skills” may not be representative of what we were testing. Cognitive knowledge and behavioral skills were required to be marked higher on the checklist score. Therefore, the checklist may also have been measuring nontechnical skills. Also, the same raters rated both ANTS and checklist scales. Haloing probably occurred as the checklists scores may have been affected by ANTS scores.

High-fidelity simulation-based training leads to improved patient care during CPB weaning when compared with interactive seminars. To our knowledge, this is the first prospective randomized controlled trial to show the superior effect of simulation-based training with direct translation into clinical practice. Although we specifically looked at CPB weaning, many similarities exist with other crisis events such as cardiac arrest and advanced cardiac life support. The results of this study suggest that patient care during these crisis events may also be improved through simulation training. As high-fidelity simulation improves toward immersion simulation, rare types of crisis may also be replicated in the simulation setting. This will hopefully spare patients from potential harm as trainees will have honed their skills before “learning on the job.”

The authors thank Jeff Dickson, M.D., Lecturer, St. Michael's Hospital, University of Toronto, Toronto, Ontario, Canada, for his assessment as the second rater. They also thank Catherine Booth, Editor's Association of Canada, Toronto, Ontario, Canada (www.editors.ca), for editing the final draft.