Abstract

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Anesthesiology has a long history of developing and implementing innovations in education, and most anesthesiologists are educators in some capacity. Anesthesiologists teach fellows, residents, and medical students. They also educate patients, peers, and colleagues in anesthesiology and other disciplines. Despite the pervasive nature of teaching and learning in this specialty, few anesthesiologists have formal training in how to effectively educate. Many anesthesiologists often teach the way they were taught and learn the way they learned in the past. Applying current evidence in education is likely to change the way many teach and learn.1 

Accordingly, in this review, we discuss current evidence in education, provide recommendations of how the evidence might be applied to teaching and learning in anesthesiology, and suggest areas of future educational research. More specifically, educators across multiple disciplines have moved away from passive, lecture-based presentations to more active learning through techniques such as the flipped classroom, peer teaching, and educational games. In order to improve actual knowledge retention, as opposed to mass learning or blocking, evidence suggests that the more effective techniques may be spaced learning, interleaving, and retrieval practice. To combat noncompliance due to knowledge and memory errors in clinical crises, cognitive aids have proven to be a critical component of anesthesiology training. E-learning provides platforms to enable flexible utilization of these knowledge acquisition, retention, and assistance techniques on a daily basis to optimize learning. In addition to the acquisition and retention of knowledge, practicing anesthesiology also entails manual dexterity and workflow management. Thus, the techniques, theory, and metrics for teaching procedural skills are of the utmost importance. Additionally, teaching and assessing clinical performance and professionalism through role modeling and role play are becoming a more recognized need in our field. Finally, experiential learning through simulation and objective structured clinical examinations is becoming more prominent in the form of both education and summative assessment.

Although this review is organized based on the teaching of (1) knowledge, (2) skills, and (3) professionalism, we recognize there is substantial overlap between these topics. This review is meant to be used as a primer on educational methods in anesthesiology for any clinician who aspires to be a better teacher. It is written in a module-based format so that the reader can easily navigate through the review and integrate the material. Figure 1 provides a tool that can be used to guide the instructional needs of your learners. For those interested in a specific topic, the references included in this review may serve as a more detailed resource.

Fig. 1.

Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.

Fig. 1.

Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.

Teaching Knowledge

One goal in anesthesiology education is to help learners acquire, retain, recall, and apply knowledge (table 1). The volume of medical knowledge continues to rapidly expand. Unfortunately, without repetition, knowledge is forgotten quickly after learning. Implementation of the evidence from cognitive science and healthcare education research offers the promise of making lessons “stickier” with improved knowledge retention.2–7 

Table 1.

Educational Methodologies Based on Domains of Utilization

Educational Methodologies Based on Domains of Utilization
Educational Methodologies Based on Domains of Utilization

Active Learning

Medical education is frequently delivered passively through teacher-centered delivery of lectures. Learners memorize facts, recite them for examinations, and then often forget the majority of the information. Active learning describes an approach to learning, not a singular educational method.8,9  This approach involves learner-centered education in which learners apply knowledge and solve problems in the classroom. Although active learning has been demonstrated to be effective in many areas of science education,8,10,11  more work needs to be done to determine its benefit in graduate medical education.

There is a variety of active learning methodologies, which all share some basic principles. First, control shifts from teacher to learner as the teacher takes on the role of a facilitator or coach, and the students assume ownership for their education. Another key component is that learners actively participate during classroom time by applying knowledge they are learning through higher-order thinking and problem solving. One advanced model of active learning is the Master Adaptive Learner. In this model, learners work actively through an iterative cycle of planning, learning, assessing, and adjusting as they master concepts or skills (fig. 2).12  This process is heavily reliant on application of critical thinking and reflection in each phase. The goal of all active learning approaches is to increase knowledge retention so that learners will be better equipped to apply that knowledge to solve new problems they encounter in the future. In addition, the teacher is better able to recognize and correct knowledge gaps as the learners are working through problems.

Fig. 2.

The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 

Fig. 2.

The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 

Flipped Classroom.

The flipped classroom is one educational method that utilizes active learning techniques. Traditionally, classroom time was used for teacher-centered lecture-based learning in which the teacher passively transferred factual information to learners. Learners then applied that information to solve problems independently on their own time as homework. Flipped classroom reverses this process. Learners obtain foundational knowledge before coming to class through “homework.” Although multiple formats can be utilized (reading assignments, electronic modules, videocasts, etc.), a brief, 15 to 20 min video is the most common format for the “homework” component. (Table 2 provides some examples of educational resources that can be used for preclass homework for anesthesiology learners.) Learners then spend in-class time using that foundational knowledge for problem solving through active learning.

Table 2.

Sampling of Educational Tools Specific to Anesthesiology Learners

Sampling of Educational Tools Specific to Anesthesiology Learners
Sampling of Educational Tools Specific to Anesthesiology Learners

There are many proposed benefits of flipped classroom. It allows flexibility in the time, place, and pace in which learners obtain foundational knowledge. It promotes teamwork, which also helps hold learners accountable to each other for being prepared for in-class sessions. By observing how learners solve problems, facilitators can identify and correct knowledge gaps in real time.13 

There have been a number of studies assessing flipped classroom in healthcare education. Utilization of flipped classroom has led to knowledge gain in pharmacy education.14–16  A recent systematic review of flipped classroom in medical education found that learners preferred flipped classroom, but there were mixed results regarding the benefit of flipped classroom in knowledge and skill gain. Most studies in the review were performed with medical student learners.17  Since this review, there have been several additional studies published in graduate medical education, again with mixed results. One study showed that within a group of internal medicine residents learning about quality improvement, there was a preference for flipped classroom, and it was found to increase knowledge acquisition.18  However, another study with emergency medicine residents utilizing flipped classroom was equivocal with respect to knowledge acquisition.19  A large multiinstitutional study with residents preparing for the American Board of Anesthesiology (ABA; Raleigh, North Carolina) BASIC Exam found that anesthesiology residents preferred flipped classroom to traditional lectures. Although there was not a significant improvement in immediate knowledge gain, testing done 4 months after learning demonstrated a significant improvement in knowledge retention in flipped classroom group compared to traditional lectures.20 

Much of the work assessing the benefit of flipped classroom utilized multiple-choice tests for knowledge assessment. Most multiple-choice questions assess the “knows” level of Miller’s pyramid, in which the learner demonstrates knowledge of factual information. Some may reach the “knows how” level, in which the learner demonstrates knowledge of factual information and also how to interpret and apply that information (fig. 3).21,22  As one of the goals of flipped classroom is to foster better problem-solving skills and improve knowledge application, multiple-choice questions may not be the best assessment tool.23  Gillispie et al. demonstrated a stronger benefit of flipped classroom in an obstetrics and gynecology medical student clerkship when assessing with an Objective Structured Clinical Exam compared to multiple-choice questions.24  Future work assessing the use of flipped classroom for anesthesiology residents may therefore benefit from testing with Objective Structured Clinical Exams in addition to multiple-choice questions.

Fig. 3.

Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 

Fig. 3.

Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 

Peer Teaching.

Peer teaching is a learning environment in which learners are taught by their peers. Technically, the learners and teachers in this scenario should be at the same level of training. Similarly, near-peer teaching occurs when learners are taught by other learners who have more experience or education (e.g., residents teaching medical students). These near-peer teachers have less experience and knowledge than faculty and typically have little to no formal education in teaching methodology.

Peer teaching and near-peer teaching have many suggested benefits. First, in this era of competing priorities and time limitations, the teaching burden is shifted away from faculty. Second, peer teachers often benefit both from their own mastery of the content and gaining early exposure to teaching opportunities. This method also fosters teamwork skills that may enable peer teachers and learners to more easily identify their strengths and weaknesses.25  In addition, faculty may succumb to the “curse of knowledge” in which they assume that learners have a stronger knowledge base than they actually do. As peers may be better equipped to put themselves in the position of the learner, they may have an advantage in teaching a new concept.

Peer teaching can be utilized in a large group learning environment, facilitated by a faculty member.26  The faculty poses a question, case, or problem to the learners. This can be done with an audience response system or through open-ended questions. Learners are given a short amount of time to think through the answer individually. They then organize into pairs, or small groups if this method is modified to “think-group-share,” to discuss their answers and further problem-solve. The entire group of learners then reconvenes to review the possible solutions to the problem under the direction of the faculty leader.

Peer teaching has a long history of utilization in anesthesiology with hands-on experience in the perioperative environments such as the operating room. For instance, senior residents often mentor and teach junior residents, while both senior and junior residents teach medical students.

Although, to our knowledge, there is no recent work on peer teaching in graduate medical education, several studies have investigated this method with medical students. A review of undergraduate medical education found some evidence that peer teachers gained greater mastery of subject matter. Although learning gains were not always demonstrated, neither was there evidence of detrimental effects to the peer teachers’ learning.25  Gregory et al. demonstrated that both preparation for teaching and the actual act of teaching increased peer teachers’ knowledge gain and retention in medical students who were peer teachers compared to those who did not peer teach.27  Peer teachers felt they benefited through the gained mentoring experience, engagement with and learning from learners, and an increased motivation to study.28  In addition, it was shown that peer teaching was as effective as faculty-led learning. This was thought to be due to the cognitive and social congruencies that peer teachers shared with their learners.25 

Educational Games.

Educational games are gaining popularity in the medical education community. These can be categorized as simulation, virtual environments, social and cooperative play, or alternative reality.29  For the purpose of this review, we will only discuss educational games that fit into the social and cooperative play category, as these are a form of active learning. Educational games have preset rules and include an element of competition. A variety of board games and TV gameshows have been adapted for medical education including Jeopardy, Monopoly, Hollywood Squares, and Snakes and Ladders.

Proposed benefits of educational games include the stimulation of higher levels of thinking, a fun and exciting educational environment that reduces stress and anxiety, and an increase in knowledge retention.29  However, a Cochrane review was unable to determine a knowledge benefit from game-based learning and concluded that more studies are needed.30  There have been a few studies done with medical students and one in continuing medical education that have not shown an increase in knowledge gain from game-based learning.31–33  However, participants in all of these studies preferred game-based learning.31–33  There is one report in graduate medical education literature on the use of an adaptation of the game show Jeopardy for teaching general surgery residents. Although there was demonstration of knowledge acquisition and retention compared to pretest knowledge, there was no comparison group utilized in this study.34 

Optimizing Knowledge Retention

Three evidence-based methods of teaching and learning that facilitate long-term retention of knowledge include spaced learning, interleaving, and retrieval practice. Although the benefits of these strategies have mainly focused on improved knowledge acquisition and retention, evidence is growing that they also may be applied to attainment of psychomotor skills.35,36 

Spaced Learning.

Repetition is a successful strategy for learning. When repetitive study is spaced out over an extended period of time, it is referred to as “distributed” or “spaced.” The “spacing effect” describes the finding that when an equal amount of cumulative study or practice time is spaced out between sessions with an intervening time gap, knowledge acquisition and long-term knowledge retention are better than if sessions of the same material are massed together. An initially short, and then expanding, interstudy time interval may be optimal. A longer interstudy interval is generally associated with improved long-term retention, but the time between studying should not be so long that the learner forgets what they previously learned.37–39  Spaced education has been used to improve knowledge retention in healthcare education including graduate medical education.40–45 

Many learners utilize massed learning or “cramming” and have experienced the short-term benefit of passing their examinations. Massing “works” when the information to be learned is not difficult and testing is in the near future. However, the more time spent in a single session, the less effective that time becomes, and massed learning leads to poor knowledge retention.

Examples of using spaced learning in anesthesiology include (1) distributing rotations throughout a multiyear curriculum rather than massing them together (e.g., pediatric anesthesia, critical care medicine, and cardiac anesthesia), (2) teaching a topic (e.g., neonatal resuscitation, mechanical ventilation management, or cardiac physiology) in shorter sessions and distributing teaching sessions over time, (3) utilizing mobile devices to deliver spaced repetition of information to learners (e.g., emails, text messages, and questions), and (4) encouraging learners to start studying early and often, and utilizing initially short time intervals between repetitions of a specific topic and then expanding the time intervals between repetitions.

Interleaving.

“Blocking,” which is typically preferred by learners, refers to the repetitive studying of information in one topic or category completely before going on to another, or practicing one skill at a time toward mastery before practicing another. Interleaving is a process where students mix, or interleave, the study or practice of several related concepts or skills. An example would be when learners are tasked to understand and retain knowledge about anesthetic pharmacokinetics in infants, adults, and the elderly. If applying the interleaving technique, they might go back and forth (i.e., utilizing comparison and contrast) between studying about different anesthetics in each of the age groups. This is opposed to a blocking technique in which they would limit their reading and studying of pharmacokinetics to one age group until obtaining mastery of the concepts before going on to another age group. A learner utilizing interleaving to help master the skill of transesophageal echocardiography would mix up their study and skill practice. They might work on acquiring images on the simulator and on patients while also reading about physics and hemodynamic calculations. This would be opposed to blocking, in which they focus only on image acquisition during their first rotation and then later learn about physics and hemodynamic calculations on their second rotation. Medical school curricula are now frequently mixed or interleaved. Separate but related topics are interleaved together such as cardiac anatomy, cardiac physiology, and cardiac pharmacology rather than presenting all anatomy topics until completion, then physiology, and finally pharmacology in a similar blocked format.

Content understanding and memory/retention for information learned through interleaving are better than through massed practice.46,47  However, interleaving feels slower and less efficient to the learner. Massing is often preferred over interleaving, even when the learner is confronted with evidence from personal performance measures that demonstrate better retention and ability to transfer knowledge to other contexts with interleaving.48,49 

Interleaving the study of separate but related topics can help the learner see similarities and differences between ideas or concepts, creating connections between related parts. It also takes advantage of the spacing effect by distributing learning. Cognitive psychology has provided evidence for the benefits of mixing up categories of topics during learning, and interleaving has been used to improve learning in mathematics, radiology, and neuroanatomy.47,50,51 

The optimal timing and patterns for interleaving may vary between learners, and determination of such requires further investigation. While there is an educational advantage to switching between categories of information, switching too often or spending too little time on any one idea may be detrimental. Interleaving appears to be most beneficial when the category discrimination is difficult; that is, differences between presented materials (cases, disease pathophysiology, and images) are difficult to distinguish from each other, and interleaving provides the opportunity for observing differences by comparison and contrast.49,52 

An example of interleaving in clinical practice is a month-long rotation in the Clinical Anesthesia 3 year of training called “Transition to Practice” in which a resident is not on a specific subspecialty for an extended period of time such as pediatrics for 1 month or cardiac for 1 month, but instead the subspecialty case type and the learning experiences vary day to day and possibly patient to patient. The resident may one day be doing predominantly pediatrics, the next day doing predominantly ambulatory, the next thoracic, etc. This format mixes up the study of related concepts and skills.

Retrieval Practice.

Cognitive science suggests that we take information (audio, visual) into our limited short-term memory, encode that information into our “unlimited” long-term memory, and then retrieve it from long-term memory when needed. The act of retrieving a memory changes it, making it easier to retrieve again later, and increases long-term retention.53,54  One example of this is the long-term retention of phone numbers that you regularly recall from memory as opposed to those you look up and enter from a contact list. Retrieval practice is the act of calling information to mind that has previously been encoded or learned rather than rereading or rehearing it.

Test-enhanced learning refers to the use of retrieval practice through testing as a way to enhance retention of knowledge.55,56  The “testing effect” refers to the use of retrieval as a powerful learning tool. A direct benefit of testing is that recalling previously learned information enhances the ability to recall information in the future. When information is recalled from memory during low- or no-stakes testing by doing questions and receiving feedback, retention is better than if the material is simply restudied over and over again.57,58 

The benefit of retrieval practice in health professions education has recently been reviewed.57  The first studies assessing the positive effect of formative assessment on learning outcomes in medical education were published within the last decade, and most studies since that time have compared retrieval practice with the passive learning strategy of restudy.58,59  However, the benefits of testing relative to active learning strategies, such as concept mapping and case-based learning, have been demonstrated.60,61  In a recent systematic review, retrieval practice demonstrated consistent and strong effects across learners in various health professions and for various learner outcomes. Moreover, the effectiveness of retrieval practice extended beyond knowledge assessed by examinations to clinical applications.57 

Ideally, tests used to enhance learning should be composed of questions related to the topic previously studied, repeated, and spaced over time; should utilize questions that require production of information rather than just recognition; and should include feedback with correct responses and rationale.46,57  Tests that require generation of information (e.g., short answer) as opposed to recognizing and selecting a response from information provided (e.g., multiple choice) and that require more effortful recall appear to have a greater benefit on knowledge retention. Repeated testing that is spaced over time allows the learner to take advantage of feedback and practice to correct errors.62  Feedback enhances the benefits of testing; although testing alone improves retention, feedback should be provided to help the learner close the gap between what they desire to learn and what they actually learned. Interestingly, delayed feedback may be more beneficial than immediate feedback.63  When the test taker is conditioned by instantaneous feedback to expect the certainty of continual correction, it may reduce the likelihood they will engage in effortful recall.63  The efficacy of repeated testing seems to be greater the more complicated the subject being studied. It is apparent that repeated testing promotes learning in ways that restudying cannot.64,65  Moreover, there is a small but growing body of evidence that those who learn foundational information using retrieval practice are more likely to transfer their knowledge to novel situations than those who learned the foundational information by repeatedly studying it.64,66,67 

Posttests are commonly administered immediately after learning sessions such as with online education or active learning didactics. However, recall is relatively easy immediately after study, and the level of performance immediately after learning is not a good indicator of future retention. Electronic flash cards with questions, question-generating applications, questions delivered to learners via mobile devices, and utilization of question banks that include feedback are examples of ways to utilize formative retrieval practice via testing. The ABA Maintenance of Certification in Anesthesiology Minute is an example of a program that uses both retrieval practice and spaced learning via quizzing (table 2). These principles might be applied beyond using test questions to enhance learning and knowledge retention to other forms of retrieval practice such as oral examinations, peer teaching, and even psychomotor skills such as cardiopulmonary resuscitation.68 

After initial encoding of information, if a learner preparing for the ABA In-Training Examination used questions from a question bank to encourage retrieval of knowledge from memory and chose to mix questions from various topics rather than answering all questions on a specific topic before moving on to another, this would be an example of interleaving and retrieval practice.

Cognitive Aids

In 1924, an article was published that called for routine training on, and use of, cognitive aids during intraoperative emergencies.69  In recent years, multiple institutions and national organizations have promoted the use of cognitive aids for perioperative emergencies, leading to the development and publication of several emergency manuals.70  One article defined cognitive aids as “tools created to guide users while they are performing a task, or group of tasks, with the goal of reducing errors and omissions and increasing the speed and fluidity of performance.”71  In the broadest sense, anything that acts as a memory aid for the clinician could be considered a cognitive aid. Specifically, within the domain of anesthesia, critical care, and perioperative medicine, the focus of research has centered around checklists.71–73  The characteristics of an optimal cognitive aid are accurate and reliable clinical content, a user-friendly design that includes human factors assessment, and iterative assessment and improvement of the aid itself using simulation or clinical use.72,74  With the advent of the smartphone also came the question of the best format for a cognitive aid: paper versus electronic.72,73  Taken together, there can be a complex taxonomy of cognitive aid and end user, which can range from “static parallel checklists” where one operator reads and performs all tasks on the checklist (e.g., anesthesia machine checkout) to dynamic clinical checklists in which one or more team members uses a tool that involves branching logic and clinical decision-making (e.g., management of pulseless arrest and return of spontaneous circulation).71,75–78 

The research concerning the effect of cognitive aids within anesthesia, critical care, and perioperative care has been extensive. Numerous prospective randomized controlled trials have evaluated the effect of cognitive aids on the technical performance of individuals or teams. Most trials concerning perioperative and in-hospital emergencies have demonstrated a positive effect for completing actions considered to be best practice, or standard of care, for a given situation.76–82  However, some have noted that the cognitive aid can inhibit performance by being a distraction,83  which is most commonly the case when cognitive aids are used without appropriate education on the tool prior to use.84  Beyond crisis situations, a number of trials have shown improved adherence to accepted care guidelines when using cognitive aids in simulated nonacute perioperative situations, such as applying guidelines for preoperative cardiac evaluation, assessment of patients on anticoagulants for regional anesthesia, and compatibility in blood transfusions.85–87  Furthermore, improvement in technical performance and nontechnical skills (including teamwork and communication) have been shown to be improved with the use of checklists in the operating room.88  In summary, evidence has resulted in numerous statements calling for increased training and implementation of cognitive aids in health care.73,89,90 

Much remains to be done, however, to include routine education on the use of cognitive aids in most training curricula. For instance, while simulation experiences are included as a program requirement, training on routine use of cognitive aids is not included in the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements or the Anesthesiology Milestones,91,92  even though the science shows clear benefit. Recent research has shown that brief operating room–based team simulations can increase familiarity with and likelihood to use cognitive aids.93  Furthermore, recent data have demonstrated that targeted education and implementation efforts, including structured use during simulation, are highly associated with self-reported use of cognitive aids in actual crisis situations.94,95  This is particularly important because studies have shown that strict adherence to practice guidelines and completion of all checklist items are associated with improved clinical outcomes.96,97  It is encouraging that national societies and major journals are publishing cognitive aids and recommending their routine use for managing critical events.

Nonetheless, more is needed. In order to have a truly ideal cognitive aid, not only must the tool itself have the three qualities listed above, but also there must be repeated familiarization and practice with the cognitive aid within the team environment, and the aid must be accessible within the context of education and clinical practice.72,74  Cognitive aids should be utilized when testing learners on scenarios such as malignant hyperthermia, anaphylaxis, or cardiac arrest. Based on current evidence, it is recommended that routine education be provided on the use of cognitive aids and that cognitive aids be referred to during the management of a life-threatening event, whether simulated or real. Given the current state of the evidence, it is incumbent upon policy makers, program directors, and faculty educators to make routine education on, and use of, cognitive aids a reality.98 

E-learning

E-learning is defined as learning utilizing electronic technologies to access internet or intranet-based resources outside of a traditional classroom.99,100  In practice, e-learning includes any digital content taught and distributed physically or online, such as web-based tutorials, virtual patients, and discussion boards.99,101 

The advantages of e-learning in medical education are clear. E-learning offers greater flexibility in terms of time and accessibility.99,100,102,103  As long as the learning outcomes of e-learning are comparable to other methods, the flexibility e-learning provides makes it particularly appealing to graduate medical education, as it can help relieve faculty constraints and allocate limited resources more efficiently.102 

In their meta-analysis on the effects of e-learning in the health professions, Cook et al. found that e-learning is associated with large positive effects compared to no intervention on all learning outcomes, including knowledge, skills, behaviors, and patient effects. Compared with traditional noninternet instructional formats, e-learning has positive effects on all educational outcomes as well, although the effects are heterogeneous and marginal.99  More recent studies in primary care and surgical education also reported predominantly positive learner perceptions and improvement in knowledge and skills as a result of engagement in e-learning compared to traditional methods.100,103  Evidence of the effects of e-learning on long-term knowledge retention are scarce and yet to be validated.100,102 

Despite the general positive impacts on e-learning found in the literature, it is important to understand how and when to use e-learning to leverage its educational value.104  Access to digital content alone is not sufficient for in-depth learning.102  The technology should be used in an appropriate context, targeted to the goal of the curriculum, adapted to the needs of the specific audience, and adhering to principles of educational psychology.102,104,105  Good educators should play active roles as content curators and technology adaptors to evaluate and select the appropriate content materials and technology tools.106  Low learner participation rate also appears to be an issue in some e-learning courses, which points to the importance of incorporating compliance measures into implementation.102  In order to maximize the benefits of e-learning, including increased engagement and self-directedness, focused and immediate feedback, repeated opportunities for improvement, and clear emphasis on areas in need of improvement should be provided to learners.102  The incorporation of spaced education and blending self-paced e-learning with in-class instruction have also proven to be successful in improving learning in medical education.100,102  (Table 2 contains examples of e-learning environments utilized for anesthesiology education.)

Teaching Procedural Skills

Technical skills related to the practice of anesthesiology are numerous and require competence with a wide range of techniques and equipment (table 1). The ACGME Milestones92  are focused on performance of arterial lines, central lines, and regional anesthetics (central and neuraxial). Point-of-care ultrasound and transesophageal echocardiography (TEE) have also emerged as valuable skills in the armamentarium of practicing anesthesiologists. Both the ACGME and ABA have recognized the importance of training residents in these skill sets and have set forward expectations for skills acquisition (ACGME) and subsequent verification of basic working knowledge in these areas (ABA). In addition to TEE and transthoracic echocardiography, lung ultrasound, and imaging of the inferior vena cava including the Focused Assessment with Sonography for Trauma and Focused Assessed Transthoracic Exam are growing in popularity.107,108 

Teaching procedural skills in anesthesiology is challenging due to the complex interplay of knowledge, manual dexterity, and workflow elements, all of which need to be learned and applied in concert. Despite the synergy of training in all these areas simultaneously, most training programs focus solely on manual dexterity. Additionally, skills application is episodic in nature, resulting in loss of facility and requiring reacquisition of skills through repeated training.109  Furthermore, verification of proficiency relies on the assumption that participation in a fixed number of procedures will result in uniform learner proficiency; best evidence suggests that this approach is fundamentally flawed.110,111  Finally, there are few validated instruments for procedural skills assessment; most are simple checklists. There is a need for validated instruments that compare the performance of the trainee not only against a list of steps, but also against objective target metrics of experts in order to verify competence.112–115 

Techniques Utilized

Direct Observation.

The most common approach to teaching procedural skills involves direct observation of skills by senior physicians, which is best facilitated by feedback (table 1). There are a variety of suggested approaches to providing feedback, including real-time, delayed, or video-based techniques.116–118  While immediate feedback to improve an element of performance may be provided effectively during the procedure, summative feedback (feedback on overall performance) should be provided later to avoid distraction and increasing cognitive load on the learner.119,120  Checklists of required elements can also be incorporated into review sessions to enhance the feedback quality to learners.121,122 

Simulation.

Partial task training has become a mainstay of procedural training (table 1). It is highly effective as a teaching modality and allows for low-stakes training of invasive procedures.123–126  Task training may be done with either high- or low-fidelity models.126,127  Low-fidelity models are useful for novices,128,129  but higher-fidelity models are often required to engage experienced learners and to assess communication skills and procedural team dynamics.130  Hybrid approaches, incorporating both live actors and procedural trainers, have also been successful in teaching communication and procedures simultaneously.131  Task-trainer–based learning is most effective when supervised by an experienced educator who can provide feedback to improve performance.121,122  Limitations of task trainers can include availability of adequate resources to train learners simultaneously, expense, and availability of teaching staff. Incorporation of small group teaching, use of low-fidelity models where appropriate, inexpensive three-dimensional printing of models,126,132  and peer teaching133  can ameliorate some of these issues. Virtual reality and augmented reality, where additional visual cues are overlaid on the physical environment to enhance learning, are both popular and proving increasingly effective as a technique for teaching procedural skills.134–140 

Live models and virtual reality simulators represent the dominant methods of teaching TEE and point-of-care ultrasound, supplemented by live patient encounters. Live models can help with teaching image acquisition skills for transthoracic echocardiography and lung ultrasound but are typically limited to healthy volunteers and thus cannot demonstrate complicating medical issues or show abnormal anatomy. They also require set teaching times, based upon live models’ availability, and therefore result in limited exposures for learners. In contrast, simulators can be made available on flexible schedules and programed to show abnormal findings.141,142 

Theory-grounded Approaches to Skills Training

Mastery Learning with Deliberate Practice.

Mastery learning entails achievement of a high set standard for all learners regardless of the time and effort required to achieve these levels.143  If learners do not achieve these goals, they must retrain until they do so without limitations of time on training; standards are typically set by experts with consideration for best evidence and data.144  Deliberate practice is the pursuit of techniques known to augment performance above rote repetition. It is characterized by a desire of the learner to focus on the task and build on existing knowledge to formulate discrete goals requiring little instruction. Immediate feedback and performance data should also be provided as part of this process.145  Deliberate practice and reflection with mentorship allow for more rapid and comprehensive acquisition of skills and knowledge. Mastery learning and deliberate practice are synergistic approaches and represent the ideal framework to apply to simulation-based technical skills teaching.143  Mastery learning is superior to conventional techniques in terms of the degree of technical skill attained and compliance with required steps to ensure safety and reduce complications.146–148  A recent best evidence review of five basic anesthesiology procedures (provision of general anesthesia, airway management, spinal anesthesia, epidural anesthesia, and central line placement) suggests enhanced patient outcomes and decreased complications with mastery-based learning, but mixed outcomes in terms of long-term retention and skill transfer.149  Mastery learning concepts are also being applied to point-of-care ultrasound training in anesthesiology, which will be discussed in detail later.111 

Workflow and Communication Skills.

Workflow and communication skills have been increasingly recognized as important to ensure that learners can practically apply skills in the clinical environment. Traditionally taught through the “hands-on” approach in the clinical environment, they should be included as part of teaching in the simulated environment in order to ensure consistent and safe skills performance.150,151 

Cognitive Load and Task Complexity.

Cognitive load theory suggests that, as working memory has limited capacity, more complex tasks take up increased amounts of that working memory. The more working memory required to complete a task, the more difficult it becomes to learn or accomplish at a consistently high level.120  Learner, patient, and supervisor features are associated with different types of cognitive load during procedural skills training.118–120,152,153  Optimizing factors impacting cognitive load (such as reducing fatigue and enhancing supervision and feedback) can improve initial learning.118  Repetitive simulation-based learning can reduce cognitive load and improve performance.152,153 

Transferability.

Making sure that skills are transferrable to the clinical realm requires different strategies than initial skill training. While skills are initially learned better by reducing variability and providing feedback, they also need to be practiced in situations with higher variability.119  Thus, a curriculum with graduated complexity is advisable.142,153 

Spaced Learning.

The spacing effect applies not only to knowledge retention but also to psychomotor skill development, as skills predictably decay over time if not reinforced. Students will lose technical skills between 6 and 12 weeks after training without further practice.109  It is unclear what training interval will keep trainees from losing skills, but more frequent training results in more rapid skills acquisition.154  Distributed practice over time has been used to improve learning of surgical and resuscitative skills.35,155,156  Both knowledge and skill acquisition are dependent on memory consolidation, and spacing allows this to occur.157  The repetitive practice of several separate but related skills, as opposed to practicing one skill at a time until “mastery,” may also improve the learning of psychomotor skills.158 

Retrieval Practice.

The principle of retrieval practice can be applied to psychomotor skills and procedures that are retrieved from memory.159  However, it is important to emphasize that retrieval practice is based on the initial encoding or learning of the information prior to testing. Repeated testing such as an Objective Structured Clinical Exam may produce more favorable skills retention than repeated practice.36  In general, the type of retrieval that is practiced during learning should be a good match for the way in which the information will need to be retrieved and used in the future whether that is a written test, oral examination, mannequin simulation scenario, or standardized patient encounter (Objective Structured Clinical Exam).

Metrics for TEE and Point-of-Care Ultrasound

TEE and point-of-care ultrasound represent excellent examples of how innovative learning programs and metrics should be rigorously applied and assessed to gauge progress toward mastery. For live model sessions, metrics are typically limited to checklists of task completion, assessment of quality of images obtained, or time spent acquiring images. While useful, they do not convey the entire range of skills required. Simulation-based metrics include the total path length taken to achieve the image, numbers of starts and stops, and latency/processing delay before beginning probe motion.160  Multiple studies in TEE and transthoracic echocardiography training have demonstrated similar patterns of progression across a range of learner cohorts and disciplines.112,113,161  The growing ability to compare images of learners to those of experts quantitatively enhances the objectivity of image quality assessment and eliminates the need for human raters.115 

Integrated teaching and testing paradigms have been essential to the assessment of skills, knowledge, and workflow for point-of-care ultrasound and TEE.107,142,161–163  Most incorporate multimodal elements of live workshops, simulation, online flipped classroom, and classroom-based learning. Testing takes the form of written exams, checklists, and analysis of simulator based motion data.142,160–162  Similar to other areas, Objective Structured Clinical Exam–type exams focused on demonstrating skills, workflow, interpretation of images, and diagnoses are proving useful to assess ability of learners to apply these skills.163,164 

The key question of transferability to the clinical setting for TEE and point-of-care ultrasound skills has been addressed via several approaches. Assessment of image quality and exam time in live patients at the end of a simulator-based course showed moderate quality images for trainees completing the course.112,113  Another approach involved course participants completing exams overread by experts, which suggested improvement in patient-level outcomes following the course.165  A more recent approach integrated a novel informatics-based solution to compare basic reports generated by trainees to elements of more complex reports generated by experts and showed excellent agreement following course completion.114  While more work needs to be done, these studies suggest that skills learned outside the clinical arena can be transferred to point-of-care ultrasound or TEE in live patients.

Teaching Professionalism

The ACGME requires programs to provide instruction and assessment in professionalism, which is more challenging than imparting knowledge and evaluating technical and clinical skills.166–168  One major educational challenge is the lack of universal agreement on the definition of, and criteria for, medical professionalism.167,169,170  Professionalism can be conceptualized differently based on virtues, personal attributes, behaviors, and identity formation processes.171–174  Some see professionalism as a unidimensional entity while others view it as a construct that encompasses everything a doctor needs to fulfill the job.171  Some think that professionalism is a commitment to lifelong self-directed learning for improvement, demonstrating compassion and respect to patients, yet some frame professionalism under a legal and regulatory model.171,172  It is a concept in flux.175  Professionalism is essentially unteachable from the perspective of those who define professionalism as an attitude.167,168  The mainstay for teaching and assessing medical professionalism has focused on professionalism as competencies that can be observed and described by behaviors, traits, and roles.167,172,174  According to the ACGME milestone framework, professionalism competencies involve strong interpersonal and communication skills as well as commitment to, and care of, others and oneself, which are separated from the medical knowledge competency.176  Nonetheless, some scholars believe that specialty expertise, including medical knowledge and clinical skills, is a key component of professionalism and that it is artificial to differentiate between medical knowledge and professionalism.168 

Another challenge in teaching professionalism is the difficulty in standardizing a formal curriculum, as much of professionalism is learned through “hidden curricula,” the informal curricula that cultivate alternative norms, values, attitudes, and domains of learning.169,170,172,177  Sometimes formal and hidden curricula contradict each other and create confusion, misunderstanding, and even cynicism in learners.172  Diversity in the perception of professionalism due to generational and cultural differences also add to the challenge in the design, implementation, and evaluation of a curriculum.172  In their 2013 systematic review, Birden et al. identified over 200 papers describing methods to teach professionalism, but few studies reported evaluation data.167  The most commonly adopted strategies in teaching professionalism were role modeling and guided personal reflections.167 

Role Play

Role play, a form of active learning, can be used to teach communication and professionalism. In role play, the learners act as themselves or a scripted character, typically a patient or clinician. The amount of detail involved in the script that the learner follows can vary. The role can be entirely scripted, improvised based on a brief description, or somewhere in between.178  Using the concept broadly, role play can be viewed as a form of simulation in which the learners play themselves in a scenario with a standardized actor. However, for our purposes, the definition will be more specific to the replacement of standardized actors, allowing learners to experience the perspectives of other team members (e.g., surgeon, nurse, patient) while simultaneously decreasing the resource utilization of the learning experience by reducing the need for standardized actors.

To our knowledge, there are no data supporting role play in graduate medical education. It is suggested that role play can increase learners’ awareness of their usage of body language, teach empathy, and provide insight into personal communication skills, fears, and emotions.179  A study done with medical students comparing the use of standardized patients to the use of learners playing the role of the patient found that both methods improved learners’ communication skills, but learners felt they had a better understanding of the patient perspective when they had the opportunity to play the patient themselves.180 

There may be some stigma associated with the idea of role play in medical education. One study showed that prior to participating in a role play exercise, only 77% of medical students thought it would provide insight into their, and others’, behaviors and allow them the opportunity to practice skills. The remaining students expressed feelings of anxiety, fear, intimidation, and embarrassment. However, after participating in role play, 99% of these students thought it was helpful in allowing them to practice their skills in a safe environment with immediate feedback.178  In order to provide a nonthreatening environment, it is critical to clarify the learning objectives in advance both to define ground rules and to allow for adequate time for preparation and role play.181  It is also recommended to actively include all learners in the role play exercise. This can be done by breaking into groups of three in which one person is the doctor, one the patient or other health professional, and the third an observer whose role is to provide feedback.182 

Assessment

The assessment of professionalism as a complex, multidimensional construct entails a combination of tools171  (table 1). In their systematic review, Wilkinson et al. identified nine categories of assessment tools that have been used separately or jointly, including (1) observed clinical encounters, (2) collated views of coworkers, (3) records of incidents of unprofessionalism, (4) critical incident reports, (5) simulation, (6) paper-based tests, (7) patients’ opinions, (8) global views of supervisors, and (9) self-administrated rating scales.171  More recent techniques, such as the use of simulated data and situational judgment tests, may also be useful to assess and teach professionalism, especially if the concept is defined by observable behaviors.183,184 

There is recognition of the importance of professionalism teaching within anesthesiology, but research has been limited.185,186  Some programs have focused on curricular development, while others aimed to promote enhanced feedback to reinforce professional behavior.185,187,188  Further research in graduate medical education is needed in this domain.

Experiential Learning: Knowledge, Skills, and Professionalism

Anesthesiologists have been using mannequin-based simulators in nonpatient care settings to enhance knowledge and skills of learners for decades.189,190  Since simulated clinical experience is a core requirement for anesthesiology residency training programs, many anesthesiologists are familiar with the benefits of learning novel concepts, or solidifying previously learned skills, in a low-stakes environment like a simulation center.91  Traditionally, residency programs focused their simulation-based educational efforts on the core competency areas of medical knowledge, patient care, and procedural skills, by teaching basic anesthesia principles, responses to anesthesia-related emergencies (e.g., crisis resource management), and “trouble-shooting” common anesthetic issues.191  However, simulation-based Objective Structured Clinical Exams are uniquely designed to provide both formative and summative assessment of all core competency areas including professionalism, interpersonal and communication skills, and practice-based learning and improvement.

An Objective Structured Clinical Exam consists of a series of brief stations or clinical scenarios, where a variety of technical and nontechnical tasks are completed. Examiners either complete an objective checklist of behaviors or provide a global rating score to evaluate examinees based upon observing their performances within each individual station. With evaluations from a variety of skill areas, completed by different examiners at each station, Objective Structured Clinical Exams provide a comprehensive evaluation of an examinee.192  The use of Objective Structured Clinical Exams in “high-stakes” situations (e.g., board certification) for anesthesiologists has been occurring for several years. In the United Kingdom, the Royal College of Anaesthetists (London) delivers an Objective Structured Clinical Exam of 17 brief stations where examinees are evaluated in a variety of skill areas such as history taking, physical examinations, x-ray film interpretation, electrocardiogram interpretation, resuscitation, and safety checks of anesthesia equipment.193  Alternatively, the Israeli Board of Anesthesiology (Jerusalem) delivers an Objective Structured Clinical Exam for board certification where examinees are exposed to five longer mannequin-based scenarios focusing on trauma and crisis management, resuscitation, mechanical ventilation, and regional anesthesia.194  The ABA recently implemented an Objective Structured Clinical Exam as part of the APPLIED examination.195  The ABA recognized that the written and oral examinations assessed lower levels of competence and they were not assessing higher level competencies required in clinical practice such as communication, professionalism, and certain technical skills.196 

High-stakes implications and summative assessments are not the only use of an Objective Structured Clinical Exam. Rather, an Objective Structured Clinical Exam can provide formative feedback to learners, educators, and an educational program as a whole (e.g., residency program).197  For instance, an anesthesiology department may decide to implement a new teaching module into their educational program on an emerging topic relevant to clinical practice (e.g., point-of-care ultrasound). In order for the department to evaluate the effectiveness of implementing the program, an Objective Structured Clinical Exam could be administered to assess examinees’ performance of the skills taught in the program. The result from the Objective Structured Clinical Exam would then give feedback to learners regarding their personal deficits, educators on areas to focus their training, and the department regarding resource utilization and overall efficacy of the training program. Objective Structured Clinical Exams may also play a role in evaluating ACGME subcompetency milestone achievement.92  Evidence is emerging that a resident’s performance in an Objective Structured Clinical Exam is associated with both their experience level and performance on standard clinical evaluations.198  As a result, an Objective Structured Clinical Exam could be used as an alternative option to assess competence on certain milestones that are either hard to capture (e.g., “Patient Care 5 – Crisis Management” when there is a moral imperative for the most competent physicians to intervene in the crisis) or those in which an opportunity for assessment is infrequent or unpredictable (e.g., “Professionalism 2 – Honesty, Integrity, and Ethical Behavior” when an opportunity to evaluate a resident addressing an ethical issue may not arise).

As part of the ABA Maintenance of Certification in Anesthesiology process, board-certified anesthesiologists have been participating in simulation courses to satisfy a portion of Part IV of Maintenance of Certification in Anesthesiology 2.0. During Maintenance of Certification in Anesthesiology simulation courses, anesthesiologists participate in high-fidelity scenarios focusing on management of difficult patient care situations, with emphasis placed on teamwork and communication.199  The course allows anesthesiologists to reflect on their experiences and subsequently propose practice improvement plans. Follow-up evaluations from the simulation course demonstrate that greater than 90% of participants successfully implement some or all of the planned practice improvements.200  One drawback of the Maintenance of Certification in Anesthesiology simulation program is the logistical challenge presented to anesthesiologists regarding the need to take time from their practice to travel to simulation centers. However, in the summer of 2017, the American Society of Anesthesiologists and Canadian Aviation Electronics Healthcare released a novel suite of screen-based high-fidelity simulation scenarios known as Anesthesia SimSTAT (table 2). The simulation scenarios are computer-based and set in a gaming environment with the same goal of improving performance in managing anesthesia-related emergencies. The scenarios allow the anesthesiologist to interact with realistic diagnostic and monitoring equipment in a virtual three-dimensional operating room. Upon completion of the scenarios, anesthesiologists are provided with data-based feedback on their performance, identification of strengths, and opportunities for improvement.

Simulation is also being used to address burnout, which is a significant area of concern for anesthesiologists.201  However, instead of using simulation to train anesthesiologists regarding burnout, nonmedical persons involved in the social support of anesthesia providers are being educated on the tasks, expectations, and stressors anesthesiologists are exposed to in an attempt to enhance understanding.202  The simulation exercises consist of hands-on activities (e.g., use of a task trainer to learn how to perform a procedure such as an epidural) and observation and debriefing of a high-fidelity simulation of a critical event in the operating room. As a result of these experiences, anesthesiologists may have an improved ability to communicate with their social support persons regarding their daily activities, which may lead to improved social relatedness and subsequently enhanced wellness of providers.

Evidence-based Education in Anesthesiology: Challenges and Future

Many educational theories and methods have been studied outside of graduate medical education. However, there are major differences between students outside of graduate medical education and anesthesiology residents.11  Most students utilize a classroom-based learning environment in which they are expected to prepare ahead of time. They often have dedicated study time, and their primary role is that of a learner with minimal other responsibilities. Our residents learn on the job while taking care of patients, which is their primary role. Study time is limited due to clinical duties. Additionally, whereas most students are evaluated through frequent formal testing, our resident learners undergo less frequent, but higher-stakes, examinations.203 

Although there is great need for additional educational research in graduate medical education, it is inherently challenging. Although it is often critiqued in the same way that one might critically assess basic science research, it is intrinsically different. Sample sizes are often small, due to the size of residency programs. It is difficult to control for confounding variables in a learning environment, and there may be disagreement on which variables are important.11  Additionally, learner randomization may be unethical or impractical.204  Our definitive goal is to improve residents’ application of knowledge, problem solving, and ultimately patient care;12  but it can be difficult to assess these endpoints. It is often logistically easiest to utilize instruments comprised of multiple-choice exams; these questions will not be successful in assessing the higher levels of Miller’s pyramid21  or application of adaptive learning principles.12 

Multiinstitutional educational studies in anesthesiology are needed to help develop and evaluate evidence that will optimize learning of knowledge, skills, and attitudes/professionalism in this specialty in the future.

Conclusions

Although there are many ways to educate our anesthesiology learners, we have presented techniques with variable levels of demonstrated efficacy in medical education (fig. 1). Education is an ever-evolving field. Anesthesiologists must continue to develop and implement innovative approaches for their residents’ education. It is vitally important to assess and validate educational methodologies in order to provide evidence-based best practices for teaching knowledge, skills, and professionalism to anesthesiology learners. Figure 1 provides suggestions on how to incorporate some of these novel educational methods at your institution.

Research Support

Support was provided solely from departmental sources.

Competing Interests

The authors declare no competing interests.

References

References
1.
Schell
RM
,
Bowe
EA
,
Fragneto
RY
: .
Education in anesthesiology should be evidenced-based.
Anesth Analg
.
2008
;
106
:
1587
8
.
discussion 1588
2.
Desy
J
,
Busche
K
,
Cusano
R
,
Veale
P
,
Coderre
S
,
McLaughlin
K
: .
How teachers can help learners build storage and retrieval strength.
Med Teach
.
2018
;
40
:
407
13
3.
Roediger
HL
,
Pyc
MA
: .
Inexpensive techniques to improve education: Applying cognitive psychology to enhance educational practice.
J Appl Res Mem Cogn
.
2012
;
1
:
242
8
4.
Van Hoof
TJ
,
Doyle
TJ
: .
Learning science as a potential new source of understanding and improvement for continuing education and continuing professional development.
Med Teach
.
2018
, pp
1
6
5.
Weidman
J
,
Baker
K
: .
The cognitive science of learning: Concepts and strategies for the educator and learner.
Anesth Analg
.
2015
;
121
:
1586
99
6.
Yeh
DD
,
Park
YS
: .
Improving learning efficiency of factual knowledge in medical education.
J Surg Educ
.
2015
;
72
:
882
9
7.
Prober
CG
,
Heath
C
: .
Lecture halls without lectures - A proposal for medical education.
N Engl J Med
.
2012
;
366
:
1657
9
8.
Gleason
BL
,
Peeters
MJ
,
Resman-Targoff
BH
,
Karr
S
,
McBane
S
,
Kelley
K
,
Thomas
T
,
Denetclaw
TH
: .
An active-learning strategies primer for achieving ability-based educational outcomes.
Am J Pharm Educ
.
2011
;
75
:
186
9.
Wilke
RR
: .
The effect of active learning on student characteristics in a human physiology course for nonmajors.
Adv Physiol Educ
.
2003
;
27
:
207
23
10.
Freeman
S
,
Eddy
SL
,
McDonough
M
,
Smith
MK
,
Okoroafor
N
,
Jordt
H
,
Wenderoth
MP
: .
Active learning increases student performance in science, engineering, and mathematics.
Proc Natl Acad Sci U S A
.
2014
;
111
:
8410
5
11.
Michael
J
: .
Where’s the evidence that active learning works?
Adv Physiol Educ
.
2006
;
30
:
159
67
12.
Cutrer
WB
,
Miller
B
,
Pusic
MV
,
Mejicano
G
,
Mangrulkar
RS
,
Gruppen
LD
,
Hawkins
RE
,
Skochelak
SE
,
Moore
DE
Jr
: .
Fostering the development of master adaptive learners: A conceptual model to guide skill acquisition in medical education.
Acad Med
.
2017
;
92
:
70
5
13.
Sait
MS
,
Siddiqui
Z
,
Ashraf
Y
: .
Advances in medical education and practice: Student perceptions of the flipped classroom.
Adv Med Educ Pract
.
2017
;
8
:
317
20
14.
Wong
TH
,
Ip
EJ
,
Lopes
I
,
Rajagopalan
V
: .
Pharmacy students’ performance and perceptions in a flipped teaching pilot on cardiac arrhythmias.
Am J Pharm Educ
.
2014
;
78
:
185
15.
McLaughlin
JE
,
Rhoney
DH
: .
Comparison of an interactive e-learning preparatory tool and a conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture.
Curr Pharm Teach Learn
.
2015
;
7
:
12
9
16.
Pierce
R
,
Fox
J
: .
Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module.
Am J Pharm Educ
.
2012
;
76
:
196
17.
Chen
F
,
Lui
AM
,
Martinelli
SM
: .
A systematic review of the effectiveness of flipped classrooms in medical education.
Med Educ
.
2017
;
51
:
585
97
18.
Bonnes
SL
,
Ratelle
JT
,
Halvorsen
AJ
,
Carter
KJ
,
Hafdahl
LT
,
Wang
AT
,
Mandrekar
JN
,
Oxentenko
AS
,
Beckman
TJ
,
Wittich
CM
: .
Flipping the quality improvement classroom in residency education.
Acad Med
.
2017
;
92
:
101
7
19.
Riddell
J
,
Jhun
P
,
Fung
CC
,
Comes
J
,
Sawtelle
S
,
Tabatabai
R
,
Joseph
D
,
Shoenberger
J
,
Chen
E
,
Fee
C
,
Swadron
SP
: .
Does the flipped classroom improve learning in graduate medical education?
J Grad Med Educ
.
2017
;
9
:
491
6
20.
Martinelli
SM
,
Chen
F
,
DiLorenzo
AN
,
Mayer
DC
,
Fairbanks
S
,
Moran
K
,
Ku
C
,
Mitchell
JD
,
Bowe
EA
,
Royal
KD
,
Hendrickse
A
,
VanDyke
K
,
Trawicki
MC
,
Rankin
D
,
Guldan
GJ
,
Hand
W
,
Gallagher
C
,
Jacob
Z
,
Zvara
DA
,
McEvoy
MD
,
Schell
RM
: .
Results of a flipped classroom teaching approach in anesthesiology residents.
J Grad Med Educ
.
2017
;
9
:
485
90
21.
Miller
GE
: .
The assessment of clinical skills/competence/performance.
Acad Med
.
1990
;
65
(
9 suppl
):
S63
7
22.
Mehay
R
: .
The essential handbook for GP training and education.
2012
.
London
, :
Radcliffe Publishing
23.
Morton
DA
,
Colbert-Getz
JM
: .
Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom’s taxonomy.
Anat Sci Educ
.
2017
;
10
:
170
5
24.
Gillispie
V
: .
Using the flipped classroom to bridge the gap to Generation Y.
Ochsner J
.
2016
;
16
:
32
6
25.
Benè
KL
,
Bergus
G
: .
When learners become teachers: A review of peer teaching in medical student education.
Fam Med
.
2014
;
46
:
783
7
26.
Crouch
CH
,
Mazur
E
: .
Peer instruction: Ten years of experience and results.
Am J Phys
.
2001
;
69
:
970
7
27.
Gregory
A
,
Walker
I
,
McLaughlin
K
,
Peets
AD
: .
Both preparing to teach and teaching positively impact learning outcomes for peer teachers.
Med Teach
.
2011
;
33
:
e417
22
28.
Lin
JA
,
Farrow
N
,
Lindeman
BM
,
Lidor
AO
: .
Impact of near-peer teaching rounds on student satisfaction in the basic surgical clerkship.
Am J Surg
.
2017
;
213
:
1163
5
29.
Akl
EA
,
Pretorius
RW
,
Sackett
K
,
Erdley
WS
,
Bhoopathi
PS
,
Alfarah
Z
,
Schünemann
HJ
: .
The effect of educational games on medical students’ learning outcomes: A systematic review: BEME Guide No 14.
Med Teach
.
2010
;
32
:
16
27
30.
Akl
EA
,
Kairouz
VF
,
Sackett
KM
,
Erdley
WS
,
Mustafa
RA
,
Fiander
M
,
Gabriel
C
,
Schunemann
H
: .
Educational games for health professionals.
Cochrane Database Syst Rev
.
2013
, pp
CD006411
31.
Shiroma
PR
,
Massa
AA
,
Alarcon
RD
: .
Using game format to teach psychopharmacology to medical students.
Med Teach
.
2011
;
33
:
156
60
32.
O’Leary
S
,
Diepenhorst
L
,
Churley-Strom
R
,
Magrane
D
: .
Educational games in an obstetrics and gynecology core curriculum.
Am J Obstet Gynecol
.
2005
;
193
:
1848
51
33.
Telner
D
,
Bujas-Bobanovic
M
,
Chan
D
,
Chester
B
,
Marlow
B
,
Meuser
J
,
Rothman
A
,
Harvey
B
: .
Game-based versus traditional case-based learning: Comparing effectiveness in stroke continuing medical education.
Can Fam Physician
.
2010
;
56
:
e345
51
34.
Webb
TP
,
Simpson
D
,
Denson
S
,
Duthie
E
Jr
: .
Gaming used as an informal instructional technique: Effects on learner knowledge and satisfaction.
J Surg Educ
.
2012
;
69
:
330
4
35.
Spruit
EN
,
Band
GPH
,
van der Heijden
KB
,
Hamming
JF
: .
The effects of spacing, naps, and fatigue on the acquisition and retention of laparoscopic skills.
J Surg Educ
.
2017
;
74
:
530
8
36.
Sennhenn-Kirchner
S
,
Goerlich
Y
,
Kirchner
B
,
Notbohm
M
,
Schiekirka
S
,
Simmenroth
A
,
Raupach
T
: .
The effect of repeated testing vs repeated practice on skills learning in undergraduate dental education.
Eur J Dent Educ
.
2018
;
22
:
e42
7
37.
Cepeda
NJ
,
Pashler
H
,
Vul
E
,
Wixted
JT
,
Rohrer
D
: .
Distributed practice in verbal recall tasks: A review and quantitative synthesis.
Psychol Bull
.
2006
;
132
:
354
80
38.
Karpicke
JD
,
Bauernschmidt
A
: .
Spaced retrieval: Absolute spacing enhances learning regardless of relative spacing.
J Exp Psychol Learn Mem Cogn
.
2011
;
37
:
1250
7
39.
Moulton
CA
,
Dubrowski
A
,
Macrae
H
,
Graham
B
,
Grober
E
,
Reznick
R
: .
Teaching surgical skills: What kind of practice makes perfect?: A randomized, controlled trial.
Ann Surg
.
2006
;
244
:
400
9
40.
Blazek
MC
,
Dantz
B
,
Wright
MC
,
Fiedorowicz
JG
: .
Spaced learning using emails to integrate psychiatry into general medical curriculum: Keep psychiatry in mind.
Med Teach
.
2016
;
38
:
1049
55
41.
Dobson
JL
,
Perez
J
,
Linderholm
T
: .
Distributed retrieval practice promotes superior recall of anatomy information.
Anat Sci Educ
.
2017
;
10
:
339
47
42.
Kerfoot
BP
,
Baker
HE
,
Koch
MO
,
Connelly
D
,
Joseph
DB
,
Ritchey
ML
: .
Randomized, controlled trial of spaced education to urology residents in the United States and Canada.
J Urol
.
2007
;
177
:
1481
7
43.
Kerfoot
BP
,
Brotschi
E
: .
Online spaced education to teach urology to medical students: A multi-institutional randomized trial.
Am J Surg
.
2009
;
197
:
89
95
44.
Kerfoot
BP
,
DeWolf
WC
,
Masser
BA
,
Church
PA
,
Federman
DD
: .
Spaced education improves the retention of clinical knowledge by medical students: A randomised controlled trial.
Med Educ
.
2007
;
41
:
23
31
45.
Smeds
MR
,
Thrush
CR
,
Mizell
JS
,
Berry
KS
,
Bentley
FR
: .
Mobile spaced education for surgery rotation improves National Board of Medical Examiners scores.
J Surg Res
.
2016
;
201
:
99
104
46.
Brown
PC
,
Roediger
HLI
,
McDaniel
MA
:
Make It Stick: The Science of Successful Learning
.
Cambridge (MA)
,
Belknap Press of Harvard University Press
,
2014
47.
Rohrer
D
,
Taylor
K
: .
The shuffling of mathematics problems improves learning.
Instr Sci
.
2007
;
35
:
481
98
48.
Kang
SHK
,
Pashler
H
: .
Learning painting styles: Spacing is advantageous when it promotes discriminative contrast.
Appl Cogn Psychol
.
2012
;
26
:
97
103
49.
Zulkiply
N
,
Burt
JS
: .
The exemplar interleaving effect in inductive learning: Moderation by the difficulty of category discriminations.
Mem Cognit
.
2013
;
41
:
16
27
50.
Rozenshtein
A
,
Pearson
GD
,
Yan
SX
,
Liu
AZ
,
Toy
D
: .
Effect of massed versus interleaved teaching method on performance of students in radiology.
J Am Coll Radiol
.
2016
;
13
:
979
84
51.
Pani
JR
,
Chariker
JH
,
Naaz
F
: .
Computer-based learning: Interleaving whole and sectional representation of neuroanatomy.
Anat Sci Educ
.
2013
;
6
:
11
8
52.
Birnbaum
MS
,
Kornell
N
,
Bjork
EL
,
Bjork
RA
: .
Why interleaving enhances inductive learning: The roles of discrimination and retrieval.
Mem Cognit
.
2013
;
41
:
392
402
53.
Karpicke
JD
,
Roediger
HL
3rd
: .
The critical importance of retrieval for learning.
Science
.
2008
;
319
:
966
8
54.
Roediger
HL
3rd
,
Butler
AC
: .
The critical role of retrieval practice in long-term retention.
Trends Cogn Sci
.
2011
;
15
:
20
7
55.
Larsen
DP
,
Butler
AC
,
Roediger
HL
3rd
: .
Test-enhanced learning in medical education.
Med Educ
.
2008
;
42
:
959
66
56.
Phelps
RP
: .
The effect of testing on student achievement, 1910–2010.
Int J Test
.
2012
;
12
:
21
43
57.
Green
ML
,
Moeller
JJ
,
Spak
JM
: .
Test-enhanced learning in health professions education: A systematic review: BEME Guide No. 48.
Med Teach
.
2018
;
40
:
337
50
58.
Larsen
DP
,
Butler
AC
,
Roediger
HL
3rd
: .
Repeated testing improves long-term retention relative to repeated study: A randomised controlled trial.
Med Educ
.
2009
;
43
:
1174
81
59.
Rowland
CA
: .
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Psychol Bull
.
2014
;
140
:
1432
63
60.
Karpicke
JD
,
Blunt
JR
: .
Retrieval practice produces more learning than elaborative studying with concept mapping.
Science
.
2011
;
331
:
772
5
61.
Raupach
T
,
Andresen
JC
,
Meyer
K
,
Strobel
L
,
Koziolek
M
,
Jung
W
,
Brown
J
,
Anders
S
: .
Test-enhanced learning of clinical reasoning: A crossover randomised trial.
Med Educ
.
2016
;
50
:
711
20
62.
Larsen
DP
: .
Picking the right dose: The challenges of applying spaced testing to education.
J Grad Med Educ
.
2014
;
6
:
349
50
63.
Butler
AC
,
Roediger
HL
3rd
: .
Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing.
Mem Cognit
.
2008
;
36
:
604
16
64.
Butler
AC
: .
Repeated testing produces superior transfer of learning relative to repeated studying.
J Exp Psychol Learn Mem Cogn
.
2010
;
36
:
1118
33
65.
Halamish
V
,
Bjork
RA
: .
When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier.
J Exp Psychol Learn Mem Cogn
.
2011
;
37
:
801
12
66.
Carpenter
SK
: .
Testing enhances the transfer of learning.
Curr Dir Psychol Sci
.
2012
;
21
:
279
83
67.
van Eersel
GG
,
Verkoeijen
PP
,
Povilenaite
M
,
Rikers
R
: .
The testing effect and far transfer: The role of exposure to key information.
Front Psychol
.
2016
;
7
:
1977
68.
Kromann
CB
,
Jensen
ML
,
Ringsted
C
: .
The effect of testing on skills learning.
Med Educ
.
2009
;
43
:
21
7
69.
Babcock
WW
: .
Resuscitation during anesthesia.
Anesth Analg
.
1924
;
3
:
208
13
70.
Emergency Manuals Implementation Collaborative.
Available at: http://www.emergencymanuals.org. Accessed February 1, 2019.
71.
Marshall
S
: .
The use of cognitive aids during emergencies in anesthesia: A review of the literature.
Anesth Analg
.
2013
;
117
:
1162
71
72.
Wen
LY
,
Howard
SK
: .
Value of expert systems, quick reference guides and other cognitive aids.
Curr Opin Anaesthesiol
.
2014
;
27
:
643
8
73.
Webster
CS
: .
Checklists, cognitive aids, and the future of patient safety.
Br J Anaesth
.
2017
;
119
:
178
81
74.
Goldhaber-Fiebert
SN
,
Howard
SK
: .
Implementing emergency manuals: Can cognitive aids help translate best practices for patient care during acute events?
Anesth Analg
.
2013
;
117
:
1149
61
75.
De Bie
AJR
,
Nan
S
,
Vermeulen
LRE
,
Van Gorp
PME
,
Bouwman
RA
,
Bindels
AJGH
,
Korsten
HHM
: .
Intelligent dynamic clinical checklists improved checklist compliance in the intensive care unit.
Br J Anaesth
.
2017
;
119
:
231
8
76.
McEvoy
MD
,
Hand
WR
,
Stoll
WD
,
Furse
CM
,
Nietert
PJ
: .
Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated “Reader.”
Reg Anesth Pain Med
.
2014
;
39
:
299
305
77.
Field
LC
,
McEvoy
MD
,
Smalley
JC
,
Clark
CA
,
McEvoy
MB
,
Rieke
H
,
Nietert
PJ
,
Furse
CM
: .
Use of an electronic decision support tool improves management of simulated in-hospital cardiac arrest.
Resuscitation
.
2014
;
85
:
138
42
78.
Burden
AR
,
Carr
ZJ
,
Staman
GW
,
Littman
JJ
,
Torjman
MC
: .
Does every code need a “reader?” Improvement of rare event management with a cognitive aid “reader” during a simulated emergency: A pilot study.
Simul Healthc
.
2012
;
7
:
1
9
79.
Arriaga
AF
,
Bader
AM
,
Wong
JM
,
Lipsitz
SR
,
Berry
WR
,
Ziewacz
JE
,
Hepner
DL
,
Boorman
DJ
,
Pozner
CN
,
Smink
DS
,
Gawande
AA
: .
Simulation-based trial of surgical-crisis checklists.
N Engl J Med
.
2013
;
368
:
246
53
80.
Lipps
J
,
Meyers
L
,
Winfield
S
,
Durda
M
,
Yildiz
V
,
Kushelev
M
: .
Physiologically triggered digital cognitive aid facilitates crisis management in a simulated operating room: A randomized controlled study.
Simul Healthc
.
2017
;
12
:
370
6
81.
Low
D
,
Clark
N
,
Soar
J
,
Padkin
A
,
Stoneham
A
,
Perkins
GD
,
Nolan
J
: .
A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency.
Anaesthesia
.
2011
;
66
:
255
62
82.
Lelaidier
R
,
Balança
B
,
Boet
S
,
Faure
A
,
Lilot
M
,
Lecomte
F
,
Lehot
JJ
,
Rimmelé
T
,
Cejka
JC
: .
Use of a hand-held digital cognitive aid in simulated crises: The MAX randomized controlled trial.
Br J Anaesth
.
2017
;
119
:
1015
21
83.
Nelson McMillan
K
,
Rosen
MA
,
Shilkofski
NA
,
Bradshaw
JH
,
Saliski
M
,
Hunt
EA
: .
Cognitive aids do not prompt initiation of cardiopulmonary resuscitation in simulated pediatric cardiopulmonary arrests.
Simul Healthc
.
2018
;
13
:
41
6
84.
Nelson
KL
,
Shilkofski
NA
,
Haggerty
JA
,
Saliski
M
,
Hunt
EA
: .
The use of cognitive AIDS during simulated pediatric cardiopulmonary arrests.
Simul Healthc
.
2008
;
3
:
138
45
85.
McEvoy
MD
,
Hand
WR
,
Stiegler
MP
,
DiLorenzo
AN
,
Ehrenfeld
JM
,
Moran
KR
,
Lekowski
R
,
Nunnally
ME
,
Manning
EL
,
Shi
Y
,
Shotwell
MS
,
Gupta
RK
,
Corey
JM
,
Schell
RM
: .
A smartphone-based decision support tool improves test performance concerning application of the guidelines for managing regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy.
Anesthesiology
.
2016
;
124
:
186
98
86.
Hand
WR
,
Bridges
KH
,
Stiegler
MP
,
Schell
RM
,
DiLorenzo
AN
,
Ehrenfeld
JM
,
Nietert
PJ
,
McEvoy
MD
: .
Effect of a cognitive aid on adherence to perioperative assessment and management guidelines for the cardiac evaluation of noncardiac surgical patients.
Anesthesiology
.
2014
;
120
:
1339
49, quiz 134953
87.
MacDougall
N
,
Dong
F
,
Broussard
L
,
Comunale
ME
: .
Preventing mistransfusions: An evaluation of institutional knowledge and a response.
Anesth Analg
.
2018
;
126
:
247
51
88.
Russ
S
,
Rout
S
,
Sevdalis
N
,
Moorthy
K
,
Darzi
A
,
Vincent
C
: .
Do safety checklists improve teamwork and communication in the operating room? A systematic review.
Ann Surg
.
2013
;
258
:
856
71
89.
Augoustides
JG
,
Atkins
J
,
Kofke
WA
: .
Much ado about checklists: Who says I need them and who moved my cheese?
Anesth Analg
.
2013
;
117
:
1037
8
90.
Webster
CS
: .
Cognitive aids, checklists and mental models.
Anaesthesia
.
2017
;
72
:
1041
2
91.
ACGME Program Requirements for Graduate Medical Education in Anesthesiology 2018 edition
Accreditation Council for Graduate Medical Education
92.
The Anesthesiology Milestone Project, The Accreditation Council for Graduate Medical Education and The American Board of Anesthesiology.
2015
.
93.
Goldhaber-Fiebert
SN
,
Lei
V
,
Nandagopal
K
,
Bereknyei
S
: .
Emergency manual implementation: Can brief simulation-based or staff trainings increase familiarity and planned clinical use?
Jt Comm J Qual Patient Saf
.
2015
;
41
:
212
20
94.
Alidina
S
,
Goldhaber-Fiebert
SN
,
Hannenberg
AA
,
Hepner
DL
,
Singer
SJ
,
Neville
BA
,
Sachetta
JR
,
Lipsitz
SR
,
Berry
WR
: .
Factors associated with the use of cognitive aids in operating room crises: A cross-sectional study of US hospitals and ambulatory surgical centers.
Implement Sci
.
2018
;
13
:
50
95.
Huang
J
,
Wu
J
,
Dai
C
,
Zhang
X
,
Ju
H
,
Chen
Y
,
Zhang
C
,
Ye
F
,
Tan
Y
,
Zong
Y
,
Liu
T
: .
Use of emergency manuals during actual critical events in China: A multi-institutional study.
Simul Healthc
.
2018
;
13
:
253
60
96.
McEvoy
MD
,
Field
LC
,
Moore
HE
,
Smalley
JC
,
Nietert
PJ
,
Scarbrough
SH
: .
The effect of adherence to ACLS protocols on survival of event in the setting of in-hospital cardiac arrest.
Resuscitation
.
2014
;
85
:
82
7
97.
van Klei
WA
,
Hoff
RG
,
van Aarnhem
EE
,
Simmermacher
RK
,
Regli
LP
,
Kappen
TH
,
van Wolfswinkel
L
,
Kalkman
CJ
,
Buhre
WF
,
Peelen
LM
: .
Effects of the introduction of the WHO “Surgical Safety Checklist” on in-hospital mortality: A cohort study.
Ann Surg
.
2012
;
255
:
44
9
98.
Gaba
DM
: .
Perioperative cognitive aids in anesthesia: What, who, how, and why bother?
Anesth Analg
.
2013
;
117
:
1033
6
99.
Cook
DA
,
Levinson
AJ
,
Garside
S
,
Dupras
DM
,
Erwin
PJ
,
Montori
VM
: .
Internet-based learning in the health professions: A meta-analysis.
JAMA
.
2008
;
300
:
1181
96
100.
Tarpada
SP
,
Hsueh
WD
,
Gibber
MJ
: .
Resident and student education in otolaryngology: A 10-year update on e-learning.
Laryngoscope
.
2017
;
127
:
E219
24
101.
De Leeuw
RA
,
Westerman
M
,
Nelson
E
,
Ket
JC
,
Scheele
F
: .
Quality specifications in postgraduate medical e-learning: An integrative literature review leading to a postgraduate medical e-learning model.
BMC Med Educ
.
2016
;
16
:
168
102.
Maertens
H
,
Madani
A
,
Landry
T
,
Vermassen
F
,
Van Herzeele
I
,
Aggarwal
R
: .
Systematic review of e-learning for surgical training.
Br J Surg
.
2016
;
103
:
1428
37
103.
Reeves
S
,
Fletcher
S
,
McLoughlin
C
,
Yim
A
,
Patel
KD
: .
Interprofessional online learning for primary healthcare: Findings from a scoping review.
BMJ Open
.
2017
;
7
:
e016872
104.
Cook
DA
: .
Where are we with web-based learning in medical education?
Med Teach
.
2006
;
28
:
594
8
105.
Taveira-Gomes
T
,
Ferreira
P
,
Taveira-Gomes
I
,
Severo
M
,
Ferreira
MA
: .
What are we looking for in computer-based learning interventions in medical education? A systematic review.
J Med Internet Res
.
2016
;
18
:
e204
106.
Simpson
D
,
Marcdante
K
,
Souza
KH
,
Anderson
A
,
Holmboe
E
: .
Job roles of the 2025 medical educator.
J Grad Med Educ
.
2018
;
10
:
243
6
107.
Mitchell
JD
,
Matyal
R
:
Teaching Transesophageal Echocardiography, Education in Anesthesia: How to Deliver the Best Learning Experience
, 1st edition. Edited by
Bowe
EA
,
Schell
RM
,
Dilorenzo
AN
.
Cambridge
,
Cambridge University Press
,
2018
, pp
121
130
108.
McCormick
TJ
,
Miller
EC
,
Chen
R
,
Naik
VN
: .
Acquiring and maintaining point-of-care ultrasound (POCUS) competence for anesthesiologists.
Can J Anaesth
.
2018
;
65
:
427
36
109.
Fisher
J
,
Viscusi
R
,
Ratesic
A
,
Johnstone
C
,
Kelley
R
,
Tegethoff
AM
,
Bates
J
,
Situ-Lacasse
EH
,
Adamas-Rappaport
WJ
,
Amini
R
: .
Clinical skills temporal degradation assessment in undergraduate medical education.
J Adv Med Educ Prof
.
2018
;
6
:
1
5
110.
Barsuk
JH
,
Cohen
ER
,
Feinglass
J
,
McGaghie
WC
,
Wayne
DB
: .
Residents’ procedural experience does not ensure competence: A research synthesis.
J Grad Med Educ
.
2017
;
9
:
201
8
111.
Jensen
JK
,
Dyre
L
,
Jørgensen
ME
,
Andreasen
LA
,
Tolsgaard
MG
: .
Simulation-based point-of-care ultrasound training: A matter of competency rather than volume.
Acta Anaesthesiol Scand
.
2018
;
62
:
811
9
112.
Matyal
R
,
Mitchell
JD
,
Hess
PE
,
Chaudary
B
,
Bose
R
,
Jainandunsing
JS
,
Wong
V
,
Mahmood
F
: .
Simulator-based transesophageal echocardiographic training with motion analysis: A curriculum-based approach.
Anesthesiology
.
2014
;
121
:
389
99
113.
Matyal
R
,
Montealegre-Gallegos
M
,
Mitchell
JD
,
Kim
H
,
Bergman
R
,
Hawthorne
KM
,
O’Halloran
D
,
Wong
V
,
Hess
PE
,
Mahmood
F
: .
Manual skill acquisition during transesophageal echocardiography simulator training of cardiology fellows: A kinematic assessment.
J Cardiothorac Vasc Anesth
.
2015
;
29
:
1504
10
114.
Bick
JS
,
Wanderer
JP
,
Myler
CS
,
Shaw
AD
,
McEvoy
MD
: .
Standard setting for clinical performance of basic perioperative transesophageal echocardiography: Moving beyond the written test.
Anesthesiology
.
2017
;
126
:
718
28
115.
Matyal
R
,
Mahmood
F
,
Knio
ZO
,
Jones
S
,
Yeh
L
,
Amir
R
,
Bose
R
,
Mitchell
JD
: .
Evaluation of the quality of transesophageal echocardiography images and verification of proficiency.
Echo Res Pract
.
2018
;
5
:
89
95
116.
Rammell
J
,
Matthan
J
,
Gray
M
,
Bookless
LR
,
Nesbitt
CI
,
Rodham
P
,
Moss
J
,
Stansby
G
,
Phillips
AW
: .
Asynchronous unsupervised video-enhanced feedback as effective as direct expert feedback in the long-term retention of practical clinical skills: Randomised trial comparing 2 feedback methods in a cohort of novice medical students.
J Surg Educ
.
2018
;
75
:
1463
70
117.
Phillips
AW
,
Matthan
J
,
Bookless
LR
,
Whitehead
IJ
,
Madhavan
A
,
Rodham
P
,
Porter
ALR
,
Nesbitt
CI
,
Stansby
G
: .
Individualised expert feedback is not essential for improving basic clinical skills performance in novice learners: A randomized trial.
J Surg Educ
.
2017
;
74
:
612
20
118.
Sewell
JL
,
Boscardin
CK
,
Young
JQ
,
Ten Cate
O
,
O’Sullivan
PS
: .
Learner, patient, and supervisor features are associated with different types of cognitive load during procedural skills training: Implications for teaching and instructional design.
Acad Med
.
2017
;
92
:
1622
31
119.
Van Merrienboer
JJG
,
Kester
L
,
Paas
F
: .
Teaching complex rather than simple tasks: Balancing intrinsic and germane load to enhance transfer of learning.
Appl Cogn Psychol
.
2006
;
20
:
343
52
120.
van Merriënboer
JJ
,
Sweller
J
: .
Cognitive load theory in health professional education: Design principles and strategies.
Med Educ
.
2010
;
44
:
85
93
121.
Issenberg
SB
,
McGaghie
WC
,
Petrusa
ER
,
Lee Gordon
D
,
Scalese
RJ
: .
Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review.
Med Teach
.
2005
;
27
:
10
28
122.
Motola
I
,
Devine
LA
,
Chung
HS
,
Sullivan
JE
,
Issenberg
SB
: .
Simulation in healthcare education: A best evidence practical guide. AMEE Guide No. 82.
Med Teach
.
2013
;
35
:
e1511
30
123.
Ziv
A
,
Wolpe
PR
,
Small
SD
,
Glick
S
: .
Simulation-based medical education: An ethical imperative.
Acad Med
.
2003
;
78
:
783
8
124.
Madenci
AL
,
Solis
CV
,
de Moya
MA
: .
Central venous access by trainees: A systematic review and meta-analysis of the use of simulation to improve success rate on patients.
Simul Healthc
.
2014
;
9
:
7
14
125.
Komasawa
N
,
Berg
BW
: .
Simulation-based airway management training for anesthesiologists - A brief review of its essential role in skills training for clinical competency.
J Educ Perioper Med
.
2017
;
19
:
E612
126.
Bortman
J
,
Baribeau
Y
,
Jeganathan
J
,
Amador
Y
,
Mahmood
F
,
Shnider
M
,
Ahmed
M
,
Hess
P
,
Matyal
R
: .
Improving clinical proficiency using a 3-dimensionally printed and patient-specific thoracic spine model as a haptic task trainer.
Reg Anesth Pain Med
.
2018
;
43
:
819
24
127.
Nagendran
M
,
Toon
CD
,
Davidson
BR
,
Gurusamy
KS
: .
Laparoscopic surgical box model training for surgical trainees with no prior laparoscopic experience.
Cochrane Database Syst Rev
.
2014
, pp
CD010479
128.
Raj
D
,
Williamson
RM
,
Young
D
,
Russell
D
: .
A simple epidural simulator: A blinded study assessing the ‘feel’ of loss of resistance in four fruits.
Eur J Anaesthesiol
.
2013
;
30
:
405
8
129.
Nachshon
A
,
Mitchell
JD
,
Mueller
A
,
Banner-Goodspeed
VM
,
McSparron
JI
: .
Expert evaluation of a chicken tissue-based model for teaching ultrasound-guided central venous catheter insertion.
J Educ Perioper Med
.
2017
;
19
:
E503
130.
Sadideen
H
,
Hamaoui
K
,
Saadeddin
M
,
Kneebone
R
: .
Simulators and the simulation environment: Getting the balance right in simulation-based surgical education.
Int J Surg
.
2012
;
10
:
458
62
131.
Goolsby
CA
,
Goodwin
TL
,
Vest
RM
: .
Hybrid simulation improves medical student procedural confidence during EM clerkship.
Mil Med
.
2014
;
179
:
1223
7
132.
Mashari
A
,
Montealegre-Gallegos
M
,
Jeganathan
J
,
Yeh
L
,
Qua Hiansen
J
,
Meineri
M
,
Mahmood
F
,
Matyal
R
: .
Low-cost three-dimensional printed phantom for neuraxial anesthesia training: Development and comparison to a commercial model.
PLoS One
.
2018
;
13
:
e0191664
133.
Huang
CC
,
Hsu
HC
,
Yang
LY
,
Chen
CH
,
Yang
YY
,
Chang
CC
,
Chuang
CL
,
Lee
WS
,
Lee
FY
,
Hwang
SJ
: .
Peer-assisted learning model enhances clinical clerk’s procedural skills.
J Chin Med Assoc
.
2018
;
81
:
747
53
134.
Piromchai
P
,
Avery
A
,
Laopaiboon
M
,
Kennedy
G
,
O’Leary
S
: .
Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat.
Cochrane Database Syst Rev
.
2015
, pp
CD010198
135.
Nagendran
M
,
Gurusamy
KS
,
Aggarwal
R
,
Loizidou
M
,
Davidson
BR
: .
Virtual reality training for surgical trainees in laparoscopic surgery.
Cochrane Database Syst Rev
.
2013
, pp
CD006575
136.
Yoganathan
S
,
Finch
DA
,
Parkin
E
,
Pollard
J
: .
360° virtual reality video for the acquisition of knot tying skills: A randomised controlled trial.
Int J Surg
.
2018
;
54
(
pt A
):
24
7
137.
Huang
CY
,
Thomas
JB
,
Alismail
A
,
Cohen
A
,
Almutairi
W
,
Daher
NS
,
Terry
MH
,
Tan
LD
: .
The use of augmented reality glasses in central line simulation: “See one, simulate many, do one competently, and teach everyone”.
Adv Med Educ Pract
.
2018
;
9
:
357
63
138.
Izard
SG
,
Juanes
JA
,
García Peñalvo
FJ
,
Estella
JMG
,
Ledesma
MJS
,
Ruisoto
P
: .
Virtual reality as an educational and training tool for medicine.
J Med Syst
.
2018
;
42
:
50
139.
Rochlen
LR
,
Levine
R
,
Tait
AR
: .
First-person point-of-view-augmented reality for central line insertion training: A usability and feasibility study.
Simul Healthc
.
2017
;
12
:
57
62
140.
Mahmood
F
,
Mahmood
E
,
Dorfman
RG
,
Mitchell
J
,
Mahmood
FU
,
Jones
SB
,
Matyal
R
: .
Augmented reality and ultrasound education: Initial experience.
J Cardiothorac Vasc Anesth
.
2018
;
32
:
1363
7
141.
Bose
RR
,
Matyal
R
,
Warraich
HJ
,
Summers
J
,
Subramaniam
B
,
Mitchell
J
,
Panzica
PJ
,
Shahul
S
,
Mahmood
F
: .
Utility of a transesophageal echocardiographic simulator as a teaching tool.
J Cardiothorac Vasc Anesth
.
2011
;
25
:
212
5
142.
Mitchell
JD
,
Mahmood
F
,
Bose
R
,
Hess
PE
,
Wong
V
,
Matyal
R
: .
Novel, multimodal approach for basic transesophageal echocardiographic teaching.
J Cardiothorac Vasc Anesth
.
2014
;
28
:
800
9
143.
McGaghie
WC
,
Barsuk
JH
,
Wayne
DB
: .
AM last page: Mastery learning with deliberate practice in medical education.
Acad Med
.
2015
;
90
:
1575
144.
Yudkowsky
R
,
Park
YS
,
Lineberry
M
,
Knox
A
,
Ritter
EM
: .
Setting mastery learning standards.
Acad Med
.
2015
;
90
:
1495
500
145.
Ericsson
KA
,
Krampe
RT
,
Teschromer
C
: .
The role of deliberate practice in the acquisition of expert performance.
Psychol Rev
.
1993
;
100
:
363
406
146.
McGaghie
WC
,
Issenberg
SB
,
Barsuk
JH
,
Wayne
DB
: .
A critical review of simulation-based mastery learning with translational outcomes.
Med Educ
.
2014
;
48
:
375
85
147.
McGaghie
WC
,
Harris
IB
: .
Learning theory foundations of simulation-based mastery learning.
Simul Healthc
.
2018
;
13
(
3S suppl 1
):
15
20
148.
Griswold-Theodorson
S
,
Ponnuru
S
,
Dong
C
,
Szyld
D
,
Reed
T
,
McGaghie
WC
: .
Beyond the simulation laboratory: A realist synthesis review of clinical outcomes of simulation-based mastery learning.
Acad Med
.
2015
;
90
:
1553
60
149.
Bisgaard
CH
,
Rubak
SLM
,
Rodt
SA
,
Petersen
JAK
,
Musaeus
P
: .
The effects of graduate competency-based education and mastery learning on patient care and return on investment: A narrative review of basic anesthetic procedures.
BMC Med Educ
.
2018
;
18
:
154
150.
Alsaad
AA
,
Bhide
VY
,
Moss
JL
Jr
,
Silvers
SM
,
Johnson
MM
,
Maniaci
MJ
: .
Central line proficiency test outcomes after simulation training versus traditional training to competence.
Ann Am Thorac Soc
.
2017
;
14
:
550
4
151.
Cox
T
,
Seymour
N
,
Stefanidis
D
: .
Moving the needle: Simulation’s impact on patient outcomes.
Surg Clin North Am
.
2015
;
95
:
827
38
152.
Andersen
SAW
,
Konge
L
,
Sorensen
MS
: .
The effect of distributed virtual reality simulation training on cognitive load during subsequent dissection training.
Med Teach
.
2018
;
40
:
1
6
153.
Haji
FA
,
Cheung
JJ
,
Woods
N
,
Regehr
G
,
de Ribaupierre
S
,
Dubrowski
A
: .
Thrive or overload? The effect of task complexity on novices’ simulation-based learning.
Med Educ
.
2016
;
50
:
955
68
154.
Stefanidis
D
,
Walters
KC
,
Mostafavi
A
,
Heniford
BT
: .
What is the ideal interval between training sessions during proficiency-based laparoscopic simulator training?
Am J Surg
.
2009
;
197
:
126
9
155.
Andersen
SA
,
Konge
L
,
Cayé-Thomasen
P
,
Sørensen
MS
: .
Learning curves of virtual mastoidectomy in distributed and massed practice.
JAMA Otolaryngol Head Neck Surg
.
2015
;
141
:
913
8
156.
Patocka
C
,
Khan
F
,
Dubrovsky
AS
,
Brody
D
,
Bank
I
,
Bhanji
F
: .
Pediatric resuscitation training-instruction all at once or spaced over time?
Resuscitation
.
2015
;
88
:
6
11
157.
Shea
CH
,
Lai
Q
,
Black
C
,
Park
JH
: .
Spacing practice sessions across days benefits the learning of motor skills.
Hum Mov Sci
.
2000
;
19
:
737
60
158.
Kantak
SS
,
Sullivan
KJ
,
Fisher
BE
,
Knowlton
BJ
,
Winstein
CJ
: .
Neural substrates of motor memory consolidation depend on practice structure.
Nat Neurosci
.
2010
;
13
:
923
5
159.
Kromann
CB
,
Bohnstedt
C
,
Jensen
ML
,
Ringsted
C
: .
The testing effect on skills learning might last 6 months.
Adv Health Sci Educ Theory Pract
.
2010
;
15
:
395
401
160.
Shakil
O
,
Mahmood
B
,
Matyal
R
,
Jainandunsing
JS
,
Mitchell
J
,
Mahmood
F
: .
Simulation training in echocardiography: The evolution of metrics.
J Cardiothorac Vasc Anesth
.
2013
;
27
:
1034
40
161.
Mitchell
JD
,
Montealegre-Gallegos
M
,
Mahmood
F
,
Owais
K
,
Wong
V
,
Ferla
B
,
Chowdhury
S
,
Nachshon
A
,
Doshi
R
,
Matyal
R
: .
Multimodal perioperative ultrasound course for interns allows for enhanced acquisition and retention of skills and knowledge.
A A Case Rep
.
2015
;
5
:
119
23
162.
Ramsingh
D
,
Alexander
B
,
Le
K
,
Williams
W
,
Canales
C
,
Cannesson
M
: .
Comparison of the didactic lecture with the simulation/model approach for the teaching of a novel perioperative ultrasound curriculum to anesthesiology residents.
J Clin Anesth
.
2014
;
26
:
443
54
163.
Rebel
A
,
DiLorenzo
AN
,
Fragneto
RY
,
Dority
JS
,
Rose
G
,
Nguyen
D
,
Hassan
ZU
,
Schell
RM
: .
A competitive objective structured clinical examination event to generate an objective assessment of anesthesiology resident skills development.
A A Case Rep
.
2016
;
6
:
313
9
164.
Mitchell
JD
,
Amir
R
,
Montealegre-Gallegos
M
,
Mahmood
F
,
Shnider
M
,
Mashari
A
,
Yeh
L
,
Bose
R
,
Wong
V
,
Hess
P
,
Amador
Y
,
Jeganathan
J
,
Jones
SB
,
Matyal
R
: .
Summative objective structured clinical examination assessment at the end of anesthesia residency for perioperative ultrasound.
Anesth Analg
.
2018
;
126
:
2065
8
165.
Ramsingh
D
,
Rinehart
J
,
Kain
Z
,
Strom
S
,
Canales
C
,
Alexander
B
,
Capatina
A
,
Ma
M
,
Le
KV
,
Cannesson
M
: .
Impact assessment of perioperative point-of-care ultrasound training on anesthesiology residents.
Anesthesiology
.
2015
;
123
:
670
82
166.
Wali
E
,
Pinto
JM
,
Cappaert
M
,
Lambrix
M
,
Blood
AD
,
Blair
EA
,
Small
SD
: .
Teaching professionalism in graduate medical education: What is the role of simulation?
Surgery
.
2016
;
160
:
552
64
167.
Birden
H
,
Glass
N
,
Wilson
I
,
Harrison
M
,
Usherwood
T
,
Nass
D
: .
Teaching professionalism in medical education: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 25.
Med Teach
.
2013
;
35
:
e1252
66
168.
Lockman
JL
,
Schwartz
AJ
,
Cronholm
PF
: .
Working to define professionalism in pediatric anesthesiology: A qualitative study of domains of the expert pediatric anesthesiologist as valued by interdisciplinary stakeholders.
Paediatr Anaesth
.
2017
;
27
:
137
46
169.
Wynia
MK
,
Papadakis
MA
,
Sullivan
WM
,
Hafferty
FW
: .
More than a list of values and desired behaviors: A foundational understanding of medical professionalism.
Acad Med
.
2014
;
89
:
712
4
170.
Hafferty
FW
: .
Professionalism–The next wave.
N Engl J Med
.
2006
;
355
:
2151
2
171.
Wilkinson
TJ
,
Wade
WB
,
Knock
LD
: .
A blueprint to assess professionalism: Results of a systematic review.
Acad Med
.
2009
;
84
:
551
8
172.
Kelly
AM
,
Mullan
PB
: .
Designing a curriculum for professionalism and ethics within radiology: Identifying challenges and expectations.
Acad Radiol
.
2018
;
25
:
610
8
173.
Chestnut
DH
: .
On the road to professionalism.
Anesthesiology
.
2017
;
126
:
780
6
174.
Irby
DM
,
Hamstra
SJ
: .
Parting the clouds: Three professionalism frameworks in medical education.
Acad Med
.
2016
;
91
:
1606
11
175.
Bryden
P
,
Ginsburg
S
,
Kurabi
B
,
Ahmed
N
: .
Professing professionalism: Are we our own worst enemy? Faculty members’ experiences of teaching and evaluating professionalism in medical education at one school.
Acad Med
.
2010
;
85
:
1025
34
176.
Stern
DT
,
Papadakis
M
: .
The developing physician–Becoming a professional.
N Engl J Med
.
2006
;
355
:
1794
9
177.
Cruess
RL
: .
Teaching professionalism: Theory, principles, and practices.
Clin Orthop Relat Res
.
2006
;
449
:
177
85
178.
Nestel
D
,
Tierney
T
: .
Role-play for medical students learning about communication: Guidelines for maximising benefits.
BMC Med Educ
.
2007
;
7
:
3
179.
Baile
WF
,
Blatner
A
: .
Teaching communication skills: Using action methods to enhance role-play in problem-based learning.
Simul Healthc
.
2014
;
9
:
220
7
180.
Bosse
HM
,
Schultz
JH
,
Nickel
M
,
Lutz
T
,
Möltner
A
,
Jünger
J
,
Huwendiek
S
,
Nikendei
C
: .
The effect of using standardized patients or peer role play on ratings of undergraduate communication training: A randomized controlled trial.
Patient Educ Couns
.
2012
;
87
:
300
6
181.
Joyner
B
,
Young
L
: .
Teaching medical students using role play: Twelve tips for successful role plays.
Med Teach
.
2006
;
28
:
225
9
182.
Stobbs
N
: .
Role-play without humiliation: Is it possible?
Clin Teach
.
2015
;
12
:
128
30
183.
Dong
T
,
Kelly
W
,
Hays
M
,
Berman
NB
,
Durning
SJ
: .
An investigation of professionalism reflected by student comments on formative virtual patient encounters.
BMC Med Educ
.
2017
;
17
:
3
184.
Goss
BD
,
Ryan
AT
,
Waring
J
,
Judd
T
,
Chiavaroli
NG
,
O’Brien
RC
,
Trumble
SC
,
McColl
GJ
: .
Beyond selection: The use of situational judgement tests in the teaching and assessment of professionalism.
Acad Med
.
2017
;
92
:
780
4
185.
Riveros
R
,
Kimatian
S
,
Castro
P
,
Dhumak
V
,
Honar
H
,
Mascha
EJ
,
Sessler
DI
: .
Multisource feedback in professionalism for anesthesia residents.
J Clin Anesth
.
2016
;
34
:
32
40
186.
Gaiser
RR
: .
The teaching of professionalism during residency: Why it is failing and a suggestion to improve its success.
Anesth Analg
.
2009
;
108
:
948
54
187.
Dorotta
I
,
Staszak
J
,
Takla
A
,
Tetzlaff
JE
: .
Teaching and evaluating professionalism for anesthesiology residents.
J Clin Anesth
.
2006
;
18
:
148
60
188.
Mitchell
JD
,
Ku
C
,
Diachun
CAB
,
DiLorenzo
A
,
Lee
DE
,
Karan
S
,
Wong
V
,
Schell
RM
,
Brzezinski
M
,
Jones
SB
: .
Enhancing feedback on professionalism and communication skills in anesthesia residency programs.
Anesth Analg
.
2017
;
125
:
620
31
189.
Denson
JS
,
Abrahamson
S
: .
A computer-controlled patient simulator.
JAMA
.
1969
;
208
:
504
8
190.
Gaba
DM
: .
The future vision of simulation in health care.
Qual Saf Health Care
.
2004
;
13
(
suppl 1
):
i2
10
191.
Isaak
RS
,
Chen
F
,
Arora
H
,
Martinelli
SM
,
Zvara
DA
,
Stiegler
MP
: .
A descriptive survey of anesthesiology residency simulation programs: How are programs preparing residents for the New American Board of Anesthesiology APPLIED Certification Examination?
Anesth Analg
.
2017
;
125
:
991
8
192.
Hastie
MJ
,
Spellman
JL
,
Pagano
PP
,
Hastie
J
,
Egan
BJ
: .
Designing and implementing the objective structured clinical examination in anesthesiology.
Anesthesiology
.
2014
;
120
:
196
203
193.
McIndoe
A
: .
High stakes simulation in anaesthesia.
Continuing Educ Anaesth Crit Care Pain
.
2012
;
12
:
268
73
194.
Ben-Menachem
E
,
Ezri
T
,
Ziv
A
,
Sidi
A
,
Brill
S
,
Berkenstadt
H
: .
Objective Structured Clinical Examination-based assessment of regional anesthesia skills: The Israeli National Board Examination in Anesthesiology experience.
Anesth Analg
.
2011
;
112
:
242
5
195.
ABA News March 2013, The American Board of Anesthesiology.
2013
.
196.
Rathmell
JP
,
Lien
C
,
Harman
A
: .
Objective structured clinical examination and board certification in anesthesiology.
Anesthesiology
.
2014
;
120
:
4
6
197.
Rebel
A
,
Hester
DL
,
DiLorenzo
A
,
McEvoy
MD
,
Schell
RM
: .
Beyond the “E” in OSCE.
Anesth Analg
.
2018
;
127
:
1092
6
198.
Isaak
RS
,
Chen
F
,
Martinelli
SM
,
Arora
H
,
Zvara
DA
,
Hobbs
G
,
Stiegler
MP
: .
Validity of simulation-based assessment for Accreditation Council for Graduate Medical Education Milestone Achievement.
Simul Healthc
.
2018
;
13
:
201
10
199.
Collins
AB
,
Chen
L-l
: .
Simulation and safety.
ASA Monitor
.
2015
;
79
:
50
2
200.
Steadman
RH
,
Burden
AR
,
Huang
YM
,
Gaba
DM
,
Cooper
JB
: .
Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program.
Anesthesiology
.
2015
;
122
:
1154
69
201.
Hyman
SA
,
Shotwell
MS
,
Michaels
DR
,
Han
X
,
Card
EB
,
Morse
JL
,
Weinger
MB
: .
A survey evaluating burnout, health status, depression, reported alcohol and substance use, and social support of anesthesiologists.
Anesth Analg
.
2017
;
125
:
2009
18
202.
Martinelli
SM
,
Chen
F
,
Hobbs
G
,
Chidgey
BA
,
Straube
LE
,
Zvara
D
,
Isaak
R
: .
The use of simulation to improve family understanding and support of anesthesia providers.
Cureus
.
2018
;
10
:
e2262
203.
Cooper
AZ
,
Hsieh
G
,
Kiss
JE
,
Huang
GC
: .
Flipping out: Does the flipped classroom learning model work for GME?
J Grad Med Educ
.
2017
;
9
:
392
3
204.
Murray
E
: .
Challenges in educational research.
Med Educ
.
2002
;
36
:
110
2