Effect of Performance Deficiencies on Graduation and Board Certification Rates: A 10-yr Multicenter Study of Anesthesiology Residents

RESIDENCY programs have a responsibility to the public and to their residents to train competent medical specialists. To fulfill this mission, residency programs monitor the progress of their residents through supervision, evaluations, and examinations. When trainees are observed to experience personal, professional, or academic difficulties, the residency program intervenes to assist these residents.

Residents with performance concerns have been extensively studied.^1–22  The prevalence of residents with performance deficiencies has been reported between 6 and 26% across specialties including psychiatry, family medicine, internal medicine, and general surgery.^1,9,10,17  These residents have been referred to variously as “problem residents,” “residents in difficulty,” “troublesome residents,” “problem learners,” and “residents in trouble.”¹  We have chosen the term “residents in trouble,” since it conveys the attitude of concern for the performance and improvement of trainees, as we might say when our patients are “in trouble.”

The methods used to study residents in trouble vary. The majority of studies used surveys of program directors to establish the prevalence of residents with deficiencies, the most commonly deficient categories, and the remediation strategies. Results are subject to both response and recall bias. In addition, surveys typically report cross-sectional data, which are less useful in determining the proportion of residents who graduate and become board certified.

The 1999 landmark survey of the Association of Program Directors in Internal Medicine identified a 7% point prevalence of “problem residents” (defined by the American Board of Internal Medicine as “a trainee who demonstrates a significant enough problem that requires intervention by someone of authority”); 94% of programs had problem residents.¹⁸  The most frequently identified problems were medical knowledge, poor clinical judgment, and inefficient use of time. A 2012 update of this survey describing deficiencies in terms of Accreditation Council for Graduate Medical Education (ACGME) Core Competencies showed that the most commonly deficient ACGME Core Competencies were Medical Knowledge, Patient Care, and Interpersonal Communication.²⁰ 

The prevalence of residents in trouble has also been assessed through single-center longitudinal studies within one specialty, allowing determination of graduation and board certification rates. A single-center 10-yr longitudinal study of surgical residents reported that 21% of 115 residents performed poorly; 75% (18 of 24) of these residents graduated. Deficits clustered around medical knowledge and its application, personal problems, including health, and interpersonal skills/professionalism.²¹  A 25-yr retrospective analysis of 230 family medicine residents at a single site revealed a 9% prevalence of residents in trouble. The most common deficits identified were in medical knowledge, attitudinal problems, and interpersonal conflict.¹ 

Despite the considerable literature on “residents in trouble,” the effect of performance gaps on graduation and board certification has not been well characterized, including in anesthesiology. There are currently over 130 U.S. anesthesiology programs accredited by the ACGME. Each program reports resident physician performance to the American Board of Anesthesiology (ABA). The aim of this 10-yr multicenter study is to report the prevalence of unsatisfactory performance, characterize the nature of performance gaps, and examine their effect on graduation and board certification.

With local Institutional Review Board approval (University of California at Los Angeles, Los Angeles, California), we performed a multicenter retrospective review of anesthesiology residents’ files. Residents entering anesthesia year 1 (postgraduate year [PGY]-2) training between 2000 and 2009 were included in the analysis. Data sources reviewed included: Clinical Competence Committee (CCC) reports; Residency Program Director memoranda; and the semiannual American Board of Anesthesiology Training Reports (ABA Record of Training/CCC Report) in force during the study period (table 1).

The CCC is an ABA- and ACGME-mandated body composed of faculty members who meet regularly (two to four times per year) to assess resident performance. The role of the CCC is to track and evaluate resident performance in order to follow the progress of residents in the program and advise the Program Director regarding suitability for graduation and independent practice.

The ABA Record of Training/CCC Report assesses resident performance based on seven “Essential Attributes” (qualities of character and professionalism deemed “essential” by the ABA; these attributes map to elements of the ACGME Core Competencies of Professionalism, Patient Care, and Interpersonal and Communication Skills), six ACGME Core Competencies, and four Clinical Skills outlined in table 1. The Chair of the CCC and Program Director are responsible for reporting resident performance to the ABA.

Each CCC at the respective residency programs established and implemented its own specific policies and procedures to evaluate the performance and progress of trainees. A pattern of poor evaluations resulted in CCC action. CCC actions included counseling, letter of concern, Unsatisfactory ABA Training Report, probation, and, if deficiencies were not resolved after successive attempts at remediation, forced resignation or dismissal. Of the range of CCC actions, only the ABA Training Reports could be compared across programs in this study. The criteria driving other actions, for example, placing a resident on probation, are unique to each institution and were therefore not amenable to comparison. Each program devised its own approach to remediation in general, which was tailored to the unique needs of each of the referred residents.

The four study sites were Massachusetts General Hospital, Boston, Massachusetts; University of California, Los Angeles (serving as the coordinating site), Los Angeles, California; University of California, San Francisco, San Francisco, California; and University of Colorado School of Medicine, Denver, Colorado. During the period under review, there were three Program Directors and a single CCC Chair at University of California, Los Angeles; two Program Directors and a single CCC Chair at University of California, San Francisco; and a single Program Director and single CCC Chair at Massachusetts General Hospital and University of Colorado School of Medicine. All sites are ACGME-accredited anesthesiology residency programs.

Residents from the four training programs who commenced clinical anesthesia year 1 between 2000 and 2009 were included in the analysis, including residents who transferred into the program at the PGY-2 level or beyond (the years devoted exclusively to training in clinical anesthesia). Since reporting criteria differ between residents and interns, only PGY-2 through PGY-4 residents were included in the study. Performance reports through 2012 were reviewed in order to include data for the class entering in 2009. ABA reports through February 2016 were used to assess board certification.

The Program Director, CCC Chair, or their designees compiled data at each site. They reviewed source documents, which included ABA Training Reports, CCC minutes, e-mails, and individual resident files. Data were deidentified and abstracted into a single database that included year of entry into the residency program, program site, and, if applicable, year of the first CCC action, level of training at the time of CCC action, the type of action, and the ACGME Core Competencies of concern. In addition, if the resident received one or more Unsatisfactory designations on the ABA Record of Training/CCC Report, the database included a notation of the primary Unsatisfactory attribute or competency, in the judgment of the training program. Outcomes measured included whether the training program was completed (graduation) and whether ABA Board Certification was attained.

Descriptive statistics were summarized using frequencies and percentages in contingency table format. In order to test for site-to-site differences, or test other associations including types of deficiencies, actions, and outcomes, we used the Pearson chi-square test or Fisher exact test as appropriate. These tests were carried out using JMP Pro 12 (SAS Institute, USA).

For assessing binary outcomes (graduation or ABA certification), generalized logistic mixed effects models were constructed. Due to site-to-site variation discovered in our descriptive analysis, a random site effect was added to these models (tables 2 to 6). Mixed effects models were constructed using SAS 9.4 (SAS Institute).

In order to adjust for multiple comparisons, Bonferroni correction was used. P values less than 0.0036 were considered statistically significant after the adjustment (α 0.05/14 tests = adjusted α of 0.0036).

Given that the follow-up period was shorter for subjects who entered late in the study period, we analyzed time from graduation to board certification, and then performed a sensitivity analysis of the impact of the follow-up period on board certification rates.

The total number of PGY-2 through PGY-4 residents at each site ranged from 127 to 275 over the study period, with a total of 865 residents at all sites who participated in this study. Of the 865 residents in training at the four sites, 215 residents received a total of 405 CCC actions. Of these 215 residents, 118 residents received at least one Unsatisfactory designation on the ABA Record of Training/CCC Report. Across sites, the proportion of residents receiving CCC actions and Unsatisfactory designations varied considerably, from 13 to 51% for CCC actions and from 3 to 37% for Unsatisfactory designations.

There was no significant difference in overall graduation rates among the four sites (97 to 98%, P = 0.96, Fisher exact test, table 2). The proportion of entering residents who achieved board certification varied from 96 to 98% (P = 0.78, Fisher exact test) among the four residency programs.

When we compared residents who were the subject of any CCC action (n = 215) to residents who were not the subject of CCC actions (n = 650), we found a difference in their graduation rate (93 vs. 99%, respectively; P < 0.001; OR: 0.08, 95% CI: 0.03 to 0.23, after controlling for site, table 3). The board certification rate of residents who were subject to CCC action was also different from that of residents who did not receive CCC actions (89 vs. 99%, respectively; P < 0.001; OR: 0.04, 95% CI: 0.02 to 0.13, after controlling for site). Among residents receiving CCC actions who ultimately graduated (n = 199), 191 achieved board certification.

When residents who received one or more Unsatisfactory rating(s) on an ABA Record of Training/CCC Report (n = 118) were compared to residents who received all Satisfactory ratings (n = 747), there was a difference in the graduation rate (88 vs. 99%, respectively; P < 0.001; OR: 0.03, 95% CI: 0.01 to 0.10, after controlling for site) and board certification rate (85 vs. 99%, respectively, P < 0.001; OR: 0.04, 95% CI: 0.01 to 0.09, after controlling for site, table 4). Among residents receiving one or more Unsatisfactory ratings who graduated (n = 104), 100 achieved board certification.

The CCC might take one of several actions when confronted with a resident who was not meeting performance expectations. Of the 215 residents who received actions by the CCC, 204 received verbal counseling, 102 received written counseling, 118 received at least one grade of Unsatisfactory on the ABA Record of Training/CCC Report, and 24 were placed on probation according to the criteria of their institution.

The number of deficient ACGME Core Competencies varied among residents who were placed on probation. In descending order, the performance gaps were noted in Patient Care (n = 21), Professionalism (n = 19), Interpersonal and Communication Skills (n = 15), Medical Knowledge (n = 10), Practice-based Learning (n = 8), and Systems-based Practice (n = 5). Of the 24 residents placed on probation, 16 residents graduated (66%); of these, 15 residents went on to achieve board certification.

The overall graduation rate of residents receiving any Unsatisfactory designation was 88% (table 4). Of the total cohort of 865 residents, 11 residents received a primary Unsatisfactory rating in an Essential Attribute (has no documented abuse of alcohol or illegal use of drugs (n = 4), learns from experience, knows limits (n = 2), demonstrates honesty, integrity, reliability, and responsibility (n = 2), reacts to stressful situations in an appropriate manner (n = 2), and demonstrates high standards of ethical and moral behavior (n = 1), (table 5). Seven of these residents were placed on probation. Of the residents with Unsatisfactory ratings reported to the ABA, those who were lacking Essential Attributes were far less likely to graduate compared to those lacking in other categories (55 vs. 92% for those without a rating of Unsatisfactory in an Essential Attribute, P = 0.0013; OR: 0.09, 95% CI: 0.02 to 0.37, after controlling for site).

Of the residents with Unsatisfactory ratings reported to the ABA, those who were lacking Essential Attributes were far less likely to obtain ABA certification than those lacking in other categories (45 vs. 89%, respectively; P = 0.0007; OR: 0.08, 95% CI: 0.02 to 0.34, after controlling for site). The overall board certification rate among residents receiving any Unsatisfactory designation on the ABA Record of Training/CCC Report was 85%.

The CCC assessed the presence or absence of a deficiency in any ACGME Core Competency. Table 6 shows the number of residents by number of ACGME competencies deficient as well as the graduation and board certification rates for each group. For both outcomes (graduation and board certification), the rates decreased for residents with more deficiencies (P < 0.0001, after controlling for site).

Table 7 shows the number of residents across all four programs with deficiencies in an ACGME Core Competency. Medical Knowledge was the most commonly deficient competency (128 residents), followed by Patient Care (n = 78), Professionalism (n = 75), and Communication and Interpersonal Skills (n = 74). Practice-based Learning and Improvement and Systems-based Practice were the least commonly deficient competencies (n = 37 and n = 13, respectively).

The median time from graduation to certification for residents in trouble was 15 months (interquartile range: 10 to 20 months). All residents who ultimately became certified did so within 5 yr. Two graduates who were not board-certified were followed for less than 5 yr. All statistically significant findings remained significant after a sensitivity analysis was done to determine whether a change in board certification status of these two residents would affect our findings.

A variety of remediation methods were utilized at all four sites in order to support all residents in trouble. Each program provided a list of the remediation techniques used to address the various deficiencies. An aggregate list is presented in table 8.

Deficiencies in multiple ACGME Core Competencies threaten graduation and board certification. Multiple deficiencies significantly decrease the likelihood of graduation and board certification, particularly when more than three ACGME Core Competency categories are involved. Roughly 3% of our study population was deficient in four or more categories. Residents with deficiencies in fewer categories are likely to graduate.

Essential Attributes are essential. Although deficiencies in an Essential Attribute were limited to a small group of residents (1.3% of the study population), the graduation and board certification rates of these residents were significantly decreased. An Unsatisfactory rating in an Essential Attribute represents a similar threat to graduation as deficiencies in multiple competency categories. Board certification rates drop below 50% for residents with deficiencies in an Essential Attribute. Deficits in Essential Attributes are known to be challenging to remediate.¹  Substance abuse, which falls into the category of Essential Attribute deficiency, was found in 0.5% of the study population. In a large study that focused on substance abuse, the graduation and certification rates were significantly decreased, similar to the respective rates of residents with Essential Attribute deficiencies in this study.²² 

Graduation rates are high overall. Even residents in trouble are likely to graduate when deficiencies are isolated and do not involve Essential Attributes. Despite documented difficulties in performance during training, residents receiving CCC actions, with or without Unsatisfactory ratings, experienced a five percentage point drop in graduation rate compared to those without deficiencies.

Board certification rates are high overall. Even residents in trouble are likely to become board-certified when deficiencies are isolated and do not involve Essential Attributes. However, residents receiving CCC actions had lower board certification rates than graduation rates (89 vs. 93%, respectively).

The most common deficiency was in the ACGME category Medical Knowledge. A comparison of the frequency of deficiencies in ACGME Core Competencies shows that Medical Knowledge is by far the most common deficient competency, followed by, in approximately equal numbers, Patient Care, Professionalism, and Communication and Interpersonal Skills. This pattern may account for the overall high graduation and certification rates among our residents in trouble, as there are many readily available tools to supplement medical content knowledge.

The large variation in CCC actions and Unsatisfactory ratings between sites did not affect overall graduation or ABA certification rates. The proportion of residents who were referred to their respective CCCs varied considerably among the four residencies. We found a similar variation among the four sites in assigning Unsatisfactory ratings on ABA Training Reports. However, the programs did not differ from one another in overall graduation and board certification rates.

The variation in CCC referrals and Unsatisfactory ratings is related to the academic policy at Program D, which automatically assigns an Unsatisfactory in Medical Knowledge to residents who score below predetermined percentile levels on the In Training Examination. Using this algorithm results in a larger number of residents at Program D receiving Unsatisfactory ratings. Despite the differences between CCC practices at the various institutions, graduation and board certification rates are similar across all four programs.

Remediation was multipronged and individualized to each resident; various combinations of rewards (educational stipends, eligibility for Chief Resident) and punitive measures (extended training time, Unsatisfactory training reports) were used to motivate residents to address deficiencies. A number of similar remediation efforts were employed among the four sites and were tailored to the competency in question. For instance, medical knowledge deficiencies were addressed with required reading, lecture attendance, and use of additional standardized assessments (e.g., the Anesthesia Knowledge Test), while patient care deficiencies were addressed by directed clinical assignments. Faculty mentors initially addressed professionalism and communication lapses, while recurring problems frequently led to referrals to behavioral therapists. Generally, program directors and CCCs tried to identify deficiencies early, in order to initiate and escalate remediation as needed.

The prevalence of residents in trouble in the current study (25%) falls within the range previously reported (6 to 26%), although definitions of “residents in trouble” varied.^1,9,10,17  The most commonly deficient ACGME Core Competencies among our residents, Medical Knowledge, Patient Care, and Professionalism, were also consistent with previous findings.^1,20  The graduation and board certification rates among the anesthesiology residents with performance deficits in our study (93 and 89%, respectively) were also comparable to those found in the 2006 study of family medicine residents (90 and 86%, respectively).¹ 

Our study was limited by the accuracy and completeness of the records maintained by the residency offices at the four sites. Nevertheless, we were fortunate to have low turnover of residency program personnel during the study period, facilitating data retrieval. Although all the residents included in this study were assessed by the Program Director and CCC Chair (or their designees) using a consensus approach to evaluate the comprehensive records at each site (CCC records, ABA Training Reports, and individual resident files), the reduction of the rich narrative data on each resident to dichotomous variables in a database created the potential for oversimplification of the deficiencies.

Nearly half (44%) of residents with deficiencies in ACGME Core Competency categories had deficiencies in two or more categories, making it difficult to isolate the impact of a deficiency in a single competency on graduation or board certification. Despite the size of the cohort in this multicenter study, an assessment of interaction effects between deficiencies in two or more ACGME Core Competency categories was not feasible.

Our study may also suffer from selection bias. Although the residency programs in this study are large and geographically diverse, they are all university based and may not be representative of anesthesiology training programs as a whole.

Several residents who entered the study during the last several years of enrollment had a shorter follow-up period during which to become board-certified. After a sensitivity analysis designed to see if our results would change if they were to become certified, P values and CIs only changed slightly, and overall conclusions were not affected.

This study was not designed to provide data on the effectiveness of remediation methods. Programs typically employ several remediation strategies simultaneously, which limits assessments of individual remediation strategies.

The ACGME provides guidance to residency training programs regarding the structure and function of program CCCs.^23,24  Nevertheless, every CCC operates according to internally developed policies and procedures. Milestones have now been created in an effort to create a standardized assessment of all residents across programs; however, the ACGME milestones did not exist during the period of our study.

There was a wide variability in the proportion of residents receiving CCC actions at the four programs. This difference between programs did not affect overall graduation rates; however, it does indicate a lack of standardization in CCC practices across programs. The current study indicates that specialty-wide policies and procedures for CCCs might assist training programs in making comparisons across programs. Standard operating procedures for all CCCs would promote internal and external consistency among programs, maintain efficiency and quality control, and provide transparency. Organizational or regulatory bodies may be helpful in establishing best practices for CCCs.

The graduation and board certification rates we have reported may serve as a benchmark for other anesthesiology programs. Graduation and board certification rates were consistently high across the four programs. There was a small reduction in graduation and board certification rates for residents receiving CCC actions for ACGME Core Competencies; nevertheless, these residents were still more likely than not to graduate in good standing and achieve board certification, perhaps due to successful remediation strategies by the training programs. Notably, performance problems in Essential Attributes carry a poor prognosis and suggest that remediation efforts may not be satisfactory, or that it may not be possible to remediate these problems in anesthesiology training. Another risk factor for failure to graduate or to achieve board certification is the presence of deficiencies in multiple ACGME Core Competency categories. Although it is beyond the scope of our data to provide an analysis of the usefulness of specific remediation techniques, it would be helpful if future work in this area examined which remediation methods are most efficient and effective in correcting particular deficiencies.

The authors thank Erin Lacey, M.S., Massachusetts General Hospital, Boston, Massachusetts, and Allison Glover, B.A., University of Colorado School of Medicine, Denver, Colorado, for data collection and coordination.

This study was supported by the authors’ institutions: Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at University of California, Los Angeles, Los Angeles, California; Department of Anesthesia, Critical Care and Pain Medicine, Harvard Medical School, Massachusetts General Hospital, Boston, Massachusetts; Department of Anesthesia and Perioperative Care, University of California, San Francisco, San Francisco, California; and Department of Anesthesiology, University of Colorado School of Medicine, Denver, Colorado. Statistical analyses described were partially supported by National Institutes of Health/National Center for Advancing Translational Science UCLA Clinical and Translational Science Institute grant number UL1TR000124.

The authors declare no competing interests.