Anesthesiologist Board Certification and Patient Outcomes

All Pennsylvania Medicare Standard Analytic Files for general surgical and orthopedic DRGs in Pennsylvania between 1991 and 1994 (Medicare Part A data) for patients 65 yr or older were analyzed for 245 hospitals. The data included 217,440 cases. For each patient, we created a longitudinal record by appending all medical and surgical inpatient and outpatient physicians’ claims (Medicare Part B data) during that time interval. Data also included the American Hospital Association Annual Surveys for 1991–1993, and the Pennsylvania Health Care Cost Containment Council Hospital Discharge Data for 1991–1994.

We developed predictive models of outcome on a random sample of 50% of Medicare patients who underwent general surgical or orthopedic procedures in Pennsylvania between 1991 and 1994, and tested our results on the other 50%. Final models were fit using the full sample of patients whose care involved an anesthesiologist and for which board certification data were available.

Of the 217,440 individual patients, there were 194,430 cases involving anesthesiologists either personally performing or directing anesthesia care. Of these cases, board certification status and medical school graduation year were available for 144,883 cases. In Health Care Financing Administration billing records, the specialty code for anesthesiologist is denoted by “05.” Anesthesiologist designation did not imply board certification. We coded board certification as present if either the 1996 American Board of Medical Specialties file or the 1993 Medicare Physician file designated that the anesthesiologist was certified.

Death within 30 days from admission was determined from the Health Care Financing Administration Vital Status file. FTR was defined as the 30-day death rate in those who either developed a complication or died without a recorded complication. It can be expressed mathematically as FTR = D/(C + D|no C), or the number of patients who died (D) divided by the number of patients with complications (C) plus the number of patients who died without complications noted in the claims data (D|no C). 1,2Complications were identified using a set of 41 events defined by ICD-9-CM and CPT codes available from Health Care Financing Administration and described in our previous work. 4Complications in this study refer not only to anesthesia-related complications, but also to adverse occurrences noted any time during the hospitalization, and were not used as an outcome measure to assess quality. Our work has shown consistently that complications based on these definitions are a reflection of underlying patient severity of illness and are not a good indicator of the quality of patient care. 1,2,4–8 

Estimates of excess deaths per thousand were derived with a direct standardization approach using the full data set for all cases. 9Using the final fully adjusted model, the probability of death was estimated twice for each of the 144,883 patients in the study, once assuming each case was performed by one group of anesthesiologists and again assuming care was provided by another group of anesthesiologists. The resultant difference between the sum of the estimated probabilities of death, divided by the sample size, and multiplied by 1,000, provides the number of excess deaths per thousand cases comparing the two groups of anesthesiologists. The advantage of this standardization approach is that all patients are used for both estimates, hence reducing bias.

We developed logistic regression models to adjust for severity of illness and case mix, one for each outcome in the 50% random sample or “development” sample. 4Candidate variables were selected if significant at the 0.05 level on univariate analysis. DRG variables were split into DRG-principal procedure categories to produce more homogeneous risk groupings based on Haberman residuals 2,10,11and then included in each model. Each model included 49 DRG-principal procedure variables and 27 patient characteristics. A total of 25 interaction terms were included in the models, having been significant at the Bonferroni-adjusted 0.05 level. We validated the derived models on the remaining 50%, or “validation” sample. Coefficients were not statistically different between models derived in development and validation sets. Pearson correlation coefficients between predicted outcomes in the development set and the validation set were always greater than 0.93. Final models were constructed using both the development and validation data sets.

To account for hospital characteristics that may have influenced our results, we further adjusted our models using a list of 10 hospital characteristics (described later) previously reported by us and other investigators. 2–5Furthermore, we constructed an indicator variable for each hospital and analyzed results adjusted for each individual hospital in the logistic regression modeling. We also performed adjustments for each hospital using Mantel-Haenszel adjusted odds ratios (ORs). 12 

The rate of board certification is highly related to the medical school graduation year of the anesthesiologist. As has been described elsewhere, 13it takes approximately 10 yr from medical school graduation before the rate of board certification becomes stable (i.e. , 10 yr represents the interval during which most anesthesiologists in a cohort achieve certification). In our data set, as can be seen in table 1, 47% of anesthesiologists who were 10 yr or less from graduation were not board certified. We refer to this period as “early career.” In the cohort of 11–25 yr from graduation, the rate of non–board-certified anesthesiologists was only 14%. This period is referred to as the “midcareer” period. Finally, in the “late-career” phase, beyond 25 yr from graduation, the number of non–board-certified anesthesiologists increased to 25%, twice the midcareer noncertification rate, in part because the perceived need for certification in this older cohort was not the same as for more recent graduates. 14 

The pattern of certification rates across graduation-year cohorts prompted us to compare the outcomes of cases performed or directed by non–board-certified anesthesiologists who belong to a high board-certification rate cohort, the midcareer cohort, to the outcomes of all the other anesthesiologists who belong to cohorts with much lower rates of certification. If certification is a meaningful proxy for quality of care, it should be reflected by worse outcomes associated with the noncertified, midcareer group of anesthesiologists as compared with all other certified or noncertified anesthesiologists, since it is the midcareer anesthesiologists who have the “least excuse” for not being certified and who have the highest rate of certification. Cohorts with high rates of noncertification would not be expected to show differences in outcomes across certified and noncertified groups, because the noncertified groups would certainly comprise many good practitioners who either did not yet have time to become certified or who, many years ago, chose not to pursue certification because it was not deemed essential for career success.

Table 2describes patient case mix, and table 3displays patient characteristics that were present in at least 1% of the population across the two comparison groups in this study: group I (n = 8,894) consisted of cases performed or directed by non–board-certified anesthesiologists in midcareer, and group II (n = 135,989) consisted of all other anesthesiologist cases. Group II included board-certified anesthesiologists in a highly certified cohort or certified and noncertified anesthesiologists who graduated in a cohort of anesthesiologists who did not have a high rate of board certification. Two ORs are presented in table 3. The first is the unadjusted OR, and the second is the Mantel-Haenszel 12OR after adjusting for DRG category. Cases performed by non–board-certified anesthesiologists at midcareer (group I) had somewhat higher odds of having comorbitities associated with congestive heart failure, insulin-dependent diabetes, and admissions through the emergency department, and had lower odds of the comorbidity of cancer.

The distribution of hospital characteristics across the two comparison groups is shown in table 4. The distribution of hospital characteristics was different between the midcareer non–board-certified anesthesiologists and all other anesthesiologists. Hospital characteristics often associated with improved quality were consistently less evident in group I, the noncertified group, suggesting that these characteristics should be used to adjust for any overall differences in outcomes.

Unadjusted death and FTR rates were greater when cases were performed by midcareer non–board-certified anesthesiologists (table 5). Table 6shows the influence of anesthesia certification on outcome after results were adjusted for patient characteristics and interaction terms including demographic information, history variables, whether the patient was transferred from another acute care hospital, whether the patient was admitted from the emergency department, and DRG-procedure categories used for this study. As in the unadjusted model, mortality and FTR rates were higher when midcareer anesthesiologists lacked board certification. The adjusted ORs for death (OR^DEATH) and FTR (OR^FTR) were significantly increased (OR^DEATH= 1.14, P < 0.02; OR^FTR= 1.16, P < 0.02). Adding patient race to this model did not change these results, nor did adding MedisGroups admission severity score (a physiologic-based score) obtained from the Pennsylvania Health Care Cost Containment Council. 1,6,7,15,16 

Table 7displays the results of board certification when 10 hospital variables were included in the three outcomes models. Midcareer anesthesiologists without board certification were associated with greater death and FTR rates: (OR^DEATH= 1.13, P < 0.04; OR^FTR= 1.13, P < 0.04), corresponding to 3.8 excess deaths per 1,000 patients and 9.2 excess deaths per 1,000 patients with complications.

Table 8shows the “marginal” and “partial” effects of hospital characteristics and the lack of certification in midcareer anesthesiologists. The marginal analysis displays the influence of each variable separately, adjusting only for the patient characteristics in table 8. The partial analysis displays the fully adjusted model, simultaneously adjusting for all hospital and anesthesiologist variables. There were only three variables that showed a significant association with mortality in both the marginal model and the fully adjusted “partial” model. These variables were hospital size, nurse-to-bed ratio, and midcareer noncertified anesthesiologists.

When the MedisGroups severity score was added to this analysis, death and failure ORs were almost exactly the same as those determined without using the MedisGroups severity score (OR^DEATH= 1.12, P < 0.06; OR^FTR= 1.13, P < 0.04). The ORs were almost unchanged, yet the P  values were slightly less significant, in part because the sample size with MedisGroups score represented only 74.1% of the study population.

Midcareer anesthesiologists who lacked board certification tended to practice in different settings than those with certification (table 4). We sought to determine whether the higher mortality associated with lack of certification in midcareer was a result of worse performance by these uncertified anesthesiologists or possibly because of worse hospitals that allowed noncertified physicians to practice. In the previous analysis we adjusted for important hospital characteristics and found that the covariate for certification in midcareer remained significant, suggestive of a quality problem with midcareer noncertified anesthesiologists. We next performed a “fixed effects” adjustment, where we created an indicator or “dummy” variable for individual hospitals and tested whether the covariate for certification remained significant. We found that the mortality measures did not remain significant (OR^DEATH= 0.99, P < 0.85; OR^FTR= 0.96, P < 0.59).

The discordance between the results of our two hospital adjustment models can be better understood when we examine more closely the distribution of midcareer noncertified (group I) anesthesiologist cases among hospitals. We found that 144 of 218 hospitals had no group I cases. Of the remaining hospitals that had group I cases, only 43 performed more than 10% of their cases using group I (midcareer noncertified) anesthesiologists. We then constructed a model to predict the presence of an undirected midcareer anesthesiologist. We found that using the patient variables from the model in table 6and including dummy variables for hospitals, we could predict with high accuracy whether a patient received care from a group I anesthesiologist, with a model c statistic of 0.94. Hence, it is not surprising that the second method of hospital adjustment, using a fixed-effects approach, showed no detrimental effect on mortality from lack of certification, since the strong clustering could shift the elevated odds associated with increased mortality to the hospital, rather than the anesthesiologist.

To further attempt to sort out the influence of provider versus  hospital, we constructed a Mantel-Haenszel 12test to adjust the OR associated with group I anesthesiologists and mortality by each of the 74 hospitals that used group I anesthesiologists. This formed a test with 2 (death) × 2 (group) × 74 (hospital) cells. The resulting OR was 1.082 (P < 0.253). Although not significant, the trend was similar to significant results of the full-model OR reported in table 7.

Taken together, these results suggest that a portion of the increase in the death rate associated with group I cases may be related to other factors that are associated with the presence of these midcareer noncertified anesthesiologists. The high level of clustering between group I anesthesiologists and specific hospitals limits our ability to better distinguish between hospital and provider effects.

When a variable reflecting the number of anesthesia procedures per hospitalization was added to the model, we found the ORs for group I to be unchanged. Because hospital practice may be associated with variations in the delivery of anesthesia care, we also asked whether controlling for personally performing a case versus  medically directing an anesthesia care team had an impact on these results. We found it did not. We further asked if adding variables denoting the size of the metropolitan area would account for the observed differences in outcome. Adjusting for the 10 hospital variables and for 5 levels of population size from rural to metropolitan areas greater than one million, we found very little difference in results (OR^DEATH= 1.11, P < 0.07; OR^FTR= 1.11, P < 0.09).

We also explored whether our results would be similar if, in a separate analysis, we compared group I midcareer noncertified anesthesiologists to midcareer board-certified anesthesiologists. Such a comparison reduced the number of cases from 144,883 to 69,516. Using the same fully adjusted model as reported in table 7, we found very similar results (OR^DEATH= 1.14, P < 0.028; OR^FTR= 1.15, P < 0.030). To further attempt to sort out the influence of provider versus  hospital, we again constructed a Mantel-Haenszel 12test to adjust the OR associated with group I anesthesiologists and mortality by each of the 74 hospitals that used group I anesthesiologists. This formed a test with 2 (death) × 2 (group) × 74 (hospital) cells among the 69,516 cases. The resulting OR was 1.15 (P < 0.110). Although still not significant, the trend was similar to significant results of the full model.

Figure 1displays the relative odds of 30-day mortality associated with 5-yr cohort groups and certification status, adjusting using the full patient and hospital model as described in table 7. We have set the midcareer board-certified group from 11 to 25 yr after graduation as the baseline comparison group and plotted the odds associated with various certified and noncertified groups in 5-yr intervals from graduation. As can be seen, early and late cohorts were similar to the midcareer board-certified group (OR near 1.0). However, the 11–15-, 16–20-, and 21–25-yr noncertified cohorts all had higher odds of mortality as compared with the baseline certified group. On the same graph, we also report the rate of uncertified anesthesiologists in these cohorts. Generally, as noncertification rates increased, the difference between certified and noncertified outcomes decreased.

The specialty of anesthesiology includes a substantial percentage of international medical graduates (IMGs). 17,18We therefore asked whether adjusting for IMG status would influence the results of our study. We found that adjusting for IMG status did not alter our findings concerning the influence of board certification on outcome in the group I midcareer anesthesiologists (OR^DEATH= 1.15, P < 0.02; OR^FTR= 1.15, P < 0.02). When fully adjusting for patient and hospital characteristics, IMG status was not associated with worse outcomes (OR^DEATH= 0.95, P < 0.09; OR^FTR= 0.95, P < 0.10). However, midcareer anesthesiologists who lacked board certification and who graduated from medical school outside the United States did have higher odds of death and FTR compared with those midcareer anesthesiologists who lacked certification and who graduated within the United States (OR^DEATH= 1.37, P < 0.007; OR^FTR= 1.38, P < 0.008). In contrast, midcareer anesthesiologists who had board certification and who graduated medical school outside the United States were associated with lower odds of death and FTR compared with those midcareer anesthesiologists with board certification who graduated within the United States (OR^DEATH= 0.88, P < 0.03; OR^FTR= 0.87, P < 0.01).

Evidence that certification by peers is associated with improved outcomes is rare. Such findings are important to document because there are many forces in the American healthcare system that push for increasing the number of board-certified practitioners, while at the same time urging specialists to manage a greater number of nonphysician or nonspecialist healthcare providers. 19For example, the new HEDIS-III^©measures of the National Committee for Quality Assurance 20and the new standards of the Joint Commission on Accreditation of Healthcare Organizations 21both provide strong incentive for documenting the board certification status of a Health Maintenance Organization's or hospital's physician population. At the same time, changing government regulations and a changing healthcare economy have led to an increase in the use of physician substitutes. 19,22–24 

The value of advanced anesthesiology training and board certification has been difficult to document, but evidence is accumulating. Slogoff et al.  25showed that the assessment of clinical skills by anesthesia residency program directors could accurately predict subsequent board certification rates of their trainees. The study provided some construct validity for the argument that clinical competency relates to board examination outcomes and therefore supported the value of the certification process. In study of Medicare patients undergoing cholecystectomy or transurethral prostatectomy, Silber et al.  1found that the higher the rate of board certification of a hospital's anesthesia staff, the lower were that hospital's mortality and FTR (death after complications) rates. In a subsequent study examining adult general surgical patients, Silber et al.  2observed a similar finding, with lower mortality and FTR associated with higher rates of anesthesiology board certification, adjusting for numerous patient and hospital characteristics. In a recent study examining the influence of anesthesiology direction on patient outcome in general and orthopedic surgery in the Medicare population, 4we also reported that the presence of an anesthesiologist in the operative suite lowered the 30-day mortality rate by 2.5 deaths per 1,000 cases, as compared with cases that did not involve direction by an anesthesiologist, suggesting that advanced training may be associated with better outcomes.

In the current study, the size of the mortality effect from lack of certification in the midcareer group was large, possibly up to 3.8 excess deaths per 1,000 cases. This figure is not inconsistent with the often-quoted “risks of anesthesia”—that being only 1 death in 200,000–300,000 cases. 26,27The latter estimates reflect the rate of adverse events occurring in the operating room in healthy individuals of all ages that lead to death or near death on the operating table, and only count subsequent deaths if directly attributable to a clear-cut anesthesia cause such as inadequate ventilation caused by gross provider error or technical failure. Such rates are used to study systems problems associated with specific medical errors. In contrast, the current report reflects a difference in 30-day mortality between the midcareer noncertified anesthesiologist and all other anesthesiologists, controlling for important patient and hospital factors in the Medicare population.

Our study suggests that board certification alone should not be viewed as a prognostic variable when used without knowledge of the graduation cohort of the anesthesiologist. Clearly, in the first 10 yr from medical school graduation, lack of board certification does not necessarily predict poor performance. Other investigators have reported that lack of certification is common within the first decade after completing medical school, 13,28and our study confirmed this. Furthermore, the rate of non–board-certified physicians is higher in the cohorts practicing more than 25 yr from medical school graduation. 14In the field of anesthesiology, the need for board certification increased greatly in the 1970s, just as board certification became more important in many other specialties. 14It may be that a cohort of noncertified anesthesiologists still practicing 25 yr after medical school graduation may select for the best of the noncertified anesthesiologists, or it may be that 25 yr of experience makes up for any shortcomings associated with failure to become certified earlier in one's career.

The importance of board certification has also been examined in other specialties with varying results. Norcini et al.  24studied myocardial infarction and found lower mortality rates associated with board-certified cardiologists rather than internists or family practitioners. Ramsey et al.  29reported that certified internists were found to provide more preventive care and to score higher in clinical competence measures than their noncertified counterparts. Kelly and Hellinger 30found lower mortality in patients with peptic ulcers undergoing stomach procedures when the surgeon was board certified, yet they found no significant certification effect for various other surgical procedures. Rutledge et al.  31showed a similar advantage in patients with ruptured abdominal aortic aneurysms. Pearce et al.  32reported that certification was significantly related to better outcomes for patients who underwent carotid endarterectomy or unruptured abdominal aortic aneurysms, but not for patients who underwent lower-extremity bypass grafting, whereas Brook et al.  33also did not find such an advantage when studying carotid endarterectomy.

The current study also provides some important findings concerning IMGs. IMG anesthesiologists did not perform differently than US medical school graduates overall. However, on further analysis we found two intriguing results. Board-certified IMG midcareer anesthesiologists had significantly lower death and failure rates than those board-certified midcareer anesthesiologists who went to medical school in the United States. In contrast, midcareer IMG anesthesiologists who were not board certified had a significantly higher risk of mortality and FTR than midcareer noncertified US graduates. The finding that certified IMG anesthesiologists performed better than their US colleagues suggests that the certification hurdle may be higher for IMG than for US graduates. If this were true, we would also have expected better performance by noncertified midcareer IMG anesthesiologists compared with their US colleagues, yet this was not observed. It would appear that the IMG anesthesiologists comprise a fairly heterogeneous group of practitioners, or it may imply that IMG practitioners without certification practice at hospitals with the highest mortality rates.

The major limitation of our findings stems from the significant clustering that occurred between the midcareer noncertified anesthesiologists and specific hospitals. Such clustering was not a problem in our previous analysis 4of anesthesiologist direction and patient outcomes. In that study, adjustment for the individual hospital provided almost exactly the same results as the fully adjusted model with hospital characteristics. However, in the current study, because of clustering, our estimates were sensitive to some hospital analyses. Hospitals where midcareer noncertified anesthesiologists practiced were clearly different from other hospitals. It may be that the environment in which noncertified anesthesiologists practice is less forgiving of mistakes, and this is reflected in the higher FTR rates. Possibly, both certified and noncertified anesthesiologists at these hospitals provided substandard care, thus obscuring the difference between these groups, as observed in the fixed-effects model. It may also be that hospital factors dominated patient outcomes, and what we observed to be a certification effect was the result of hospital effects. Given the differences between adjustment model results, we believe our evidence suggests that there was a real prognostic benefit from choosing a certified anesthesiologist, although the extent of this advantage may be advanced by the environment in which such a certified anesthesiologist practices.

In summary, the current study provides strong evidence that anesthesiologist board-certification status is an important factor associated with surgical outcomes, but it must not be used in isolation. Lack of board certification may be a result of reasons other than inability to pass the board examinations, and in such instances lack of certification does not imply potential shortcomings on the part of the anesthesiologist. Furthermore, noncertified anesthesiologists generally practice in very different hospital settings than do certified practitioners, and hospital factors appear to play an important role in determining outcome status. Although understanding the cause of these differential outcomes will require further analysis using large-scale chart review, we can conclude from this claims analysis that midcareer anesthesiologists who lack board certification, and the hospitals in which they are employed, appear to be associated with worse outcomes for surgical patients.

The authors thank William J. Greeley, M.D., M.B.A. (The Center for Outcomes Research, Department of Anesthesiology and Critical Care Medicine, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania), and Paul R. Rosenbaum, Ph.D. (Department of Statistics, The Wharton School, The University of Pennsylvania, Philadelphia, Pennsylvania), for helpful comments and suggestions; and Traci Frank (Administrative Coordinator, The Center for Outcomes Research, Department of Anesthesiology and Critical Care Medicine, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania), for assistance in preparing the manuscript.