Monitoring Obstetric Anesthesia Safety across Hospitals through Multilevel Modeling

ON the top of the list of the 27 patient safety indicators (PSIs) issued by the Agency for Healthcare Research and Quality (AHRQ) in 2002 was the rate of complications of anesthesia or PSI-01.^*1  Based on routinely collected administrative data, PSI-01 was designed for reporting and monitoring anesthesia safety across hospitals and for identifying safety concerns and targeting areas for safety improvement. Despite a long-standing culture of safety and safety indicators in anesthesia, application of PSI-01 in anesthesia research and practice has remained scarce.^2–5  PSI-01 has lagged behind other PSIs, with some of them being publicly reported in the annual National Healthcare Quality and National Healthcare Disparities Reports and routinely calculated with hospitals information technology systems.^†

The conventional approach for calculating the rate of adverse events and making it comparable across hospitals is risk adjustment.⁶  Risk adjustment takes into consideration differences in characteristics of patients (case-mix) and types of procedures (procedure-mix). It is based on logistic regression models that express the relationship between the binary outcome (i.e., the patient did or did not have an adverse event) and a set of predictors describing the case- and procedure-mixes. Recent research indicates that, to produce a more precise estimate that takes into consideration correlations of patients within hospitals (clustering), the rate of adverse events should be further adjusted for the hospital identifier (i.e., patients’ hospital affiliation) through multilevel modeling.^7–9  In surgery, adjustment based on multilevel models has been demonstrated to increase the precision of the estimated rate of adverse outcomes and to significantly change ranking of hospitals that may also change the priority targets for safety measures.^7,9  Furthermore, multilevel models quantify the level of confidence one can have in the estimated rate of adverse events for each hospital with rankability.^10,11  Rankability identifies hospitals that should not be included in league tables or be identified as nonrankable in league tables. Finally, multilevel model–based adjustment may provide a better prediction for future patients than risk adjustment.^12–14  Multilevel-based adjustment is now adopted by the American College of Surgeons in monitoring adverse outcomes after surgery across hospitals but has received little attention in anesthesia.^15,16 

Each year, over 50 million surgical procedures are performed in the United States; of them, about 8% are related to labor and delivery.^‡ The median cost of anesthesia-related adverse events (ARAEs) in obstetrics is nearly twice compared with other anesthesia specialties.^17,18  Despite the decrease in anesthesia-related mortality and severe morbidity during the last two decades, the cost of obstetric anesthesia-related complications has not significantly decreased.^18–21  Currently, ARAEs occur in about one out of every 200 parturients.²  This figure may be increasing owing to the increased rate of cesarean section and parturients’ request for analgesics during labor.^22,23  This study aims therefore to develop and assess a multilevel model–based rate of ARAEs in labor and delivery using administrative data for monitoring obstetric anesthesia safety across hospitals.

The study protocol was reviewed by the Institutional Review Board of Columbia University Medical Center and was granted exemption under the Code of Federal Regulations, Title 45, Part 46 (not human subjects research). The study adheres to the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement.²⁴ 

The study sample consisted of all women admitted for labor and delivery in the State of New York between January 1, 2008, and December 31, 2011. Data for years 2008–2009 were used to develop the logistic and multilevel models and data for years 2010–2011 to assess predictive ability of both models. Hospital discharge record data for these women collected in the de-identified New York State Inpatient Database were analyzed. State Inpatient Databases (SIDs) are part of the Healthcare Cost and Utilization Project sponsored by the AHRQ. SIDs capture all inpatient discharges from nonfederal acute care community hospitals in participating states since 1988. Nonfederal community hospitals account for 85% of U.S. hospitals. For each discharge, the SID includes patients’ demographic, economic, and outcome characteristics, one hospital identifier, and up to 15 procedural and 25 diagnostic codes defined in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). Discharges with neonatal or maternal diagnoses and procedures identified with the neonatal-maternal code provided by the SID and female sex were first selected. Then, discharges indicating labor and delivery were identified with a combination of ICD-9-CM diagnosis and procedure codes developed by Kuklina et al.²⁵  (appendix 1). However, Diagnosis-Related Group codes were not used in this study since they changed during the 4-yr study period. In addition, discharges were excluded if the hospital identifier was missing or if delivery took place in a hospital with less than 2 deliveries/year.

ARAEs were identified with a combination of ICD-9-CM diagnosis and procedure codes developed by Cheesman and colleagues^2,3  (appendix 2). We also analyze the subgroup of ARAEs related to neuraxial anesthesia and local anesthetics (appendix 2). This subgroup of complications was selected owing to both its high incidence and preventability.^2,26 

The following demographic and delivery characteristics were recorded directly from the SID: age, admission for delivery during weekend, and admission type for delivery (elective or nonelective). Other patient- and procedure-related risk factors for ARAEs were identified with ICD-9-CM diagnosis and procedure codes (appendix 3).

Consequences of multilevel model–based adjustment were assessed with (1) the interhospital variability of the estimated rate of ARAEs; (2) the reclassification of hospitals based on their outlier status; and (3) the ability of the multilevel model developed with the 2008–2009 data to predict future hospital performance in 2010–2011. In addition, the confidence in the point estimate for each hospital or hospital rankability was estimated with the multilevel model.

Since the rate of ARAEs may have been influenced by coding practice at each hospital, the relationship between the reporting index of ICD-9-CM codes for each hospital and the multilevel model–based rate of ARAEs was assessed. For each hospital, the reporting index was defined as the ratio of the sum of ICD-9-CM diagnosis (including E-codes) and procedure codes recorded for each discharge to the number of discharges.²⁷ 

Results are expressed as mean ± 1 SD or number (%). When indicated, 95% CI was calculated.

The statistical analysis was performed with R version 3.0.2 (R Foundation for Statistical Computing, Austria) and specific packages for the multilevel model (lme4 and arm).

A three-stage approach was used to develop the logistic and multilevel models using data for years 2008 and 2009. The first stage was a logistic regression model specifying each component of the case- and procedure-mixes. The second stage was a multilevel model including the hospital level only (“empty model”). The third stage was a multilevel model specifying each component of the case- and procedure-mixes as the first-level variables and the hospital identifier as the second-level variable. At each of the three stages, goodness of fit was assessed with the Akaike information criterion, discrimination with the c-index, and calibration with the Hosmer–Lemeshow test. A lower Akaike information criterion indicates a better fit with a difference greater than 6 indicating a strong difference.²⁸  For the logistic model, univariate comparisons between discharges with and without ARAEs were made using unpaired Wilcoxon test for quantitative variables and chi-square test or Fisher exact test for qualitative variables. Unadjusted odds ratios were calculated with univariate logistic regression. Variables with a P value less than 0.2 in the univariate analysis were entered in the logistic model with a backward selection using the entire dataset for 2008–2009. For the multilevel model, hospital affiliation was treated as a random-effect predictor. It corresponded to the hospital identifier with the assumption of a normally distributed hospital intercept and a constant slope.

The risk-adjusted and multilevel model–based rates of ARAEs for each hospital were calculated as the ratio of the observed to the expected (O/E) rate multiplied by the observed rate in the study sample. The expected rate was the mean of the individual probabilities of experiencing an ARAE in that hospital. For the risk-adjusted rate, probabilities were calculated with a logistic regression model including the case-mix and the procedure-mix as fixed-effect predictors. For the multilevel model–based rate, probabilities were calculated with a multilevel model including the case-mix and the procedure-mix as fixed-effect predictors (first level) and the hospital identifier as a random-effect predictor (second level).

The extent to which multilevel model–based adjustment reduced interhospital variability was assessed by the comparison of the SDs and skewnesses of the grand mean of the rates (i.e., the mean of the hospitals in the sample study).

Rankability of each hospital was calculated with the following formula: where σ² indicates the variance.^8,10,11  The between-hospital variance corresponds to the variance of the random effect in the multilevel model. It is sometimes described as the “signal” since it corresponds to the difference between hospitals beyond chance. The within-hospital variance corresponds to the variance of the random effect for each hospital. It is sometimes described as the “statistical noise” since it corresponds to the within-hospital uncertainty. Rankability ranges from 0 to 1. Rankability greater than 0.7 is considered as good and greater than 0.9 as excellent. As indicated by the formula, rankability depends not only on the difference between hospitals and the measurement or sampling error but also on the hospital volume.

Hospitals were divided into three groups based on their outlier status with risk and multilevel model–based adjustment methods and reclassification tables built. Hospital outliers were defined according to the American College of Surgeons’ National Surgical Quality Improvement Program criteria.^13,16 

For risk adjustment, outlier definition used the hospital O/E ratio. O/E ratio was calculated as the ratio of the observed to the expected rate in the hospital as described in the section “Interhospital Variability of the Estimated Rate of ARAEs,” but without including the constant term observed rate in the study sample. Definitions of outliers were as follows: high outlier or bad performer if O/E was greater than 1 with its 95% CI not including 1, low outlier or good performer if O/E was less than 1 with its 95% CI not including 1, and as expected or average performer if the 95% CI of O/E included 1. The lower and upper limits of the 95% CI of O/E was calculated as LL (or UL)/E, where LL (or UL) was the lower (or upper) limit of the CI of a Poisson distribution for the observed number of cases in the hospital and E the expected numbers of ARAEs in the hospital.

For multilevel model–based adjustment, outlier definition used the hospital odds ratio calculated directly from the multilevel model as the exponential of the random effect for each hospital estimated in the multilevel model. The definitions of high outlier, low outlier, or as expected were identical to the ones used for risk adjustment. The 95% CI of the hospital odds ratio was calculated as ±1.96 standard error, where the standard error was estimated in the multilevel model.

The prediction of future performance for hospitals present both in 2008–2009 and 2010–2011 was based on hospital outlier status based on risk adjustment and multilevel model–based adjustment in 2008–2009. It was assessed in two ways: (1) the adjusted odds ratio of ARAEs for the high- and average-outlier status relative to the low-outlier status and (2) the proportion of between-hospital variance in ARAE rates in 2010–2011 explained by hospital outlier status in 2008–2009.^29,30  To calculate the adjusted odds ratio, two logistic regression models were developed for patients admitted in 2010–2011 with the occurrence of ARAEs as the dependent variable and the previously identified patient- and procedure-level risk factors in 2010–2011 and the 2008–2009 outlier status as independent variables. The first model used the outlier status based on risk adjustment and the second model used the outlier status based on multilevel model–based adjustment. If the adjusted odds ratio was significantly greater than 1 for the high- and average-outlier status, then past hospital performance did predict future hospital performance. To calculate the proportion of between-hospital variance in ARAE rates in 2010–2011 explained by hospital outlier status in 2008–2009, three multilevel models were developed with the occurrence of ARAEs as the dependent variable. The first model used patient- and procedure-level risk factors previously identified as the first-level variables (fixed effect) and the hospital identifier as the second-level variable (random effect) for the years 2010–2011. In the two other models, the hospital outlier status in 2008–2009 based on either the logistic or the multilevel model was added to the set of the first-level variables. The proportion of variation in subsequent ARAE rates explained by hospital outlier status was calculated by the percent reduction in the between-hospital variance between the multilevel model without the hospital outlier status and the multilevel model with the hospital outlier status.

For each hospital, the association between the reporting index of ICD-9-CM codes and the multilevel model-based rate of ARAEs was assessed with the Pearson correlation coefficient. Comparison of the reporting index across hospitals was based on the Kruskal–Wallis test.

A sensitivity analysis was performed using the subset of hospitals with a rankability greater than 0.7 regarding performance of the multilevel model, between-hospital variability in ARAEs rate, and prediction of future hospital performance.

During the years 2008–2009, 466,442 discharges in 144 hospitals met the inclusion and exclusion criteria for labor and delivery and were included in the analysis (fig. 1). At least one ARAE was recorded in 2,156 discharges, yielding an observed rate in the study sample of 4.62 per 1,000 discharges (95% CI, 4.43 to 4.82). At least one ARAE related to neuraxial anesthesia and local anesthetics was recorded in 1,746 discharges (3.74/1,000; 95% CI, 3.57 to 3.92) (appendix 2).

Seven risk factors for ARAEs were identified in the logistic model using data for years 2008–2009: age, obesity, pulmonary hypertension, cardiac valvular disease, asthma, cesarean delivery, and postpartum hemorrhage (tables 1 and 2). The c-index of the model was 0.60 (0.58–0.61), and the Hosmer–Lemeshow test P value was 0.28 (appendix 4). The results from univariate analysis, multivariate logistic regression, and multilevel modeling for ARAEs related to neuraxial anesthesia and local anesthetics are presented respectively in appendices 5-7.

The grand mean of the risk-adjusted ARAE rate for the 144 hospitals was 5.29 per 1,000 deliveries, whereas the grand mean of the multilevel model–based rate was 4.38 per 1,000 deliveries. Compared with risk adjustment, multilevel model–based adjustment reduced both the SD from 4.68 to 1.35 and the skewness from 2.08 to −1.05 of the distribution of the estimated rates across the 144 hospitals. Multilevel-based adjustment tended to shrink estimated individual hospital ARAE rates toward the grand mean of all hospitals. The magnitude of shrinkage increased as the hospital volume of deliveries decreased (fig. 2).

The mean rankability of ARAE rates for the 144 hospitals was 0.81 ± 0.11. One hundred eighteen hospitals (81.9%) had a rankability greater than 0.7. The rankability increased with the hospital volume of deliveries (fig. 3). The mean volume of deliveries for the 26 hospitals with reliability less than or equal to 0.7 was 254.

Eleven of the 26 low-outlier hospitals (42.3%) identified with risk adjustment were reclassified as as-expected outlier with multilevel model–based adjustment (table 3). Six of the 93 as-expected outlier hospitals (6.4%) identified with risk adjustment were reclassified as high outlier with multilevel model–based adjustment. With multilevel model–based adjustment, the proportion of low outliers decreased from 18.0% to 10.4% and the proportion of high outliers increased from 17.4% to 21.6%.

One hundred thirty-nine hospitals were present during both the 2008–2009 and 2020–2011 periods in the SID. During the 2010–2011 period, 453,617 discharges were analyzed and at least one ARAE of any type was recorded in 2,133 discharges, yielding an observed rate of 4.70 per 1,000 discharges (95% CI, 4.50 to 4.90). At least one ARAE related to neuraxial anesthesia and local anesthetics was recorded in 1,755 discharges (3.87/1,000; 95% CI, 3.69 to 4.05).

The adjusted odds ratios of experiencing an ARAE for a patient admitted during 2010–2011 in a high and average outlier status relative to a low outlier based on the logistic model in 2008–2009 were 1.84 (95% CI, 1.63 to 2.08) and 3.34 (95% CI, 2.93 to 3.81), respectively (table 4). The outlier status explained 47.1% of the between-hospital variance in the subsequent ARAEs rates. The adjusted odds ratio and proportion of variance explained were similar for outlier status based on the multilevel model.

The mean reporting index of ICD-9-CM codes for hospitals was 6.7 ± 3.0 codes per discharge, with a significant difference across hospitals (P < 0.0001). No significant association was observed between the reporting index and the multilevel model–based rate of any type of ARAEs (r = 0.11; 95% CI, −0.06 to 0.27) and between the reporting index and the multilevel model–based rate of neuraxial and local anesthetics of ARAEs (r = 0.11; 95% CI, −0.05 to 0.27).

Diagnostic statistics of the logistic regression model and the multilevel model changed little after excluding the 26 hospitals with rankability less than or equal to 0.7. As expected, excluding the 26 hospitals with rankability less than or equal to 0.7 decreased the interhospital variability and increased the proportion of between-hospital variance explained by hospital outlier status in both the logistic regression model and the multilevel model.

Results of this study indicate that compared with risk adjustment, multilevel model–based adjustment considerably reduces the between-hospital variability in ARAE rates, leads to reclassification of hospitals, and identifies nonrankable hospitals. The predictive validity of the two adjustment methods, however, is similar.

The most striking result of the multilevel model–based adjustment was the reduction in the between-hospital variability in ARAE rates. This phenomenon is also known as shrinkage toward the grand mean. When the number of deliveries is low in a given hospital, the estimated rate of adverse events can be very unreliable. Multilevel models combine the limited information from the particular hospital with the information from all hospitals in the study sample to produce a more robust estimate of the rate in this particular hospital. Multilevel-based adjustment with the multilevel model tends therefore to shrink estimated individual hospital ARAE rates toward the grand mean, which is the mean of ARAE rates across all hospitals in the study sample.¹⁶  The magnitude of shrinkage increases as the hospital volume of deliveries decreases. One limitation of shrinkage, which is also the source of controversies about the multilevel model–based adjustment method, is that low-volume hospitals can be credited with average performance. However, the level of confidence in the shrunk estimate for each hospital and the ability to compare this hospital with other hospitals can be assessed with the rankability of each hospital.^31,32  A rankability greater than 0.7 is considered as good and is suggested as the threshold to include one hospital in league table.^8,11  In the current study, 26 hospitals (18.1%) had rankability less than 0.7 and corresponded to low-volume hospitals. These hospitals should therefore not be compared with other hospitals or should be identified as nonrankable in league tables. However, all the hospitals in the study sample had a reliability greater than 0.5, which is sometimes used to define a “fair” rankability. This is very different from surgery where rankability is lower (i.e., less than 0.5) owing to a high number of very low case volume hospitals, raising concern about the validity of reporting and comparing surgical outcomes across hospitals.^7,8,10,31,32  In addition to the reduction in the between-hospital variability, multilevel based–adjustment resulted in a significant reclassification in hospital outliers ranking. More specifically, it increased the proportion of bad performers, which may allow a more efficient targeting of hospitals that may benefit the most from further investigation.

Compared with logistic regression models, multilevel models have been suggested to improve the prediction for individual hospitals of the rate of adverse outcomes in a subsequent time period based on models developed using data from an earlier time period.¹²  Using different metrics to assess future prediction (median absolute difference, root median square error, and percentage of hospitals whose predicted ARAE rates are within 95% CIs of the observed rates), the improvement in prediction was reported in mortality after uncommon surgical procedures.^13,14  The improvement was less for mortality after more common surgical procedures or for mortality in trauma patients. With metrics to assess performance identical to the ones used in the current study (adjusted odds ratio, proportion of between-hospital variance explained by hospital outlier status), no significant improvement in prediction of mortality among trauma patients was observed with multilevel models compared with logistic model.³⁰  In the current study, the risk of experiencing an ARAE for a patient admitted during 2010–2011 in a high and average outlier status was 1.8 and 3.3 times the risk in a low outlier status. The estimated odds ratios associated with outlier status were similar between logistic regression and multilevel-based models. Our results are consistent with previous reports and suggest that multilevel modeling does not seem to improve future prediction of ARAE rates within hospitals compared with logistic modeling.^13,14,30  However, we do not think the performance of predictive validity within the same individual hospitals over time is a diagnostic statistic directly relevant to the purpose of our study. In essence, hospital ranking on anesthesia safety is the comparison of performance across hospitals at a given time point (i.e., a cross-sectional comparison) rather than the forecast of future performance within the same hospitals. In addition, changes can be observed over time such as the number of hospitals included or individual hospital performance, making the multilevel developed on a previous time period no longer valid for a next time period. In other words, comparison of hospitals should probably be based on a regularly updated multilevel model that takes into consideration these possible changes.

The definitions and coding practice of adverse events at different hospitals may raise concerns about the use of indicators based on administrative data for routine surveillance. First, as previously reported in the literature, the definition of adverse events is complex and somewhat subjective compared with other clear-cut outcomes such as death.³³  However, the marked decrease in anesthesia-related mortality and severe morbidity in obstetric anesthesia over the last 20 yr precludes the use of most severe outcomes to assess anesthesia safety at the hospital level.^20,21  Second, adverse events may be recorded inconsistently across hospitals. The lack of association between the reporting index and the rate of ARAEs suggests that the pattern of coding may have little influence on the validity of our study results. These concerns should not be viewed as a limitation of this type of indicators based on administrative data but rather considered as an incentive to improve medical record documentation and accuracy of coding. The alternative options to administrative data are prospective registries such as the National Anesthesia Outcomes Registry or the Society for Obstetric Anesthesia and Perinatalogy Serious Complication Repository.^26,34,35 . They may ensure a more homogeneous definition of ARAEs and a more consistent recording across hospitals. However, relying on these data systems usually poses a significant delay between data collection, analysis, risk identification, and development of interventions, making it difficult to implement timely safety improvement measures. Moreover, the rarity of adverse events in obstetric anesthesia with an incidence rate of 5/1,000 may preclude their comprehensive capture and sufficient statistical power if the volume of the gathered data is not large enough, as recently illustrated with the Society for Obstetric Anesthesia and Perinatology's Serious Complication Repository project.²⁶  Finally, creating and maintaining quality registries requires a significant amount of financial resources, which may threaten the long-term viability of these specialty data systems.³⁶  Assessing and monitoring anesthesia safety must consider the tradeoff between the perceived higher credibility of prospectively gathered clinical data and the low-cost and readily available administrative data. In that sense, the administrative-data approach and the prospective registry–data approach should be viewed as complementary means to the same end.

This study has several limitations. First, it was conducted in New York and included only 144 hospitals. The number of community hospitals in the United States is about 5,000, and the analysis performed on this limited sample may not be generalizable to all the hospitals in the United States. Second, obstetric patients are usually healthy with little comorbidity. The results may therefore not apply to different patient populations and anesthetic specialties, such as cardiac or vascular anesthesia where the weight of the case- and procedure-mixes is probably higher. Third, the definition of ARAEs was based on a combination of ICD-9-CM codes, and the ICD-10-CM is expected to be introduced soon. However, Li et al.³  demonstrated that ARAEs can also be identified with ICD-10-CM codes. Fourth, the AHRQ definitions of PSIs and the definition of ARAEs used in this study are very heterogeneous and nonspecific as they include complications of varying severity. Use of a more specific indicator, such as the one for ARAEs related to neuraxial anesthesia and local anesthetics, may allow identification of frequent and preventable adverse events that may benefit the most from safety measures.²⁶ 

In conclusion, the multilevel modeling approach allows us to assess the rankability of the study hospitals while providing similarly accurate estimate of the risk of obstetric ARAEs as the conventional risk-adjustment method. Therefore, the multilevel modeling approach could serve as a practical alternative to the risk-adjustment method in monitoring obstetric anesthesia safety across hospitals.

The authors thank Joanne Brady, Ph.D., and Barbara H. Lang, M.P.H., from the Department of Anesthesiology, Columbia University College of Physicians and Surgeons, New York, New York, for their invaluable data and administrative assistance.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.