The Cardiac Anesthesia Risk Evaluation (CARE) score, a simple Canadian classification for predicting outcome after cardiac surgery, was evaluated in 556 consecutive patients in Paris, France. The authors compared its performance to those of two multifactorial risk indexes (European System for Cardiac Operative Risk Evaluation [EuroSCORE] and Tu score) and tested its variability between groups of physicians (anesthesiologists, surgeons, and cardiologists).
Each patient was simultaneously assessed using the three scores by an attending anesthesiologist in the immediate preoperative period. In a blinded study, the CARE score category was also determined by a cardiologist the day before surgery, by a surgeon in the operating room, and by a second anesthesiologist at arrival in intensive care unit. Calibration and discrimination for predicting outcomes were assessed by goodness-of-fit test and area under the receiver operating characteristic curve, respectively. The level of agreement of the CARE scoring between the three physicians was then assessed.
The calibration analysis revealed no significant difference between expected and observed outcomes for the three classifications. The areas under the receiver operating characteristic curves for mortality were 0.77 with the CARE score, 0.78 with the EuroSCORE, and 0.73 with the Tu score (not significant). The agreement rate of the CARE scoring between two anesthesiologists, between anesthesiologists and surgeons, and between anesthesiologists and cardiologists were 90%, 83%, and 77%, respectively.
Despite its simplicity, the CARE score predicts mortality and major morbidity as well the EuroSCORE. In addition, it remains devoid of significant variability when used by groups of physicians of different specialties.
NUMEROUS multifactorial risk indexes have been constructed to predict outcome after cardiac surgery. 1–10Despite their usefulness, their relative complexity may explain why few physicians use these scores in clinical practice. Recently, Dupuis et al. 11have described the Cardiac Anesthesia Risk Evaluation (CARE) score, a simple intuitive risk ranking system which is similar to the American Society of Anesthesiologists (ASA) physical status classification. These authors 11reported reliable performances of this score in predicting mortality and major morbidity with a discrimination and a calibration comparable with those obtained by three popular multifactorial risk indexes. 1,2,5However, because these interesting findings result only from a single institution, more evaluations are required to confirm the apparently good predictive accuracy of this new classification. In addition, the predictive performance of this simple stratification was not compared with the most recent multi-factorial risk index, the European System for Cardiac Operative Risk Evaluation (EuroSCORE). 9Therefore, the main objectives of the current study were (1) to determine the predictive performance of the CARE score for predicting mortality and major postoperative morbidity in cardiac surgical adult patients from a center other than where it was initially developed; (2) to compare its predictive performance with those of two multifactorial risk indexes (the EuroSCORE 9and the Tu score 5); and (3) to evaluate variability of CARE scoring among cardiac anesthesiologists, surgeons, and cardiologists.
Materials and Methods
This prospective study was conducted at the Institute of Cardiology in the Pitié-Salpêtrière Hospital, Paris, France, from January 2002 to May 2002, and was approved by our local institutional medical ethics committee. Because data were collected while care of patients conformed to standard procedures currently used in our institute, authorization was granted to waive informed consent for the study. During a 16-week period, all adult patients undergoing cardiac surgical procedures with or without cardiopulmonary bypass at our institute were enrolled. We excluded patients scheduled to undergo cardiac transplantation, implantation of ventricular assistance devices, and peripheral vascular surgery. Patients with congenital heart disease were not included in this study.
Scoring of Patients
In the immediate preoperative period, the CARE score risk category of each patient was determined by an attending anesthesiologist (the definitions of each category are summarized in table 1). Simultaneously, the Tu score and the EuroSCORE were calculated by the same attending anesthesiologist (the components of these two multifactorial risk indexes are summarized in table 2). Because the CARE score is similar to the ASA physical status scale, a model familiar to surgeons, a CARE score category was also determined by the surgeon in the operating room. All CARE score stratifications were only performed by senior physicians. The day before the surgery, an attending cardiologist also determined the CARE score category. The intraoperative care was left to the discretion of both the surgeon and the anesthesiologist. At the arrival in the cardiac intensive care unit, a new CARE score category was given by a second anesthesiologist in charge of the patient. After each assessment of the CARE score by a physician, the value was enclosed in an envelope. Consequently, each physician was unaware of the result previously assessed. The results of the scoring by the different physicians were only analyzed when the patient left the hospital or died. We evaluated the variability of the CARE score between two anesthesiologists, between anesthesiologists and surgeons in the immediate preoperative period, and between anesthesiologists and cardiologists in the preoperative period. To eliminate small differences of the CARE score assessment with the fewest clinical implications, we evaluated the variability of the score between different specialist groups with the score divided into only four categories: a low-risk group (CARE 1), an intermediate-risk group (CARE 2 and 3), a high-risk group (CARE 4), and a very-high-risk group (CARE 5).
The two main outcomes of our study were the inhospital mortality and major postoperative morbidity. In-hospital mortality was defined as a death occurring during the hospital stay. Major morbidity was defined using criteria previously reported by Dupuis et al. 11(table 3).
The performance of the CARE score for predicting in-hospital mortality and major morbidity in our population was assessed by determining the calibration and discrimination of the score. The calibration was assessed using the Pearson chi-square goodness-of-fit test, 12which compared the predicted outcomes (in-hospital mortality and severe morbidity) with those observed in the population studied. Expected outcome values for three classifications were calculated from the reference group of the studies of Nashef et al. 9and Dupuis et al. 11Because the EuroSCORE was defined only as a multifactorial risk index for predicting mortality, the calibration of this score was only assessed using this outcome. The discrimination of the three classifications was assessed by building the receiver operating characteristic (ROC) curve. 13The ROC curves were obtained using the whole range of scores of each classification. The areas under the ROC curves and their 95% confidence intervals (CIs) were calculated and compared by using a nonparametric test from a method previously described by Hanley et al. 14The ROC curve obtained with the CARE score rating by the attending anesthesiologists was used as the reference for comparisons with all the other curves.
The interrater variability of the CARE scoring was assessed by calculating the agreement rate between groups of physicians. In addition, the κ measure of agreement between the scoring by the attending anesthesiologist and the other groups of physicians (e.g. , surgeons and cardiologists) was calculated. Data are expressed as mean ± SD or percentages and their 95% CIs. All P values are two-tailed, and a P value of less than 0.05 was considered significant.
During a 16-week period, 556 consecutive patients were included prospectively in our study. The characteristics of these patients are presented in table 4. The overall in-hospital mortality and major morbidity rates were 5.8% (95% CI, 3.9–7.7%) and 24.5% (95% CI, 20.9–28.1%), respectively.
Predictive Performances of the CARE Score
The calibration analysis for mortality and severe morbidity in our population is summarized in table 5. No significant difference was found between expected and observed outcomes. The areas under the ROC curves with the CARE score for predicting mortality (fig. 1) and major morbidity (fig. 2) were 0.77 (95% CI, 0.69–0.85) and 0.71 (95% CI, 0.65–0.77).
Comparison of Predictive Performances of the CARE Score with Two Multifactorial Risk Indexes (EuroSCORE and Tu Score)
In the population studied, the calibration analysis showed no significant difference between the expected and observed outcomes for both multifactorial risk indexes (tables 6 and 7). The area under the ROC curve for mortality (fig. 1) was 0.78 (95% CI, 0.71–0.85) for the EuroSCORE, 0.77 (95% CI, 0.69–0.85) for the CARE score, and 0.73 (95% CI, 0.63–0.83) for the Tu score (not significant [NS]). Similarly, the areas under the ROC curves for morbidity were also comparable among the three classifications (fig. 2). For all classifications, the discrimination was significantly lower for major morbidity than for mortality.
Variability of the CARE Score among Anesthesiologists, Surgeons, and Cardiologists
All patients (n = 556) were scored at arrival in the cardiac intensive care unit by a second anesthesiologist who was unaware of the CARE score category given in the preoperative period. The level of agreement of the CARE score between two different anesthesiologists was 90% (95% CI, 87.5–92.5%), and the κ value was 0.739 (SE = 0.033; P < 0.0001). The goodness-of-fit test when the patient was scored by a second anesthesiologist showed a significant difference between expected and observed rates of mortality (chi-square = 20.4; P = 0.009) and but not for morbidity (chi-square = 13.3; P = 0.102). The CARE score, determined by the second anesthesiologist at the arrival of patient in the intensive care unit, had an area under the ROC curve of 0.78 (95% CI, 0.70–0.86) for mortality and 0.72 (95% CI, 0.66–0.78) for morbidity. These values were not significantly different from those obtained by the anesthesiologist in the preoperative period.
The preoperative determination of the CARE score category by surgeons was available in 463 patients. The level of agreement of the CARE score between anesthesiologists and surgeons in preoperative period was 83% (95% CI, 79.7–86.4%), and the κ value was 0.613 (SE = 0.038; P < 0.0001). The goodness-of-fit test when the patient was stratified by surgeons showed no difference between expected and observed rates of mortality (chi-square = 10.7; P = 0.219) and morbidity (chi-square = 3.2; P = 0.921). The CARE score determined by surgeons had areas under the ROC curves of 0.86 (95% CI, 0.78–0.94) for predicting mortality and 0.75 (95% CI, 0.69–0.81) for predicting major morbidity. A total of 463 patients were scored both by the surgeon and the attending anesthesiologist. In these patients, the areas under the ROC curves were not significantly different for predicting mortality (0.86 [95% CI, 0.78–0.94]vs. 0.84 [95% CI, 0.76–0.92]; NS) or predicting major morbidity (0.75 [95% CI, 0.69–0.81]vs. 0.73 [95% CI, 0.66–0.79]; NS).
A total of 367 patients were also scored by a cardiologist using the CARE score. The overall rate of agreement was 77% (95% CI, 72.7–81.3%), and the κ value was 0.453 (SE = 0.049; P < 0.0001). The goodness-of-fit test when the patient was scored by a cardiologist showed a significant difference between expected and observed rate of mortality (chi-square = 23.3; P = 0.003) but not for morbidity (chi-square = 13.7; P = 0.09). The CARE score determined by the cardiologist had areas under the ROC curves of 0.77 (95% CI, 0.63–0.91) for predicting mortality and 0.70 (95% CI, 0.64–0.76) for predicting major morbidity. A total of 367 patients were scored both by the cardiologist and the attending anesthesiologist. In these patients, the areas under the ROC curves were not significantly different for predicting mortality (0.77 [95% CI, 0.63–0.91]vs. 0.79 [95% CI, 0.69–0.89]; NS) or predicting major morbidity (0.70 [95% CI, 0.64–0.76]vs. 0.69 [95% CI, 0.63–0.75]; NS).
The main findings of the current study are that (1) the CARE score, a simple scoring system recently initially described in a group of Canadian patients, is equally applicable in a cardiac surgical population from another center; (2) its performance is similar to those obtained with two popular multifactorial risk indexes, including the EuroSCORE; and (3) the stratification by the CARE score is devoid of significant variability when used by different group of physicians, including surgeons and cardiologists.
Up to the current time, the risk-adjusted prediction of outcome after cardiac surgery has been preferentially assessed by multifactorial risk indexes. 1–10,15These scores are obtained from multiple regression logistic analysis, which allows for identification of risk factors of morbidity, mortality, or both. Because these scores are not easily calculated at the bedside, few clinicians use them routinely. Recently, Canadian authors have reported reliable performance of the CARE score for predicting duration of stay in the hospital, morbidity, and mortality after cardiac surgery. 11In contrast to multifactorial risk indexes, the CARE score is an intuitive risk ranking similar to the ASA physical status classification. The score system was constructed on both clinical judgment and three classic risk factors, comorbidity, surgical complexity, and operative priority. Although simple, the CARE score has a predictive performance similar to three popular multifactorial risk indexes. Because the CARE score has only been evaluated in a single institute, further investigations were required to confirm the reliability of this new simple classification at other centers. Therefore, we assessed both the calibration and the discrimination of the CARE score at our institute. We confirmed that CARE score remains a reliable outcome predictor in a cardiac surgical population from a center other than where it was initially developed. We found no significant difference between expected and observed outcomes when the CARE score was evaluated preoperatively by an anesthesiologist. In our population, the CARE score provided an acceptable discrimination for predicting mortality and major postoperative morbidity. It is worth noting, as recently reported by Heijmans et al. , 16that an area under the ROC curve between 0.7 and 0.9 means that the classification has acceptable discrimination. In addition, the performance of the scores obtained in our population is consistent with those reported by Canadian authors. As previously reported by Dupuis et al. , 11we found that the CARE score was better for predicting mortality than for predicting morbidity. To our knowledge, the current study is the first study to validate the predictive performance of the CARE score in a cardiac surgical population with different preoperative characteristics.
In the population studied, the overall mortality rate was higher than that observed by Dupuis et al. 11Two principal reasons could explain this divergence. First, our study included a larger proportion of patients with advanced age: 26% of our patients were aged 75 yr or older, whereas Dupuis et al. 11included only 16.5% of this age group. Several studies 17–19reported that elderly patients undergoing cardiac surgery have a higher inhospital mortality rate. However, the overall mortality rate found in the current study is consistent with those reported in previous European studies. 9,20Second, 22% of the patients studied (only 14.5% reported by Dupuis et al. 11) underwent complex or combined surgical procedures, which are universally known to be a important risk factor of mortality in cardiac surgery. 1,5,9On other hand, the overall morbidity rate obtained in the current study was comparable with that observed by Dupuis et al. 11
The EuroSCORE has been reported by numerous authors 21–24as a robust multifactorial index. Initially described in European population, 9this score has been more recently validated in North American patients. 23To date, no previous study has compared the predictive performances of this score with those of the CARE score. In our population, both scores had an acceptable calibration. In addition, the area under the ROC curve for predicting mortality and morbidity were comparable between these two scores. It should be noted that the performances of these two scores were always better than that obtained from the Tu score. These findings are consistent with those previously reported by several authors. 11,21
Despite its relative subjectivity, a good concordance was found between CARE scoring by different groups of physicians. The level of agreement of the CARE score between anesthesiologists was 90%, with a κ value of 0.739. These results are consistent with those previously reported by Dupuis et al. 11We also evaluated the concordance of the CARE score between anesthesiologists and surgeons. The agreement rate of the CARE scoring was 83%, with a κ value of 0.613. The lowest agreement rate (77%) was obtained between anesthesiologists and cardiologists. However, the κ value of 0.453 (SE = 0.049; P < 0.0001) suggested that this variability is not significant.
The following points must to be considered in the assessment of the clinical relevance of our study. First, we included only 556 patients, and consequently, few patients with high risk were enrolled. However, the aim of our study was only to evaluate the predictive performances of a score that has been previously validated in a Canadian population. Second, we evaluated the performances of the EuroSCORE for predicting in-hospital mortality. Nevertheless, the EuroSCORE was initially described for predicting 30-day mortality. Third, we evaluated the variability between physicians by arbitrarily choosing the immediate preoperative scoring by the anesthesiologist as the reference point. Fourth, we did not evaluate the performances of the CARE score for predicting the duration of stay in hospital as previously reported by Dupuis et al. 11However, these authors have already emphasized the modest performance of this score for predicting this outcome. Fifth, we did not assess the variability associated with the use of the multifactorial risk index. However, it has been previously shown that the use of risk scoring systems may be associated with errors in data entering and computing. 25
In conclusion, our study confirms that the CARE score remains a simple but efficient tool for predicting mortality and major morbidity after cardiac surgery at a center other than that where it was initially developed. Its predictive performance is similar to multifactorial risk indexes, including the EuroSCORE. In addition, the CARE score remains devoid of significant variability when used by different groups of physicians, including surgeons and cardiologists.
The authors thank David Baker, M.D., F.R.C.A. (Staff Anesthesiologist, Department of Anesthesiology, Centre Hospitalier Universitaire Necker-Enfants Malades, Paris, France), for reviewing the manuscript.