We thank Dr. Hyder for his interest in our recent article published in Anesthesiology, “Preoperative Surgical Risk Predictions Are Not Meaningfully Improved by Including the Surgical Apgar Score: An Analysis of the Risk Quantification Index and Present-On-Admission Risk Models.”1 

As suggested by Dr. Hyder, we performed additional analyses using an alternative sampling interval for vital signs and added a calculation of risk reclassification to better test the clinical utility of the Surgical Apgar Score (SAS) when combined with preoperative risk stratification models.

A sampling method for slowest heart rate (HR) and lowest mean arterial pressure (MAP) was established before initiating data analyses. The method was based on “windows” or intervals of data and was established as follows: 10-min nonoverlapping windows, with windows beginning at the time of incision (0 to 10 min, 11 to 20 min, 21 to 30 min, etc.). Within each window, a median value was determined. Median values for HR and MAP were the basis for the original SAS investigations, and median values were chosen for this investigation. Estimated blood loss as recorded by the in-room anesthesia provider was calculated for the entire case.2 

We also added a calculation of risk reclassification to better test the clinical utility of the SAS. The use of a reclassification measure may be applied to provide a more clinically meaningful assessment of change in risk prediction. A concept of categorizing patients into high- and low-risk groups is clinically intuitive and actionable, as we treat high-risk patients differently, such as with admission to the intensive care unit. Traditionally, risk prediction models have been evaluated using the area under the receiver operating characteristic curve, along with model calibration, Brier score, information criteria, etc., but this can be an insensitive measure for model comparison in a healthcare setting, providing little direct clinical relevance. Since its description in 2006, much interest has been generated in reclassification, which assesses the ability of new models to more accurately classify individuals into higher or lower risk strata. This has led to new methods of evaluating and comparing risk prediction models, including the reclassification calibration test and the net reclassification index (NRI). Pencina et al.3  developed the NRI and the integrated discrimination improvement (fig. 1).

Fig. 1.

Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.

Fig. 1.

Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.

After performing analyses using alternative sampling for vital signs and calculating risk reclassification, the Risk Quantification Index and present-on-admission preoperative risk models were not meaningfully improved by adding intraoperative risk using the SAS, as determined by the NRI value of 0.02 (P = 0.10). These analyses supported the original findings: adding the SAS did not substantively improve predictions. In addition to the estimated blood loss, lowest HR, and lowest MAP, other dynamic clinical parameters from the patient’s intraoperative course may need to be combined with procedural risk estimate models to improve risk stratification.

Acknowledgments

This work was funded, in part, by the Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee, and the Foundation for Anesthesia Education and Research and Anesthesia Quality Institute Health Services Research Mentored Research Training Grant, Schaumburg, Illinois (to Dr. Wanderer).

Competing Interests

The authors declare no competing interests.

References

1.
Terekhov
MA
,
Ehrenfeld
JM
,
Wanderer
JP
:
Preoperative surgical risk predictions are not meaningfully improved by including the Surgical Apgar Score: An analysis of the risk quantification index and present-on-admission risk models.
Anesthesiology
2015
;
123
:
1059
66
2.
Hyder
JA
,
Kor
DJ
,
Cima
RR
,
Subramanian
A
:
How to improve the performance of intraoperative risk models: An example with vital signs using the surgical apgar score.
Anesth Analg
2013
;
117
:
1338
46
3.
Pencina
MJ
,
D’Agostino
RB
Sr
,
D’Agostino
RB
Jr
,
Vasan
RS
:
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond.
Stat Med
2008
;
27
:
157
72; discussion 207–12