Development and Validation of an Intraoperative Predictive Model for Unplanned Postoperative Intensive Care

AKEY aspect of providing safe and effective care for surgical patients is determining the appropriate level of postoperative care. Intensive care unit (ICU) admission allows for close monitoring and rapid interventions, yet this resource is limited and expensive.¹  Identifying the postoperative patients who require ICU admission is a challenging but necessary daily task. Having advanced notice of ICU needs is essential in the allocation and management of limited ICU resources. ICU requests that can not be accommodated due to lack of resources are associated with increased costs and morbidity,²  as are delays in ICU transfer.³ 

A variety of strategies have been used to reduce the number of unplanned ICU admissions. Preoperative assessment clinics have been demonstrated to reduce unplanned ICU admissions,⁴  and patient risk factors for unplanned ICU admission have been identified for specific types of surgeries.^5–7  An alterative and complementary strategy is to identify patients who may need postoperative ICU admission in near-real time so that appropriate resources can be arranged in advance. Algorithms that continuously read patient care data feeds to identify patients at risk are referred to as “sniffers” and have been deployed successfully to identify sepsis,⁸  acute lung injury,⁹  transfusion-related acute lung injury,¹⁰  acute kidney injury,¹¹  and deterioration of patients on general care floors.¹² 

Creating a sniffer capable of identifying postoperative care needs in real time for patients’ ongoing surgical procedures requires the development and validation of a predictive model. Although many operative cases that need postoperative ICU admission are identified in advance, methods of predicting unanticipated ICU bed needs for surgical cases have not been developed. The goal of this study was to develop and internally validate an intraoperative predictive model for unplanned postoperative ICU use, based on anesthesia information management system (AIMS) data.

This study received approval from the Partners HealthCare Institutional Review Board (Boston, Massachusetts; protocol 2011P000253). We identified patients aged 16 yr and older who had anesthetics between March 2007 and May 2011 at the Massachusetts General Hospital during an in-patient admission. After excluding anesthetics for cardiac surgery, electroconvulsive therapy and obstetrics, 71,996 anesthetic records were included for analysis in this study.

Length of stay (LOS) and mortality endpoints, patient age, and intensive care billing data were derived from the Research Patient Data Repository. This repository is a data warehouse that contains patient data from multiple Partners HealthCare System hospitals and is updated with data from the National Death Index on a regular basis. For each anesthetic, the Massachusetts General Hospital’s operating room scheduling system was used to determine the primary surgical service and whether postoperative intensive care was planned. Surgical services that had above-average use of postoperative intensive care beds were designated as high-risk surgical services, which at our institution were vascular, thoracic, transplant, radiology, neurosurgery, and general surgery. Postanesthesia care unit (PACU) data were used to determine whether patients were admitted to the PACU after surgery, which was available between April 2008 and May 2011. Our institution uses MetaVision (iMDsoft, Needham, MA), which permits direct, real-time querying of all data through Microsoft SQL Server (Microsoft, Redmond, WA) database. All intraoperative variables were obtained from the Massachusetts General Hospital’s MetaVision AIMS database. Categories of intraoperative variables evaluated in the study include patient variables, case details, medications given, venous and arterial access, fluids administered, laboratory values, and physiologic variables (table 1). Anesthetic records that were missing critical data such as case start and end times were excluded from the analysis (fig. 1). A significant fraction of records had missing estimated blood loss (EBL) data, a finding consistent with previous analysis of EBL data from AIMS records.¹³ 

Patient variables obtained from AIMS data were American Society of Anesthesiologists (ASA) physical status and documentation of a difficult intubation in the current anesthetic record. Case details were length of case, case start during daytime hours (7:00 AM – 7:00 PM), case end during daytime hours, case handoff between anesthetists (residents or nurse anesthetists), case handoff among attendings, and EBL. Analysis of medication administration was restricted to vasoactive medications, specifically phenylepherine, norepinephrine, epinephrine, dopamine, and vasopressin. Central venous access was inferred from documentation of the placement or presence of a central line, whereas physiologic invasive mean arterial blood pressure values (>25 mmHg, <180 mmHg) were used as a surrogate for arterial access. A delayed arterial line was defined as the absence of invasive blood pressure values for a period of time (60 and 90 min), starting with the induction of anesthesia followed by the appearance of invasive blood pressure recordings after that time period elapsed.

Intravenous fluids were included in the analysis, specifically crystalloid solutions (lactated Ringer’s and normal saline), albumin, 6% hydroxyethyl starch, erythrocyte units, fresh frozen plasma, cryoprecipitate, and intraoperative blood salvage. Intraoperative laboratory values, defined as values that were determined between the start and end of the anesthetic record, were restricted to hemoglobin, hematocrit, white blood cell count, platelet count, serum sodium, serum potassium, and serum pH. The presence of any of these values was recorded, as were the subsets of pH values less than 7.20 and hemoglobin values less than 7 g/dl.

In order to reduce error from spurious input from monitoring devices, all physiologic data were subjected to filtering techniques intended to mitigate the impact of isolated irregular values. Physiologic data filters were implemented using Microsoft SQL Server and Microsoft Visual Basic.NET. Average heart rate was computed as a mean value across the length of the anesthetic. The minimum Spo₂ to Fio₂ ratio was computed by iterating through 5-min blocks of time during the anesthetic. At each block, median Spo₂ and median Fio₂ values were independently calculated. A ratio measurement was made only when both variables had at least three values during a block. The minimum ratio across all blocks was thus computed. The count of mean arterial blood pressures below specific thresholds (50 and 65 mmHg) was determined by iterating through 6-min blocks of time because it is common practice at our institution to cycle blood pressure cuffs every 3 min. The median value of all noninvasive mean blood pressures and invasive mean blood pressures was computed for each block, and each median below the designated threshold was counted.

A composite Surgical Apgar Score¹⁴  was computed using EBL, minimum intraoperative heart rate, and minimum intraoperative mean arterial blood pressure. Both the minimum heart rate and mean arterial blood pressure were calculated using medians of 5-min blocks as previously described. Missing EBL data for calculation of the Surgical Apgar Score were imputed as zero.

Use of intensive care services was defined on a per-day basis from billing data as identified by any Current Procedural Terminology code for critical care evaluation (99291 or 99292) or a record of a billed ICU day. These billing codes are a reliable record of which patients receive critical care services on a per-day basis. Records of billed ICU days were used to validate Current Procedural Terminology–based billing codes. Additionally, a subset of 100 records was selected at random and manually checked against physician notes to validate the accuracy of these billing codes. These data were combined with the anesthetic data set described above and an ICU booking flag from our operating room scheduling system. Planned postoperative ICU admission was defined as any record with an ICU booking flag and ICU use on the day of surgery, or use of an ICU on the day before surgery. The percentage of patients with ICU bookings but no actual ICU use was computed. Unplanned postoperative ICU use was defined as receiving intensive care services on the day of surgery or within 6 h after the end of surgery combined with the lack of an ICU booking flag. Emergency cases requiring postoperative ICU use were, by definition, unplanned. Patients who experienced intraoperative death were excluded from analysis. Table 1 compares the characteristics of the cases of planned and unplanned postoperative intensive care. After creating the composite data set and categorizing the records as described above, 8,983 anesthetics with planned ICU admission were removed.

Data analysis was performed using SSPS version 17.0 (IBM, Armonk, NY) and R version 2.15.1 (R Foundation for Statistical Computing, Vienna, Austria) with the rms package. Normality was assessed using Shapiro–Wilk analysis. Continuous variables with a normal distribution are expressed as means and SDs. Nonnormally distributed continuous variables are expressed as medians with 25 and 75% percentile ranges. Categorical variables are expressed as counts and percentages. Categorical variables were compared using chi-square analysis whereas continuous variables were compared using the Mann–Whitney U test.

Univariate predictors of unplanned ICU admission were identified using simple logistic regression modeling. A multivariate logistic regression was constructed using a forward, stepwise likelihood ratio approach to variable selection, initially with all variables available for inclusion. Using this process, EBL was not included in the model. The process was repeated including all variables with the exception of EBL, which established the variables selected for the final model. Odds ratios for model variable, CIs, and P values are displayed for the logistic models. Internal validation was performed by bootstrap using 50 iterations as described in detail below. Overall model performance was assessed with the Brier score. Model calibration was assessed with a calibration plot also using bootstrap with 50 iterations. The final model was assessed for multicollinearity by performing a linear regression with the same variables and obtaining the variance inflation factor for each variable. Variance inflation factor values ranged from 1.019 to 1.460, all below the threshold of 10, which would indicate problematic multicollinearity. Additionally, we generated a correlation matrix for the final model variables that demonstrated bivariate correlations that ranged in absolute value from 0.003 to 0.323.

After constructing the multivariate logistic regression model, we evaluated the performance in predicting unplanned ICU admission at five different time points using bootstrap validation with 50 iterations. The model was evaluated at the time of the case booking by restricting model inputs to patient age, ASA physical status, emergency case status, and surgical service. The end of the case was defined as the time at which the patient left the room, whereas the start of the case was defined as the beginning of continuous care. We additionally created three interval time points, which were defined as 1, 2, and 3 h from the end of each case. Each of the 12 time-dependent variables was computed for each of the three interval time points using only data that would have been available before each respective time point. Additionally, the performance of the Surgical Apgar Score was evaluated in a similar fashion using all available data. Additionally, predicted probabilities of unplanned ICU admissions were generated using the logistic regression model by applying the logistic function probability=1/ (1 + e^−β), where β represents the summation of the model constant and the covariates.

Mortality endpoints were calculated at hospital discharge and at 30 days from discharge. LOS for the hospital admission represents the total number of days in the hospital, whereas ICU-free days represents the hospital admission LOS less the number of days spent in the ICU. LOS and mortality endpoints were used to compare planned and unplanned postoperative intensive care. Effects of time were tested by comparing the overall in-hospital mortality during the first half of the study period with the mortality in the second half of the study period using chi-square analysis. The effect of time on the percentage of unplanned ICU admissions was analyzed in the same manner.

Of the 71,996 anesthetic records in the data set, 6,359 had requested postoperative ICU admission (8.8%), 4,233 were in the ICU preoperatively (5.9%), and 4,847 had unplanned ICU use (6.7%). Of the 4,000 unplanned ICU admissions for whom PACU data were available, 3,509 went directly from the operating room to the ICU (87.7%). Of the requested ICU admissions, 1,609 did not use the ICU postoperatively (25.3%).

We compared the preoperative and intraoperative characteristics of the cases that received routine postoperative care, cases with planned ICU admissions, and cases with unplanned ICU admissions (table 1). There were a number of significant differences among these patient populations. Compared with planned ICU admissions, patients with unplanned ICU admissions were older, had a higher ASA classification, and were more likely to have had emergency procedures. Those procedures started and ended outside of daytime hours more frequently and were more likely to have had attending and anesthetist handoffs. Epinepherine and phenylepherine were used more frequently in these anesthetics. No difference was found in the rate of central line use, and fewer arterial lines were used; however, cases with unplanned ICU admission had a higher rate of delayed arterial line placement. Labs were drawn more frequently for this population, with no detected difference in severe acidosis (pH <7.20) but an increase in severe anemia (Hb <7.0 g/dl). Difficult intubations were more frequent. Average heart rate was higher, as was the number of episodes of profound hypotension (mean arterial blood pressure <50 mmHg for 6 or more min). Patients with unplanned ICU admission had higher EBL, more crystalloid administered, and received erythrocyte units, albumin, 6% hydroxylethyl starch solutions, and cryoprecipitate more frequently. They received fresh frozen plasma and salvaged blood less frequently. Patients with unplanned ICU admission also had lower Surgical Apgar Scores.

Compared with routine postoperative care, patients with unplanned ICU admissions were older, had a higher ASA classification, and were more likely to have had emergency procedures. Those procedures started and ended outside of daytime hours more frequently, were longer, and were more likely to have had attending and anesthetist handoffs. Epinepherine, norepinephrine, vasopressin, dopamine, and phenylepherine were used more frequently in these anesthetics. Central and arterial lines were used more frequently and there was a higher rate of delayed arterial line placement. Labs were drawn more frequently for this population, with more severe acidosis and severe anemia (Hb <7.0 g/dl). Difficult intubations were more frequent. Average heart rate was higher, as were episodes of hypotension. The number of cases with a ratio of Spo₂ to Fio₂ less than 1 was higher. Patients with unplanned ICU admission had higher EBL, more crystalloid administered, and received erythrocyte units, albumin, 6% hydroxylethyl starch solutions, fresh frozen plasma, and salvaged blood and cryoprecipitate more frequently. These patients also had lower Surgical Apgar Scores.

There were no data entries for EBL in 25,866 cases (40.1%), which is represented as a zero value within the AIMS user interface. As described above, this variable was not selected by forward, stepwise regression for inclusion in the model.

Univariate predictors of unplanned ICU admission identified using simple logistic regression modeling are displayed in table 2. All variables reached statistical significance with the exception of difficult intubation.

A multivariate logistic regression model was developed as described above. Of the initial 34 variables, 16 remained in the final multivariate model (table 3). Of those variables, 12 rely on intraoperative documentation and automated record-keeping. The remaining four variables are patient age, ASA physical status, emergency case status, and high-risk surgical service. The model constant was −7.193.

The area under the curve for the receiver operating characteristic was 0.905 (95% CI, 0.905–0.909) for the data set that the model was derived from, and the Brier score was 0.045. The performance of the model for prediction of unplanned postoperative intensive care usage was evaluated at multiple time points as previously described using bootstrap validation. The area under the curve was 0.513 at case booking, 0.688 at 3 h before case end, 0.738 at 2 h before case end, 0.791 at 1 h before case end, and 0.809 at the end of the case. By contrast, the area under the curve for the Surgical Apgar Score receiver operating characteristic curve was 0.696 (table 4). Model calibration was assessed by construction of a calibration plot, which demonstrated good calibration (fig. 2).

The receiver operating characteristic curve provides a plot of the true-positive rate and the false-positive rate at varying thresholds. Clinically, this model is intended to be used as a screening tool to automate the identification of surgical cases that should be evaluated for postoperative ICU care, thus a threshold should be chosen to balance ICU planning needs with resources required to investigate ongoing surgical cases. With a threshold of 5% predicted probability of unplanned ICU admission, the model is positive for 4,145 of 4,847 unplanned ICU admissions (true positive) and for 11,661 of 58,166 routine postoperative admissions (false positive). At this threshold, the model has a sensitivity of 0.86 and a specificity of 0.80. With our unplanned ICU admission prevalence of 4,847 of 63,013 planned routine admissions, the positive predictive value of the model at the same threshold is 26.3% and the negative predictive value is 98.6%. Of the 4,000 unplanned ICU admissions for whom PACU data were available, 491 (12.3%) had postoperative escalation of care. In this subset using the same predicted probability threshold, the model is positive for 247 of 491 unplanned ICU admissions (50.3%).

We compared mortality and LOS outcomes between planned and unplanned ICU admissions (table 5). The median ICU-free days for unplanned ICU admissions was 5 days (25% percentile 1, 75% percentile 9 days), which was slightly more than the 4-day (1, 9 days) median ICU-free days for planned admissions (P < 0.001). Median hospital LOS was 9 days (5, 15 days) for unplanned ICU admission, which is less than the 10-day (5, 24 days) median stay for planned admissions (P < 0.001). There was no detectable difference for in-hospital mortality between unplanned and planned ICU admissions (5.6 vs. 6.0%; P = 0.248) or 30 day mortality (7.9 vs. 8.0%; P = 0.847). There were no changes in overall in-hospital mortality between the first half and the second half of the study period (1.6 vs. 1.5%; P = 0.264) whereas there was a slight increase in the percentage of unplanned ICU admissions in the second half of the study period (6.2 vs. 7.2%; P < 0.001).

We have developed and internally validated a model for prediction of unplanned postoperative ICU admission. Our study has demonstrated that it is feasible to use near real-time AIMS data to reliably identify patients who may require postoperative ICU care from those who do not. These cases are marked by a higher ASA physical status classification, more frequent emergency case status, placement of arterial and central lines, use of epinephrine, vasopressin and norepinephrine, high-risk surgical services, blood product transfusion, higher heart rates, worse Spo₂/Fio₂ ratio, intraop laboratory studies, and prolonged cases. Of these, the most powerful markers were arterial line placement, emergency case status, and epinephrine use. Some markers, such as average heart rate, were significant in the model but had median values that were physiologically similar.

We chose to evaluate our model at multiple time points to understand how it might perform in actual clinical circumstances. At our institution, it is customary to give the ICU notice at least 1 h before arrival, thus this time point is the most relevant in determining whether this model would be clinically useful. Often further advanced notice is beneficial, so we included earlier time points as well. The Surgical Apgar Score is to our knowledge the only validated acuity scoring system that relies on intraoperative data, so we chose that as our comparator and used the end-of-case time point that the scoring system was developed on. Bootstrap validation indicated that our model was superior from 2 h before case end to case end.

A number of alerting systems have been developed that use near real-time AIMS data. These systems have been demonstrated to improve documentation,¹⁵  increase compliance with antibiotic administration,¹⁶  encourage appropriate administration of antiemetics,¹⁷  and remind anesthesia providers to reactivate physiologic alarms after separating from cardiopulmonary bypass.¹⁸  The latter example uses pulse pressure thresholds and mechanical ventilation status derived from AIMS data to detect the separation event and triggers an electronic alert visible to the in-room providers. By contrast, we envision a sniffer system that uses our model to detect cases to be evaluated for postoperative ICU admission and electronically notifies the anesthesia personnel supervising the operating environment. Given the model’s relatively low positive predictive value in our patient population, it would not be appropriate for ICU admission decisions to be made on the basis of the model output; rather, this system could be used to identify cases that require investigation by experienced clinicians to see whether further resources are necessary. Thus appropriate resources can be mobilized without distracting the in-room anesthesia team from patient care. Alternatively, rather than flagging individual cases, this model could be used to predict the aggregate number of unplanned ICU beds likely to be necessary, which might help inform the process of ICU bed management.

Interestingly, unplanned ICU admissions in this study were associated with more ICU-free days and shorter hospital stays compared with planned ICU admission with similar rates of mortality between the groups. This is in contrast to studies that have found higher rates of mortality among unplanned ICU admissions.^19–21  Likely this reflects a difference in patient populations. The aforementioned studies analyzed predominately medical ICU admissions, where ICU admission may reflect worsening of underlying systemic disease. Unplanned postoperative admission to the surgical ICU population, by contrast, can be indicated for managing short-term physiologic stress, such as after an unexpectedly large blood loss, transient inflammatory response to surgery, or the need for temporary mechanical ventilation after substantial fluid shifts.

Our rate of requested postoperative ICU beds that were not used was unexpectedly high (25.3%). This may reflect a triage approach to allocating a scare resource, where requested postoperative ICU beds are given to emergency cases that develop rather than to planned elective cases. Alternatively, it may be that the threshold for requesting an ICU bed may be relatively low for some surgeons at our institution and thus a significant fraction is in fact not needed, or that such a determination is not made until the end of the surgical case or finally that some surgeons are not sufficiently involved in the request process to provide appropriate guidance. Understanding the factors that lead to unused requested ICU beds is as important as understanding the factors in unplanned ICU admission in terms of appropriate resource allocation.

This study has several limitations. Substantial effort was made to filter physiologic variables in the data set to limit artifact but it is possible that not all artifacts were eliminated. This would have the impact of reducing the discrimination capacity of the model. This model contains many variables, which would make it difficult to manually calculate and use. However, it is intended to be implemented programmatically and thus has not been optimized for manual calculation. As this was a retrospective study, no standard criteria for ICU admission were applied and not all ICU admissions might have been appropriate. This model may not be applicable or helpful at another institution with different ICU admission criteria or workflow for providing ICU services. Prediction of ICU needs also may not improve allocation of ICU beds if other factors such as limitation of beds prevent process improvement. Additionally, ICU use was determined by provider and bed billing data, which specify the day but not the time of use. This is one of the limitations of the administrative data set that we used for the study. As only intraoperative data were analyzed in this study, physiologic data in the PACU or on the floor would not have been included. This also may have reduced model performance. Additionally, our data set lacked EBL data for 40% of the records studied. Although this compares well with a recent study that found EBL data missing in 62% of records studied,¹³  improved documentation of EBL may have led to the development of a model that included this variable and may have had better performance compared with the model derived in this study. Finally, we did not perform an external validation of the model that we derived, and additional work in a different study population would be required to externally validate the derived model.

In conclusion, we have developed and internally validated an intraoperative predictive model for unplanned postoperative ICU use based on AIMS data. Incorporation of this model into a real-time AIMS data sniffer may improve the process of allocating ICU beds.

The authors thank Warren S. Sandberg, M.D., Ph.D., Professor and Chair, Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee, for his inspiration and support.