Comparison of Differences in Cohort (Forward) and Case Control (Backward) Methodologic Approaches for Validation of the Hypotension Prediction Index

We conducted a retrospective analysis of prospectively gathered anonymized data. This study adhered to the Enhancing the Quality and Transparency of Health Research guidelines.¹⁰  The data were analyzed from subjects from nine previously reported studies.^{2–5,8,9,11–13}  Institutional review board approval was not required because we analyzed pre-existing anonymized data. Analyses were based on invasive arterial waveforms via radial arterial cannulation, and noninvasive arterial waveforms were derived from a finger cuff (ClearSight, Edwards Lifesciences, USA). Detailed inclusion and exclusion criteria are presented in the individual reports, and a summary is provided in supplemental material table S1 (https://links.lww.com/ALN/D526). Briefly, all studies enrolled adults greater than 18 yr of age. Two studies collected noninvasive measurements in subjects undergoing noncardiac surgery.^2,4  Five studies analyzed invasive blood pressure measurements in patients undergoing moderate to major surgery; one included elective total hip arthroplasties,¹²  two included elective major noncardiac surgery only,^8,13  one included elective cardiac surgery requiring cardiopulmonary bypass only,³  and one included a mix of elective major noncardiac surgery and off-pump cardiac surgery.¹  One study included patients in the intensive care unit (ICU) with a diagnosis of COVID,⁹  and one study included a mixed group containing predominately ICU patients but also mixed and surgery that included cardiac, neurosurgical, general, vascular, and thoracic surgeries.⁵ 

From the collected data, variables were computed, including both the index and MAP. All collected variables are shown in supplementary data file 1 (https://links.lww.com/ALN/D527) and supplementary data file 2 (https://links.lww.com/ALN/D528). Our analysis was based on 20-s averages of beat-to-beat values. Poor arterial waveform signals (e.g., line flushing, significantly damped waveforms, and other waveform artifacts) were detected by the arterial beat detection algorithm and excluded from the analysis. Across all studies, 1.1% of data was categorized as unreliable for invasive A-line data, and 2.1% of data was categorized for noninvasive A-line data, equally distributed.

Descriptive statistics are presented as mean (SD) for normally distributed continuous data or median (25th to 75th percentiles) for non-normally distributed data. Normality of data was assessed by using the Shapiro–Wilk statistic. The data set was analyzed in its entirety to describe the total number of hypotensive events (MAP less than 65 mmHg for at least 1 min measured either invasively or noninvasively), absolute duration of hypotension, area under the threshold of 65 mmHg, and time-weighted average of area under the threshold. Hypotension Prediction Index and MAP data were plotted to assess their relationship, and the coefficient of determination was calculated to determine the proportion of the variation in the dependent variable that is predictable from the independent variable. In addition, a reclassification table was created to compare the performance of Hypotension Prediction Index and MAP for the prediction of a future MAP of less than 65mmHg (full details in supplementary material, https://links.lww.com/ALN/D526).

Receiver operating characteristic curve analysis was used to evaluate the performance of variables in predicting hypotension 5, 10, and 15 min before the hypotensive event. The receiver operating characteristic curve analysis was performed with both a backward analysis with dual boundary conditions (gray zone) and a forward analysis using all data points. Analysis was performed separately for invasive arterial waveform and noninvasive waveform data. We also performed receiver operating characteristic curve analysis on MAP, stroke volume (SV), and heart rate (HR) at various thresholds to assess the effect of receiver operating characteristic curve analysis on time series data to predict future values.

No data were available to determine whether potential interventions occurred to treat or prevent hypotension, which could include surgical incision, change in position, or vasopressor treatment, among others. Treatment interventions were assumed when MAP increased more than 5 mmHg within 20 s (most probably caused by vasopressor or inotropic injections) or when MAP increased more than 8 mmHg within 2 min (change in vasopressor or inotropic infusion rate or fluid bolus) when the MAP was less than 75 mmHg. As sensitivity analyses for the effects of excluding presumed treatment periods, we also conducted a forward analysis with a threshold of 10 mmHg MAP change to define an intervention, as well as an analysis with no interventions censored. We used bootstrapping to account for repeated measurements from each subject for the calculation of CI.¹⁴ 

Receiver operating characteristic curve analysis was performed to evaluate the performance of variables to predict a future hypotensive event including the index, MAP, and ΔMAP at different blood pressure thresholds. ΔMAP analysis was performed in the MAP ranges of 75 to 85 and 85 to 95 mmHg. ΔMAP was the difference in two MAP measurements 3 min apart. Our methodology has been described previously.⁵  In summary:

• A hypotensive event was identified as a section of at least 1-min duration, with an MAP of less than 65 mmHg for all three 20-s averages. A positive data point was chosen as the data point at 5, 10, or 15 min before the hypotensive event. All positive data points were included in the analysis regardless of their MAP values.

• A nonhypotensive event was identified as a 30-min continuous section of data points at least 20 min apart from any hypotensive event and with an MAP of greater than 75 mmHg for all data points in that section. A negative data point was chosen as the center data point of the nonhypotensive event to reduce intraclass correlation. Only one data point within a 30-min window was selected because selecting all the data points within the window would have introduced statistical bias because neighboring 20-s data points are highly correlated.

• This approach incorporates a “gray zone” between MAPs of 65 and 75 mmHg, and values within this range were excluded. This is a zone of a quantitative test in which there is an area of values where the discriminatory performance is “insufficient,” in the sense that a value in the gray zone does not allow the target disease to be scored as either present or absent.¹⁵  A flow diagram illustrates the backward methodology in figure 1.

We used receiver operating characteristic curve analyses to evaluate the performance of the same variables as in the backward analyses. As previously, a hypotensive event was defined by an MAP of less than 65 mmHg persisting at least 1 min, with all other pressures being considered nonhypotensive events. In this analysis, every data point except those that were during hypotensive events or that had an unreliable arterial line trace were examined. Every prediction index data point was compared to a threshold (for example 85), and a window of 5, 10, or 15 min immediately after the data point was searched to identify any hypotensive events. No gray zones were used in this analysis.

As in previous work assessing the performance of real-time and continuous risk scores,^4,9,16–19  the analysis sequence was as follows: A true positive prediction occurred when the index was above the designated threshold and there was a hypotensive event in the search window. A false positive prediction occurred when the index was above the designated threshold and there was no hypotensive event in the search window. A true negative prediction occurred when the index was below the designated threshold and there was no hypotensive event in the search window. A false negative prediction occurred when there was a hypotensive event that the index failed to predict at least 2 min in advance. The above process was repeated so that the designated threshold covers all possible values of the index. Similar analyses were also performed for other variables. A flow diagram illustrates the forward methodology in figure 2, similar to published clinical analyses of other real-time and continuous risk scores.^4,9,16–19 

The index versus MAP analysis was performed to assess the relationship between the index and MAP. All paired data points of the index and MAP from all patients were analyzed together; then, for each integer value of MAP, we calculated the mean, SD, minimum, and maximum of the corresponding index values. Similarly, we calculated the mean, SD, minimum, and maximum of corresponding MAP values for each integer value of the index. All statistics were performed with MATLAB (version R2021a; The MathWorks Inc., USA). The data are available upon reasonable request and institutional review board approval. The analysis script can be found at https://github.com/InstabilityPrediction/Analysis2024.

A total of 2,022 patients (913 women and 1,109 men) with a mean age of 61 (14) years were included in the analysis. Based on 20-s recording epochs, there were a total of 4,152,124 data points. Among the enrolled patients, 339 (17%) were from an ICU, and 1,683 (83%) were intraoperative surgical patients. The data were obtained from an arterial catheter in 1,210 patients (60%) and from a ClearSight noninvasive cuff in 812 patients (40%).

The median monitoring time per patient was 230 min (interquartile ratio, 144 to 420 min), and 1,683 of 2,022 patients (83%) had at least one hypotensive event, which was defined as a MAP of less than 65 mmHg for 1 min or longer. In total, 24,654 hypotensive events were detected, and the median number of events per patient was 4 (1 to 10) with a median duration of 2 (1 to 5) min per event. The median cumulative duration of hypotension per patient was 13.3 (2.7 to 44.0) min, which was 5.5% (1.0 to 16.4%) of total monitoring time. The median area under the threshold of 65 mmHg was 79.9 mmHg/min (15,271 mmHg/min), and the median time-weighted average area under the threshold was 0.30 mmHg (0.05 to 0.96 mmHg).

A plot of all paired index and MAP data measured at the same time points is shown in figure 3. The R² for the variation in the index due to MAP was 0.77. Mean, SD, minimum, and maximum values of the index for different MAP thresholds and vice versa are shown in supplemental material table S2 (https://links.lww.com/ALN/D526).

Receiver operating characteristic curves for the index predicting hypotension 5, 10, and 15 min in advance by backward and forward analysis are shown in figure 4 separated for invasive and noninvasive arterial waveform analysis. For invasive arterial waveform data, the areas under the receiver operating characteristics curve (AUC) for the index predicting hypotension by backward and forward methodology, respectively, were 0.957 (95% CI, 0.947 to 0.964; sensitivity, 86%; specificity, 95%) versus 0.923 (95% CI, 0.912 to 0.933; sensitivity, 87%; specificity, 84%) 5 min before the event, 0.933 (95% CI, 0.924 to 0.942; sensitivity, 81%; specificity, 93%) versus 0.923 (95% CI, 0.911 to 0.933; sensitivity, 88%; specificity, 83%) 10 min before the event, and 0.929 (95% CI, 0.918 to 0.938; sensitivity, 81%; specificity, 92%) versus 0.926 (95% CI, 0.914 to 0.937; sensitivity, 87%; specificity, 84%) 15 min before the event.

For noninvasive arterial waveform data, the AUC for the index predicting hypotension by backward and forward methodology respectively was 0.950 (95% CI, 0.939 to 0.961; sensitivity, 85%; specificity, 94%) versus 0.917 (95% CI, 0.908 to 0.925; sensitivity, 88%; specificity, 84%) 5 min before the event; 0.930 (95% CI, 0.914 to 0.944; sensitivity, 81%; specificity, 93%) versus 0.918 (95% CI, 0.906 to 0.928; sensitivity, 87%; specificity, 86%) 10 min before the event, and 0.912 (95% CI, 0.893 to 0.929; sensitivity, 79%; specificity, 91%) versus 0.923 (95% CI, 0.910 to 0.933; sensitivity, 88%; specificity, 86%) 15 min before hypotension. The full details of all analyses for all hemodynamic variables including area under the curve, area under the precision recall curve, sensitivity, specificity, positive predictive value, negative predictive value, and optimal cutoff values are shown in supplemental data file 1 (backward analysis, https://links.lww.com/ALN/D527) and supplemental data file 2 (forward analysis, https://links.lww.com/ALN/D528).

The AUCs using forward analysis for MAP predicting a MAP of less than 65 mmHg for at least 1 min 5, 10, and 15 min in the future were 0.932 (95% CI, 0.920 to 0.940; sensitivity, 86%; specificity, 86%), 0.929 (95% CI, 0.918 to 0.938; sensitivity, 87%; specificity, 85%), and 0.932 (95% CI, 0.921 to 0.940; sensitivity, 88%; specificity, 84%), respectively. Figure 5 shows receiver operating characteristic curves for SV, HR, and MAP predicting a decrease in future values below various thresholds 10 min into the future for invasive arterial waveform data only. All variables have high AUCs in predicting their future values regardless of the threshold chosen. MAP predicted its future value 10 min in advance of being less than 75 mmHg with an AUC of 0.908 (95% CI, 0.898 to 0.917), less than 70 mmHg with an AUC of 0.921 (95% CI, 0.911 to 0.929), and less than 65 mmHg with an AUC 0.929 (95% CI, 0.911 to 0.929). SV predicted its future value being less than 60 ml with an AUC of 0.924 (95% CI, 0.917 to 0.932) and less than 50 ml with an AUC of 0.929 (95% CI, 0.920 to 0.936). HR predicted its future value 10 min in advance, with an AUC of less than 60 beats/min of 0.973 (95% CI, 0.967 to 0.977) and an AUC for less than 50 beats/min of 0.980 (95% CI, 0.968 to 0.987).

The area under the curve, sensitivity, specificity, positive predictive value, negative predictive value, and optimal cutoff for SV, HR, and MAP predicting their future values 10 min into the future at different thresholds are shown in supplemental data file 3 (https://links.lww.com/ALN/D529). A net reclassification improvement is shown in supplementary material table S3 (https://links.lww.com/ALN/D526) with a 45% net improvement in false positives compared to false negatives with the index compared to MAP assigning a 10:1 ratio to emphasize the importance of a nonevent and a 7.6% improvement assuming equal importance.²⁰ 

Using a forward methodology in the analysis of the index to predict hypotension defined as a MAP of less than 65 mmHg for at least 1 min, there was a significant reduction in the AUC compared to our previously reported backward methodology at 5 min only; however, the AUC remained greater than 0.9.^1,5  Thus, the utility of the index to predict future hypotensive events remained excellent.

The original validation used a backward (case control methodology) in which hypotensive events were identified and then looked backward in time to see whether a variable predicted the event. This method means that, unlike the forward approach, interventions that avoid hypotension do not need to be identified. Using a forward approach, due to the lack of contemporaneous information on clinical treatment, potential interventions must be accounted for by predefined rapid changes in MAP, which is a potential weakness. Future studies aimed at validating predictive algorithms will require highly annotated data on interventions to avoid this confounder. However, it is a fair criticism that using a backward analysis does not represent how clinicians use data and that a forward methodology better mimics the flow of data in clinical practice.

As part of the backward methodology, a gray zone or zone of uncertainty was used in which hypotensive events were defined as a MAP of less than 65 mmHg and nonhypotensive events as greater than 75 mmHg. These definitions were selected because they span the most common range of reported harm thresholds for acute kidney injury and myocardial injury after noncardiac surgery; i.e. they have a higher prevalence below the lower threshold and a lower prevalence above the higher threshold.

The use of an uncertainty zone is common in clinical care because there is often ambiguity about disease states, especially at early stages. This type of ambiguity exists in many clinical areas with most measured clinical parameters because physiology is not a binary process, but rather an ensemble of complex and fuzzy processes.²¹  For example, such areas exist in labeling of definitive sepsis and definitive absence of sepsis,²²  between clearly normal and clearly abnormal levels of blood pressure, and in the labeling of brain natriuretic peptide data for diagnosis of heart failure.¹⁵  Using a zone of uncertainty was proposed to evaluate biomarkers.²³  This concept is common in machine learning, and as machine learning applications expand more in the medical field, using the concept of the zone of uncertainty in their development and validation will become critical for the clinical adoption of such technologies. In a sensitivity analysis, exclusion of the uncertainty zone did not significantly alter the predictive ability of the Hypotension Prediction Index.

Enevoldsen and Vistisen,⁶  in a simulated data set, suggested that a selection bias was present because of the definition of a nonhypotensive event and that this bias would cause an overestimation of the predictive ability of the index. In the original validation study, hypotension was defined by a MAP of less than 65 mmHg for 1 min using the full range of preceding MAP values. A nonhypotensive event was identified at the midpoint of a 30-min continuous section of data points at least 20 min away from any hypotensive event and with a MAP greater than 75 mmHg for all data points in that section. Enevoldsen and Vistisen⁶  also concluded that the index was therefore taught that if the MAP is less than 75 mmHg, then the only possible future event is hypotension. This is not how the index was trained, and hence, it does not perform as proposed. At a MAP of 75 mmHg, the index can take any value from 7 (low probability of hypotension) to 98 (high probability of hypotension); therefore, it does not assume that hypotension is the only possible outcome. Using all data points (no selection bias) resulted in only a small reduction in the AUC; however, there is the possibility that the predictive ability is overestimated or underestimated by using points that have high intraclass correlation; hence, the original methodology avoided this, also allowing for a balanced receiver operating characteristic curve analysis. Area under the precision recall curve can be used for analysis of unbalanced classes; however, the methodology uses the positive predictive value. The positive predictive value is not a direct metric to demonstrate the performance of algorithms because it is primarily affected by the prevalence of the hypotension. If prevalence is low, then the positive predictive value is low even if the algorithm has extraordinarily high performance of sensitivity and specificity.

Various investigators have asserted that the index is simply a mirror of MAP^24,25  and that MAP itself can be used to predict and treat hypotension. First, although there is an average nonlinear relationship between MAP and the index, there is a difference when looking at the individual samples. The two variables convey different information, and the relationship between the index and MAP not only changes within individuals at different time points but also varies between different individuals.²⁶  Neither MAP, systolic pressure, nor diastolic pressure are direct features in the index model. All 23 features of the index model are mathematically combinatorial features and represent combinatorial interaction effects and not static effects. There is no single dominant feature that influences the index, as the median influence values are similar.

Using current MAP to avoid hypotension is clearly an obvious solution and represents current practice. However, MAP has been available for hemodynamic management for a considerable period, and yet hypotension remains common.²⁷  Although it is appealing to think the associations of intraoperative hypotension and poor outcomes has changed practice, recent studies have shown that the prevalence of hypotension remans high across all settings, surgery types, and patient groups.^28,29  It is, however, a reasonable question to ask whether current MAP predicts future hypotension, as well as a complex proprietary algorithm.

A simple linear extrapolation of MAP predicts future hypotension better than the change in MAP in patients having high-risk noncardiac surgery,³⁰  although the index was not compared in that analysis. Using the absolute value of MAP to predict future hypotension, rather than linear extrapolation or delta MAP, gives a high AUC as in this analysis. However, this reflects a statistical relationship that arises when using any variable as both the independent and dependent variable, as is shown in this analysis (fig. 4), in which SV, HR, and MAP all predict future values at any threshold in their range with a high AUC. Mathematical coupling of data in statistical analyses has been previously described.³¹  Due to succession of numbers in time series, any variable has high receiver operating characteristic curve when compared to a fixed threshold representing a number in the range of the times series variable. When using the current MAP to predict a MAP of less than 65 mmHg in the future, the receiver operating characteristic curve analysis simply illustrates the likelihood that a MAP value of 65 mmHg will be preceded by other MAP values (e.g., a MAP of 67 or 70 mmHg), which is intuitive because to reach a hypotensive event from a higher MAP, the blood pressure must pass through these values.

If it is assumed that MAP can predict itself, then there are additional clinical considerations. The optimal cutoff to predict a MAP of less than 65 mmHg would be 72 mmHg; therefore, this value should be always treated to avoid future hypotension, as has been suggested.^6,25  This is an extrapolation of a statistical analysis that does not translate into reasonable clinical practice. Clinicians do not treat an absolute value of MAP without examining the previous trajectories. If a threshold of 72 mmHg is taken as an absolute treatment threshold, this would lead to overtreatment. In our data set, taking every MAP of 72 mmHg in the 1,210 patients monitored with an arterial catheter, only 18% led to a MAP of less than 65 mmHg within the next 15 min. Therefore 82% of the episodes would have been treated inappropriately because hypotension would not have occurred. This is of potential concern because increasing doses of vasopressors have been associated with worse outcomes.^32,33  The index should be used in combination with MAP to distinguish patients likely to subsequently develop hypotension from those who are unlikely to, because it is a probabilistic model that provides that information.

The use of the index in clinical practice reduces hypotension compared to using a target MAP^12,13,34  in some although not all trials.⁸  Education of clinicians on the importance of maintaining a target MAP threshold did not result in a reduction in hypotension, despite them believing they had sufficient knowledge and skills. The use of the index reduced hypotension and had perceived clinical utility.³⁵ 

There are several limitations to this analysis. First, this is a retrospective analysis, and data on interventions were not available. Interventions such as certain drug administration, change of ventilation or blood loss will predict hypotensive events, and the analysis does not account for these interventions in the operating room or the ICU. Future studies should include these data and need to evaluate the effect of using the index to minimize hypotension focusing on clinical outcomes such as acute kidney injury, myocardial injury after noncardiac surgery, or a composite outcome rather than numerical measurements of hypotension depth and duration. We thus had to make assumptions about interventions based on MAP patterns. However, the results were similar over a wide range of assumptions, including no adjustment for any interventions. Second, some clinicians had access to the index data, which presumably resulted in some potential future hypotensive episodes being treated before they occurred. However, treatment would generate false-positive predictions, making the index perform worse. Fully addressing this issue will require a data set, presumably obtained in the context of a trial, in which all interventions are well documented.

As predictive tools become more prevalent in clinical practice, consensus will be needed regarding the best methodology for validation of complex algorithms that considers the impact of both statistical bias and clinical utility. In the meantime, our analysis shows that the index software accurately predicts hypotensive events regardless of the validation methodology used.

Support was provided solely from institutional and/or departmental sources.

Dr. Davies, Dr. Fleming, Dr. Veelo, Dr. Maheshwari, Dr. Sessler, Dr. Sander, and Dr. Cannesson have received consultancy fees, speaker fees and funded research from Edwards Lifesciences (Irvine, California). Dr. Mythen was paid consultancy and speaker fees from Edwards Lifesciences. As of August 1, 2023, Dr. Mythen is a paid employee of Edwards Lifesciences. Dr. Mythen recently resigned as a founding Director of the Perioperative Quality Initiative and is also a board member of Evidence Based Perioperative Medicine CiC and co-editor in chief of TopMedTalk. Dr. Hatib, Dr. Jian, Dr. Scheeren, and Dr. Settels are paid employees of Edwards Lifesciences. Dr. Scheeren has received consultancy fees, speaker fees, and funded research from Masimo Inc. (Irvine, California). Dr. Vlaar has received consulting fees from Edwards Lifesciences (paid to institution) and Philips. Dr. Cannesson is the founder of Sironis (Newport Beach, California) and Perceptive Medical (Newport Beach, California) and owns patents and receives royalties for closed loop hemodynamic management technologies that have been licensed to Edwards Lifesciences. Dr. van der Ster declares no competing interests.

Supplemental material: Additional analyses and tables, https://links.lww.com/ALN/D526

Supplemental table 1: Patient characteristics

Supplemental table 2: Corresponding Hypotension Prediction Index and MAP values at different thresholds.

Supplemental table 3: Reclassification table

Supplemental data file 1: Backward methodology full analysis, https://links.lww.com/ALN/D527

Supplemental data file 2: Forward methodology full analysis, https://links.lww.com/ALN/D528

Supplemental data file 3: Analysis of HR, SV, and MAP to predict future values, https://links.lww.com/ALN/D529