Is the Standard Supplied by the Association for the Advancement of Medical Instrumentation the Measure of All Things for Noninvasive Continuous Hemodynamic Devices?

The systematic review and meta-analysis by Kim et al.¹  raise important questions regarding the correct way to technically evaluate the new generation of continuous noninvasive arterial pressure monitors (CNBP).

Continuous noninvasive arterial pressure monitors devices have been designed to extend the physiological information available regarding a patient’s hemodynamic status. As rightly mentioned by De Hert,²  they cannot completely replace invasive monitoring, because blood gas sampling may also be required. However they have the potential to “bridge the gap between noninvasive but intermittent and continuous but invasive arterial pressure measurements.”¹ 

Continuous noninvasive arterial pressure monitors are designed to measure blood pressure (BP) trending data, beat-to-beat rhythms, and waveforms. The BP signal can then be further processed to provide additional, and important insights into dynamic fluid responsiveness parameters, stroke volume, and cardiac output changes. If accurate and clinically useful, these monitors may potentially have quite profound impact on patient care. It is important to build on our understanding of the performance and limitations of these newly available devices.

There is currently no accepted standard way to evaluate the clinical performance of this new generation of monitors. A standard from the Association for the Advancement of Medical Instrumentation (AAMI) is available³  for intermittent noninvasive cuff sphygmomanometers (NBP) by utilizing an upper-arm cuff. In the absence of consistency in reporting accuracy results by the papers reviewed in the meta-analysis, the authors used this AAMI standard benchmark for accuracy and precision (5 ± 8 mmHg). In doing so, the authors’ conclusion was that the precision and accuracy results found for CNBP are outside those deemed acceptable by the AAMI standard. However, we believe that there are some important technical and methodological limitations associated with applying this standard that lead us to suggest using caution with the interpretation and extrapolation of these results.

Firstly, we fully agree with Kim et al. that “most studies evaluating these devices did not report bias and error the way the AAMI recommended, and following the recommendations in that standard would have led to significantly lower error values.” Indeed, it is accepted by Kim et al. that AAMI recommendations^3,4  have not been implemented correctly in any of the studies reviewed in their meta-analysis. In fact, the AAMI criteria define, in a very detailed manner, the way comparisons of intermittent oscillometric sphygmomanometers with intraarterial BP (IBP) measurements should be conducted. The critical analytical method missing from these studies is the use of the “zero-zone” proposed within the AAMI guidelines. The “zero-zone” is calculated using IBP-values of a minimum of 30 s: Beat-to-beat IBP-values are averaged and their standard deviations (SDs) are calculated. This results in the “zero-zones” (mean ± 1 SD), to which the device-under-test is then compared.

Figure 1 illustrates the process to determine differences to the device-under-test derived using the AAMI zero-zone definition. The range of the reference diastolic BP (dBP) is 66 to 74 mmHg, which defines the zero-zone. If the dBP value determined by the sphygmomanometer is 67 mmHg the error for this determination is 0 mmHg! The range of the reference systolic BP (sBP) is 123 to 136 mmHg. If the sBP determined by the sphygmomanometer is 121 mmHg the error for this determination is −2 mmHg.

Using the zero-zone approach, the AAMI benchmark of acceptable accuracy and precision of 5 ± 8 mmHg are wider than suggested in the analysis used by Kim et al. This change in allowable bias is mathematically equivalent to increasing the AAMI benchmark by half of the zero-zone range (the SD of IBP). Adding the maximum allowable ranges of the zero-zones as defined by the AAMI (20 mmHg for sBP and 12 mmHg for dBP)³  results in the benchmarks being extended by 10 to 15 ± 18 mmHg for sBP and by 6 to 11 ± 14 mmHg for dBP. Would that still be a clinically acceptable bias?

None of the studies cited by Kim et al. applied this zero-zone, but simply calculated accuracy as the average (more or less beat-to-beat) difference between CNBP and IBP (for this example −9 mmHg for sBP and −3 mmHg for dBP). It is obvious that this omission has a profound negative impact on the reported accuracy and precision. In our view, the data should be, where possible, reanalyzed in the manner outlined in the AAMI standard. Comparison of the data with the AAMI benchmark standard without such “zero-zone” analysis is not appropriate.

Secondly, the original AAMI approach requires patients to be hemodynamically stable, so that the sphygmomanometer is able to deliver plausible values. Maximum allowable ranges of the zero-zones are exactly defined by AAMI and all data from a subject shall be excluded if the reference zero-zone is greater than the maximum ranges.³  Establishing the accuracy of CNBP in hemodynamically unstable situations like induction of anesthesia^5,6  or even transfemoral aortic valve implantation procedures⁷  provokes additional and challenging questions as to what is the appropriate methodology and what statistics to use? Including studies in the meta-analysis with data gained from unstable patients or patients’ episodes may help explain the high variation in accuracy and precision results reported. As stated by Kim et al., their inclusion criteria may limit the collective comparison of all studies in a single meta-analysis.

The need to continue to explore methodologies for the testing and evaluation of all BP monitoring devices under real-life clinical conditions is emphasized by the results reported by Wax et al.⁸  They analyzed the difference between sphygmomanometer and IBP readings from 24,225 patients in daily clinical routine. The “real-world” average bias (SD) was −1 mmHg (16 mmHg) for sBP and 5 mmHg (11 mmHg) for dBP—without applying a zero-zone.

In conclusion, there is a growing clinical requirement for the noninvasive monitoring of continuous BP, fluid, and stroke volume. Such devices are now available to clinicians and are being used in increasing numbers. It is therefore of major importance to further discuss acceptable and reasonable continuous BP evaluation standards. A new evaluation standard should be defined taking into consideration the inherent beat-to-beat nature and trending capability of these new devices.

CNSystems Medizintechnik AG (Graz, Austria) develops, manufactures, and markets the continuous noninvasive arterial pressure technology. LiDCO Ltd. (London, United Kingdom) has integrated the continuous noninvasive arterial pressure technology into their LiDCOrapid products. The authors are inventors and named on one or more patents of continuous noninvasive technology.

Drs. Fortin and O’Brien are CEOs and founders of their companies, receive salary, and have equity interests. Ms. Lerche and Dr. Flotzinger are employees of CNSystems AG.