To the Editor:
We would like to thank Drs. Wakeam and Hyder1 for their excellent discussion and description of reliability adjustment in the recent issue of Anesthesiology. The authors correctly highlight the important role that the statistical analysis of data submitted to the various registries can play in the ranking of institutions. This is particularly important now that the Centers for Medicare and Medicaid Services requires providers to participate in a Physician Quality Reporting System2 using a Qualified Clinical Data Registry. These requirements are a precursor to altering physician payments based upon measures of care quality.
We would like to raise the issue of another area of “ reliability”: the reproducibility of the underlying data themselves. While some registries such as National Surgical Quality Improvement Program do periodic data audits and have well-described accuracy thresholds,3 many do not. In fact, some registries, including the Anesthesia Quality Institute and the American Society of Anesthesiologists Perioperative Surgical Home initiative, allow for widely divergent methods of data collection, yet lump these data together assuming they are comparable. For example, one group might define postoperative nausea and vomiting based on postanesthesia care unit antiemetic administration, while another bases it on direct patient interviews. Other registries, such as some maintained by the National Quality Foundation, utilize administrative claims data, which have been shown to be discordant with data collected by other methods.4–8 Despite these very different methods of data collection, all of these examples are considered equally valid national quality registries.
We find the idea that the underlying data used in these registries may be inconsistent to be worrisome. Ideally, the data on patients in various registries should be identical regardless of the method by which they were collected. At the very least, even if the data are not identical between registries, it is critical that within a registry, the data from various sites be of equal quality and have the same definitions, something the major registries in our own specialty lack. If the data inputs are not consistent, we are left with the question of which data to believe, and the conclusion is that the risk adjustment models used may be unable to control for patient-specific risk factors the way they are intended.
It seems inevitable that in the near future, providers will be compared to each other and paid partially based on these comparisons. This concept is based upon the unverified supposition that we can effectively compare patients across institutions. On the basis of the current landscape, we find this supposition unlikely, and we are concerned that using these inadequate tools may lead to incorrect choices in the near future. Drs. Wakeam and Hyder are absolutely correct that “big data” require more than assembling a large sample size and assuming that the “N” will solve the problem, but rather a thorough understanding of statistics and attention to detail. Unfortunately, it seems that the goals of some of the quality registries are outpacing the science behind them.
The authors declare no competing interests.