Complication Rates in Anesthesia: Why Retrospective Studies May Be Extraordinary

There are relatively few large studies describing perioperative complications in children. In this issue Gleich et al.¹  describe the complication rates after perioperative arterial cannulation in children. In this single-center retrospective study, data were reviewed for all children aged less than 18 yr receiving anesthesia between 2006 and 2016. A total of 5,142 cannulations were performed in 4,178 children. Two-thirds of the cannulations were radial, and about 30% were femoral. There were 11 major complications, all when the femoral artery was cannulated and all in children less than 5 yr of age. Eight of the major complications were vascular, and three were infectious. The vast majority of femoral cannulations were for cardiac procedures.

The data would imply that radial cannulation is a relatively safe procedure in children. It might also be inferred that femoral cannulation has a small but relatively high risk of complication, especially in younger children. These data are perhaps not surprising, because complications after femoral artery cannulation have been described before in several small studies where the artery had been cannulated for a variety of reasons. A strength of this study is its size. There are, however, some interesting limitations. The first limitation is that it is single center. Hospitals tend to follow particular practice patterns. At this center there was an unusually high rate of femoral cannulation, which may reflect their high cardiac surgery case load and their preference for femoral cannulation in this particular patient population. This may limit generalizability of their data. The other limitations relate to the retrospective nature of the study. Retrospective studies rely on accurate data entry and coding. Although positive cases can usually be verified by closer inspection of the record, such retrospective studies almost inevitably miss some cases. The true rate of complications was probably higher than that described in the article.

The other limitation of retrospective studies is more subtle and particularly applies to rare events, such as complications. Why did the authors do the study? In the introduction they indicate that they did the study “based on clinical observational of several complications related to femoral lines.” In other words, they suspected there was a problem, so they went back and systematically examined the data; and sure enough, they found a high complication rate in femoral lines when compared with radial lines. These data are valid, but are they generalizable? Let’s assume a bunch of hospitals are similar in terms of patient load and anesthesia work practice. If complications are multifactorial, which they often are, we would expect the number of complications to vary randomly across hospitals. Some centers will have more of one type of complication simply owing to chance. Complications in pediatric anesthesia are generally rare and usually examined carefully in morbidity meetings or some form of quality process. A string of complications would attract attention. This could trigger a more detailed review, as happened in this case. So was the string of femoral line complications attributable to randomness, and this center was just unlucky, or did they uncover a fundamental issue with the technique that would be applicable to all centers? In such a single-center retrospective study there is no way of knowing. The fact that the authors looked because they had a suspicion might imply a random cluster of complications had triggered their investigation, or you could equally argue that this center simply has a more finely tuned awareness of complications.

The problem isn’t only applicable to high rates of complications. This journal has published similar studies where there were low rates of complications. For example, one retrospective single center study looking at central venous cannulation in preterm infants found a particular technique had a high success rate and very low complication rate.²  Does this imply that this is a highly effective and safe technique, or had they just been lucky? If they had had more failures, or more than one or two complications, would they have bothered going back to systematically look at the data? Would they have written the data up into a manuscript? Would it have seemed interesting enough to be published?

This problem of bias attributable to clusters of events (or lack of events) is subtly different from the problem of data mining where electronic datasets are examined retrospectively in multiple ways and significant associations found simply because of the sheer number of analyses. The bias attributable to clusters is more akin to data-driven research rather than data mining or P hacking.

Of course, we shouldn’t dismiss these data simply because there may be a bias. Such retrospective analyses of complications are tremendously important for defining the risk of rare events. So how do we mitigate the risk of bias? The first step is for journals to ask why the authors did the analysis. Was there a trigger? This may provide a clue of a bias. If so we could ask to exclude the index cases, but this is not as simple as it seems when there may have been more than one complication that triggered the review, and indeed excluding the index cases may lead to underestimating the true event rate. The ideal strategy would be to do a prospective study, but this may be impractical if the event is rare. Another strategy could be to do a multicenter retrospective study where large numbers of cases are included from centers that had nothing to do with the index cases that sparked the decision to do the review.

The ideal way to quantify complications is through prospective multicenter registries with clearly defined a priori queries that capture complications. There are numerous of these already in a variety of anesthesia subspecialties. In pediatric anesthesia alone they range from the very large and broad, such as Wake Up Safe,³  to the more focused such as the Pediatric Difficult Intubation Registry.⁴  In the United Kingdom there has been a series of nationwide prospective audits aimed at quantifying rare complications.⁵  However, there will always be some data about some complications or rare events that are not captured in these registries. For these we will continue to rely on retrospective studies. Whenever you see these, look to see why the authors did the study and consider whether the extraordinary finding is just extraordinary.