THIS issue of Anesthesiology contains two fascinating outcome studies. Cheney et al. 1provide a trend analysis of the proportion of closed malpractice claims involving death or permanent brain damage reported to the American Society of Anesthesiologists Closed Claims Project. Lienhart et al. 2present an innovative approach to identifying specific causes of anesthesia-related deaths in France. Both authors describe their unique sampling methodologies and peer review processes, along with some limitations, but do not fully acknowledge the effect that these limitations could have on their conclusions. Shakespeare’s Hamlet pondered the question “to be or not to be,” but for the readers of this issue of Anesthesiology, to see or not to see—that is the question.
Cheney et al. use the American Society of Anesthesiologists Closed Claims database to determine changes in the proportion of claims for death or permanent brain damage over a 26-yr period and to identify factors associated with the observed changes. Their methodology includes a structured evaluation of adverse outcomes from 6,894 closed anesthesia malpractice claims that shows that the proportion of claims for death or brain damage decreased between 1975 and 2000. The authors conclude that the significant decrease in the proportion of claims for death or permanent brain damage from 1975 through 2000 seems to be unrelated to a marked increase in the proportion of claims where pulse oximetry and end-tidal carbon dioxide monitoring were used. That is, they could not find an association between the increase in monitoring and decrease in death or permanent damage.1
This should not come as a surprise, because malpractice claims, in general, may be unrelated to the quality of patient care.3–5In fact, one study, specific to anesthesia care, found absolutely no relation between human errors by anesthesiologists that resulted in disabling patient injuries and subsequent risk of malpractice litigation. In that study, the investigators identified 13 cases in which human error by an anesthesia provider, as determined by peer review, resulted in patient injury over a 3-yr period. None of those 13 incidents resulted in a closed malpractice claim or even a letter of intent. During that same time period, 18 cases involving legal action directed at the anesthesiologists were judged by peer review to be devoid of human error with regard to the anesthetic management.3
Still, the proponents of closed claims analyses argue that the use of such data circumvents the problem of gaining access to low-frequency adverse events, despite inherent limitations that must be considered when interpreting the data.6Among these limitations is the lack of any real denominator data. Specifically, the authors do not know how many anesthetics resulted in the number of closed claims present in their database. Instead, the authors use the total number of closed claims reported to the database annually as a denominator, and report their outcome measures as a proportion of all annual claims. The authors admit that the number of annual claims reported by a select group of insurance companies is not a random sample,6but have they acknowledged the impact of changing the insurance companies over time? The current study reports that the Closed Claims Project has 18 insurance organizations in its active panel, but as many as 35 insurance companies have contributed to the database over the study period. It seems likely that changing insurance companies might produce a change in the number and type of annual claims. For example, the authors excluded claims from 1970 through 1974 and 2001 because there were “insufficient” numbers (n = 21 and n = 15, respectively) per year for meaningful analysis. With so much variability in the denominator data, the readers must be cautious when interpreting trends.
Readers must also give consideration to the numerator data used by the Closed Claims Project. As noted previously, not all patient injury due to anesthesia provider error results in a claim. Also, closed claims can occur in the absence of human error.3For example, in the current study, overall standard of care was judged as less than appropriate in only 28% of the cardiovascular-related damaging events. Therefore, closed claims occurrences may not be a valid indicator of patient safety or quality of care. If the frequency of closed claims were a valid indicator of patient safety, one might conclude that 2001 was a very “safe” year for anesthetized patients.
The proportion of claims for death and brain damage is as likely to be affected by legal practice as it is by medical practice. Malpractice attorneys operating under a contingency-based system are becoming more inclined to pursue litigation involving disabling injury leading to lost wages in younger clients than to pursue cases involving death or permanent brain damage in older patients because of trends toward caps on pain and suffering. Similar economic motivations might cause insurance companies that represent hospitals and physicians simultaneously to settle on behalf of the hospital with a larger award, in return for the physicians being dropped from the claim. This allows the insurance company lawyer and plaintiff’s attorney to settle with the cost of only a single negotiation. The physicians are then able to continue to generate income for the hospital without wasting time in litigation. Some malpractice insurers have even offered discounted premiums to physicians who gave up their right to refuse settlement. This allowed the insurance companies to determine whether it was cheaper to settle or defend against a claim based purely on the costs of doing so. Cheney et al. argue that it is unlikely that plaintiff’s attorneys became more inclined to sue for less serious anesthesia-related injuries over the 1975–2000 time period because premiums for professional liability insurance decreased from approximately $30,000 per year in 1985 to $20,000 in 2005. Perhaps practitioners became more inclined to settle, rather than defend against, these smaller claims when it became apparent that there was little relation between quality of care and malpractice litigation. This could have resulted in significant savings in litigation costs for the insurers and a subsequent decrease in premiums.
Although closed claims databases are an important source of outcome data for risk management, the inferences drawn in this study with respect to monitoring devices are unrealistic. If the downward trend in the proportion of claims involving death or brain damage had begun in 1985, would the authors have concluded, as closed claims investigators have in the past,7,8that pulse oximetry and end-tidal carbon dioxide monitoring has an impact on mechanism of injury or outcome? Closed claims analyses should never attempt to show efficacy of monitoring devices because the population denominator remains unknown. One could also argue that the numerator of adverse outcomes is equally unknown because not all adverse outcomes result in a claim, particularly a closed claim. The fact remains that we have no idea what causes a closed malpractice claim. Closed claims investigators often try to relate claims occurrences to patient management, but claims may bear little relation to any aspect of anesthesia care (not just pulse oximetry or end-tidal carbon dioxide monitoring).
In the second outcome study, Lienhart et al. 2introduce new methodology to estimate the number and characteristics of anesthesia-related deaths in France for 1999. They then compare their estimate with data from a previous nationwide study by Tiret et al. ,9who used different methodology between 1978 and 1982, to suggest a 10-fold decrease in the rate of anesthesia-related deaths in France during this 20-yr time frame. The authors of the current study again acknowledge many of the limitations of their methods but are confident in their conclusions because, in their opinion, many of these limitations are likely to lead to underestimates of the authors’ more recent anesthesia-related mortality rate. As such, the authors’ innovative approach to identifying specific causes of anesthesia-related deaths in France may be both a strength and a weakness.
As in the previously discussed closed claims study, the investigators lack denominator data. Therefore, to calculate an anesthesia-related mortality rate, they must estimate the number of patients receiving an anesthetic in France during 1999. They do this by using data from a 1996 French survey.10Although Lienhart et al. acknowledge that this is merely an estimate, they rationalize the potential effect that this could have on their conclusions by saying, “The error, if any, again seems minimal because data from the French Ministry of Health providing the number of procedures and their type do not show any increase in anesthetic activity during this small time frame.” The authors go on to say that a small reduction in surgical activity, between 1% and 5%, is even suggested by the French Ministry of Health. Readers may wonder why an estimate from 1996 data were necessary if the French Ministry of Health collects this data, but let us put that aside for a moment. When the 1996 survey data were published in Anesthesiology in 1999 by Clergue et al. ,10they concluded that the number of anesthetic procedures had increased by 120% and the rate of anesthetic procedures had increased from 6.6 to 13.5 per 100 French inhabitants since 1980. Clearly, the number of anesthetic procedures has the potential to change over time and, more importantly, this change may not represent a trend, but merely normal variation. This may bring the current authors’ estimated denominator into question. Still, the readers must accept that low rates are affected more by changes in numerator data than denominator data.
There are also reasons why the current study may have underestimated the numerator of anesthesia-related deaths in 1999 in comparison with previous studies. Although the current study used an elaborate peer review process to assure that cases in which the death certificate suggested potential for anesthetic involvement were analyzed appropriately, the initial screening process did not involve an anesthesiologist. The flowchart in figure 1 should really show a medical certifier (examiner) using a medical record, and possible autopsy results, to assign International Classification of Diseases, Ninth Revision (ICD-9) codes as the initial screening step.2Handwritten anesthesia records are notoriously inaccurate, and ICD-9 codes certainly do not allow description of all factors contributing to an anesthesia-related death. It is possible that the medical examiners were unable to capture the more subtle anesthetic contributions to perioperative deaths and these were lost to the subsequent peer review process of the investigators. This is again rationalized by the investigators, who chose to review a sample of 500 hospital deaths in which no preselected ICD-9 codes had implicated anesthetic management as a contributor to the death. In this sample, they found no anesthesia-related deaths but reported that the upper limit of the 95% confidence interval could have produced as much as a 6.7% error. More concerning is the exclusion criteria applied during the peer review process. The investigators chose to exclude cases in which the medical history “explained why death occurred.” The authors admit that these cases included very sick patients undergoing high-risk surgery. In fact, these cases are the most likely to involve human error by an anesthesiologist.11Eliminating the sickest patients from studies of anesthesia-related morbidity and mortality is a relatively common practice, but it negates comparison to studies in which these patients are included.12The authors dismiss this weakness by implying that their study is not “devoted to human error.” The authors also state that because the rate of American Society of Anesthesiologists physical status III and IV patients has increased severalfold in the more recent study, to conclude that anesthesia-related mortality has declined overall, in comparison to the study by Tiret et al. , seems “sound.” In fact, a higher proportion of these sicker patients might have led to a higher proportion of patients being excluded from the current study, even if the same methodologies had been used.
Differing methodologies create different operational definitions of anesthesia-related death, each with their own unique limitations, and readers must be acutely aware of this when evaluating comparisons. For example, Lienhart et al. state that the “rate of deaths totally related to anesthesia was close to other published values” and reference a study by Lagasse. In fact, Lagasse reported, in the referenced study, that no deaths were considered to be due solely to anesthetic management.11The current authors seem to be comparing their estimated rate of deaths partially and totally related to anesthesia, which excludes deaths where anesthetic care could have played a minor role, to the Lagasse definition that included all deaths where even minor errors by the anesthesia providers were judged to have contributed. Similarly, Lienhart et al. compare their findings to those of Tiret et al. , who did not employ the same sampling methods or exclusion criteria. This does not detract from the current study’s unique methodology; it merely brings the comparisons to other studies into question. Although space limitations will not allow a lengthy discussion, suffice it to say that the limitations of the current study methodology are compounded by the limitations faced by Clergue et al. and Tiret et al. when making comparisons.
The readers should appreciate the adversity that outcome researchers must deal with to bring us answers to difficult questions. Lack of standardized definitions and methodologies, inadequate risk adjustment models, and a hesitancy to share data are some of the frustrations facing outcomes researchers in the field of anesthesiology. We must applaud the work of Cheney, Lienhart, and their colleagues who continue to push forward with the best that anesthesiology has to offer. This work brings us closer to the truth. If I may again paraphrase Shakespeare’s Hamlet, we are fortunate that these investigators believe it is “nobler in the mind to suffer the slings and arrows of outrageous fortune than take arms against a sea of troubles, and by opposing end them.”
Department of Anesthesiology, Albert Einstein College of Medicine and Montefiore Medical Center, Bronx, New York. boblagasse@yahoo.com
