In Reply:—
We thank Dr. Raghunathan et al. for their letter regarding their disappointment that we did not publish a cautionary editorial regarding the reports by Wilder et al. 1and Kalkman et al. 2in the April issue of Anesthesiology. These clinical articles, which were published with laboratory work presented at the Anesthesiology/Foundation for Anesthesia Education and Research session at the 2008 Annual Meeting of the American Society of Anesthesiologists, were accompanied by an editorial by Drs. Patel and Sun,3thought leaders in research regarding the mechanisms and clinical relevance of neurodevelopment after exposure to anesthetics. Regarding the clinical article, they concluded in their editorial, “Although two retrospective studies herein suggest that a correlation between anesthetic exposure early in life is associated with learning and behavioral abnormalities later in life, the data cannot be considered to be evidence of the existence of anesthetic neurotoxicity in humans. The absence of rigorously conducted prospective randomized trials precludes recommendations on clinical practice.”3In our opinion, this statement expresses appropriate and adequate express caution regarding the application of these data to clinical practice.
We would also like to comment regarding the statistical analysis and presentation in the article by Kalkman et al. ,2as questioned by Dr. Raghunathan et al. Their study focused on effect sizes and not on statistical significance judgments. This was a prudent choice because of the pilot nature and goals of the effort. This clear focus on effect sizes is made abundantly clear by the fact that in the article by Kalkman et al. 2there is not a single P value reported. Instead, Kalkman et al. 2referenced the size of the observed effects throughout. For a properly powered study, making a claim about an effect that is not statistically significant is, indeed, anathema. However, in this clearly defined pilot study, reminding a reader than an observed effect size did not reach statistical significance is actually a responsible practice. The uncovered effect sizes in a pilot study are estimates of their population values, but as Kalkman et al. 2overtly stated, these estimates are in the context of very wide confidence intervals.
We strongly believe that there is a place for small n research in Anesthesiology. Small n research is tricky to report. We have a sophisticated community of researchers (mostly bench scientists) who successfully add to our knowledge base while using studies that are not optimally powered. Again, this reinforces the importance of clear effect size reporting (as in the two mentioned studies), a priori power analyses to overtly report assumptions, and exact P value reporting to arm a reader with enough information to properly interpret experimental effects.
Regarding their statement on post hoc power analyses, Raghunathan et al. are wise to be concerned about power calculations that are based on observed P values. We agree with this sentiment, articulately voiced by Hoenig and Heisey,4and for that reason actively discourage such power calculations. The provided power calculation, though, was clearly presented as the primary aim of the study, and posits that the observed risks are the population values, and to reject a null hypothesis of no added risk (under a traditional set of inference assumptions), a future prospective study would need to study 2,268 children (thus making it similar to power analyses conducted throughout the research world; this one is simply in print). There is a difference between stating “These differences would be statistically significant with n patients”versus “If these differences are population values, we need n patients to reject a null hypothesis in our next study.” In that regard, Kalkman et al. 2have succeeded in providing a context for interpreting their study.
*Statistical Editor, Anesthesiology, Wake Forest University School of Medicine, Winston-Salem, North Carolina.thoule@wfubmc.edu. †Editor-In-Chief, Anesthesiology, Wake Forest University School of Medicine.
