To the Editor:—
Recently, Wu et al. 1published a systematic review comparing the treatment of postoperative pain by intravenous patient-controlled analgesia (PCA) versus epidural analgesia—either patient-controlled epidural analgesia (PCEA) or continuous infusion epidural analgesia (CEI), usually with a local anesthetic–opioid mixture. The literature search discovered 50 articles. Using visual analog scale measurements of pain as the primary outcome, the meta-analysis used a fixed effect analysis of variance (ANOVA) to compare the treatment groups—PCA versus PCEA and CEI. The average visual analog scale values for all data (mean ± SD) were 3.2 ± 1.6 versus 2.1 ± 1.3; this difference favoring PCEA and CEI was declared statistically significant at P < 0.001.
The authors specially limited their literature search to the English language, but some non-English research reports otherwise qualifying for inclusion were incidentally identified. The authors report that inclusion of five such studies would not have changed the meta-analytic results. Current recommendations for performance of systematic reviews specifically discourage exclusion—without good reason—of publications in languages other than English.2Empirical research has shown that under some circumstances, trials not published in English demonstrated statistically significant results less often3; other research on language bias in systematic reviews concluded that exclusion of non-English language trials had shown unpredictable consequences on the summary statistics of a meta-analysis.4The authors report no reasons for limiting their literature search to the English language. They should reconsider their exclusion of trials in other languages.
The numbers of patients reported in the 50 trials were 1,625 (PCEA and CEI) and 1,583 (PCA), whereas the numbers of observations included in the ANOVA of overall data were 7,744 (PCEA and CEI) and 7,666 (PCA). This difference in the number of observations versus the number of patients is the consequence of including visual analog scale scores obtained at multiple times in each patient. These multiple observations in each patient are not considered independent variables. The inclusion of multiple observations has been denoted as a “unit of analysis” error.5The likely consequence of a unit of analysis error is a spurious precision in the calculation of SD. The authors should restrict their meta-analysis to the observations obtained independently; this can be done simply by dropping all analyses using “overall” and “all” data in table 2.1
The authors chose ANOVA as the statistical method for comparing PCEA and CEI versus PCA. ANOVA is a method for hypothesis testing. The main emphasis in a systematic review is the effect measure. The effect measure is a single number that contrasts the treatments; statistically, this is parameter estimation, not hypothesis testing. Because the visual analog scale score may be considered approximately a continuous variable, the relevant effect measure is the difference in mean values—also known as the weighted mean difference.6By contrast, the ANOVA results presented show the mean values for each treatment group. The meta-analysis should be redone using the weighted mean difference effect measure. The calculation of a summary effect measure is accompanied by statistical tests of heterogeneity. The identification of heterogeneity may encourage the use of a different statistical model, the random effects model. The fixed effect ANOVA reported in this study does not allow identification of heterogeneity among the 50 studies. Finally, with the use of an effect measure, the results should be presented in a Forest plot.7This allows the inspection of the effect measure for each individual trial as well as the summary value of the effect measure; heterogeneity may become more easily recognizable. The Forest plot also makes clear the statistical significance of both the individual studies and the summary effect measure. For a weighted mean difference effect measure, if the lower and upper boundaries of the 95% confidence interval do not bound the zero value, the estimate of the effect measure is declared to be statistically significant.
Although this systematic review was not created for the Cochrane Collaboration, the methods for systematic reviews and meta-analysis presented in their publications provide a rigorous guide for this research.8The authors should reconsider several of their experimental methods that may have produced biased inferences. It is possible and to be desired that the conclusions of a revised systematic review will be unchanged from the current version.
University of Utah, Salt Lake City, Utah. n.l.pace@utah.edu
