How to Stop the Unknowing Citation of Retracted Papers

Imagine you were involved in a court case. Your attorney submits a clear and convincing argument that the lawsuit should be dismissed. But it turns out your lawyer has cited precedents that have been overturned, and the judge rules against you. Fortunately, the legal system has a way to prevent such mishaps: authors of legal documents can use online databases to automatically check the veracity and status of a case as an authoritative citation, and whether it has been criticized or overruled. That process is known as “Shepardizing” due to its origins in the printed Shepard’s Citations, an index of court cases and decisions.

But law is not medicine, and overturned precedents—in this case, retracted papers—keep being cited as if they were still reliable. And just as citing overturned precedents can lead to bad legal outcomes, citing retracted papers can lead to a deeply flawed base of evidence for clinical decision-making. In this issue, a study by De Cassai et al.¹  demonstrates that anesthesiology is not immune to the citation of retracted papers.

De Cassai et al. identified 478 articles in the field of anesthesia and intensive care that had been retracted by August 16, 2021. They found that roughly half—46%—of the retracted papers had been cited at least once. In about three quarters of relevant cases, the citations occurred after the paper in question had been retracted. While we and others who study retractions have long speculated about the reasons researchers continue to cite retracted papers, De Cassai et al. sought answers. They sent surveys to the 1,297 corresponding authors of those citing articles; 417 returned the survey. The overwhelming majority of the surveyed authors—89%—reported having been unaware of the retracted status of any of their references, which had been checked by most authors in some manner before submission of their manuscript. Authors attributed gaps in knowledge to a failure of journals to adequately note the change in status for the papers and the use of hard copies. Almost all the responding authors indicated that nothing about the status of their references had been mentioned by either reviewers or the editors.

This problem is not unique to anesthesiology.^2,3  While we have no solid comparative data that would allow us to tell whether the problem is better or worse in anesthesiology, it is clear that the issue permeates the entire biomedical literature. Nor is the problem small; more than 4,000 (about 30%) of the total citations received by the “top 10 most highly cited retracted papers”⁴  occurred post-retraction.

Why is citing retracted literature so problematic? Perhaps that is obvious in cases of misconduct or fraud. Researchers would hardly wish to build the basis of their work on a house of cards. The same would be true of work retracted for honest error, a courageous step that should be celebrated but is responsible for a minority of retractions. Regardless of the reason for retraction, however, citing retracted literature causes real-world harm.

Use of fabricated data has obvious risks, as demonstrated by the useless chemotherapy protocols derived from fraudulent data created by Duke University’s Anil Potti.⁵  Meta-analyses and systematic reviews are particularly vulnerable to such hazards. For example, a meta-analysis looking at mortality associated with the use of hydroxyethyl starch for volume expansion found the practice safe (risk ratio, 0.91). When seven questionable studies by Joachim Boldt were excluded from the analysis, however, hydroxyethyl starch was found to be “significantly associated with increased death ([risk ratio], 1.09).”⁶ 

Falsified data, however, is not the only reason for retraction, and with each reason comes a different problem for downstream literature. Some of the most common reasons are issues with data, particularly images, or in the interpretation of results. Others are failure to obtain proper institutional approval, lack of informed consent, or even threats of litigation about alleged defamation. Awareness of paper mills, and their misconduct, has led to a spate of retractions recently.

Plagiarism is another typical reason. When plagiarized or duplicated articles are included in a systematic review or meta-analyses, the practice can skew the results. Simply put, citation of retracted literature—except in papers that refer to the retracted status of a paper—is detrimental to the literature for a plethora of reasons, virtually none of them good.

How to prevent the infiltration of the citation record with retracted papers? To be sure, authors and journal editors bear responsibility for vetting references in manuscripts. But as De Cassai et al. learned, more than two thirds of respondents in their survey stated that they checked their references before submission. And yet most said they were unaware of the retracted status of any of their references. These findings seem contradictory, until one considers just how inconsistent publishers are at notifying databases such as PubMed and Crossref about retractions from their journals.⁷  Authors could check those databases, which is considered good practice, but still not learn that a paper had been retracted.

For example, a search for “retracted publication” in PubMed results in about 12,000 entries at the time of this writing. The Retraction Watch Database⁸  has almost 35,000. What accounts for this large discrepancy? Some of it is due to the fact that the Retraction Watch Database includes all available journals regardless of subject, and PubMed only includes journals that relate to biomedicine, and only a subset of those subject to quality criteria.

The majority of the discrepancy, however, is due to the fact that many publishers fail to transmit metadata about retractions in a timely or accurate way to PubMed, which does not gather retraction data itself and is entirely reliant on that process. Bakker and Riegelman demonstrated this to be the case for the mental health literature,⁷  and we have seen that it is true for many other fields. In other words, a check of PubMed will miss many, if not most, retractions.

We therefore urge authors to use bibliographic management software that includes a consistently updated and dynamic database of retractions. We also encourage publishers to implement cross-checks of references against such a database before publication, and to automatically notify corresponding authors of published articles that reference a retracted article. Publishers should also provide support for editors investigating complaints concerning the integrity of the references in a publication.

Preventing the downstream pollution of the scientific literature will require the combined effort of those up and down the shore as well as those already in the water. With that in mind, it is worth recalling a recent editorial in this journal noting that “peer review certainly was and remains an essential initial check and quality control that has weeded out, or corrected before publication, innumerable reports of research of insufficient quality or veracity that otherwise would have been published and thereby become publicly accessible.”⁹  Reviewers have a role in preventing pollution to begin with by performing careful and rigorous reviews. But it is also worth noting that the vast majority of papers will eventually be published somewhere despite their flaws, and that peer review has missed many papers that should be retracted but have not yet been.¹⁰ 

Science must double down on fraud and error detection efforts if it hopes to maintain or improve the public trust.