Effect of Using a Safety Checklist on Patient Complications after Surgery: A Systematic Review and Meta-analysis

OVER the last decade, checklists have become commonplace in healthcare practice as a strategy to improve patient safety. Surgical checklists emphasize several salient components of patient safety: safe anesthesia and airway function, correct surgical site/side, infection prevention, and effective teamwork.^1,2 * The intent of a checklist as a safety tool is to standardize, and make more predictable, team performance across a diverse range of individuals, situations, and clinical environments.²  The fervent introduction of checklists such as the World Health Organization’s (WHO) Surgical Safety Checklist (SSC) has heralded a legislative mandate for the implementation of checklists in operating rooms (ORs) in over 122 countries.^† Thus, checklists have become synonymous as best practice in high-risk areas such as surgery.^2–4 

Checklists hold the promise of reducing catastrophic errors such as wrong site/wrong patient surgery,⁵  improving interprofessional communications,^6,7  enhancing work satisfaction,^8,9  and flattening the hierarchy that often characterizes the culture of surgical teams.¹⁰  A recent systematic review on the impacts and implementation of checklists suggests that checklist use was associated with increased detection of potential safety hazards.⁴  However, the results of many of the studies included reflect patterns of practice on a local or regional level rather than amassed on a wider scale. Thus, evaluating the impact of new clinical practice initiatives on the outcomes of care is problematic in the absence of accurate, large-scale patient outcome data.^1,8,11  Further examination to evaluate to what extent checklists improve clinical outcomes is needed. To this end, we conducted a systematic review and meta-analysis of available studies that tested the effects of surgical safety checklists on complication rates in surgical patients.

The study protocol was informed by guidelines developed by groups such as the Cochrane Collaboration¹²  and the Institute of Health Improvement. A systematic literature search was undertaken using MEDLINE, CINAHL (EBECOhost source), Proquest (Nursing and Allied Health Source), and the Cochrane Library. Publications dated from January 2000 to May 2013 were included. MeSH search terms and their combination included “randomized controlled trial,” “checklist,” “mortality,” “surgery,” “morbidity,” and “intervention” and “complication.” Specific database functions such as “apply related words” and “explosion” were used to maximize the search. Reference lists of retrieved articles were further screened for additional publications. Studies meeting the following criteria were included: Design—randomized controlled trial (RCT), prospective and retrospective cohort, quasi-experimental and interrupted time series; Population—patients undergoing elective or emergency surgical procedures; Intervention—a single OR-specific surgical checklist; Comparator—control group where a surgical checklist was not used; Outcome—postoperative complications, however defined by the primary study authors. We excluded articles if they were (1) not written in English and (2) used a checklist that was not perioperative-specific or encompassed other clinical settings (e.g., SURgical Patient Safety System).^‡

Study-specific descriptive information included country, study design, type of patients, surgical procedures, specific nature of checklist intervention, and quantitative patient outcomes. Patient morbidity and mortality rates were extracted. Data extraction was independently performed by two authors, and discrepancies were adjudicated by a third review author. Authors of primary studies were contacted where necessary (Personal Written Communication: Mehrdad Askarian, M.D., M.P.H., Department of Community Medicine, Shiraz University, Shiraz, Iran, February 27, 2013; Thomas Weiser, M.D., M.P.H., Department of Surgery, General Hospital, Boston, Massachusetts, February 29, 2013) to obtain additional information on published data.

A postoperative complication was broadly defined as “an undesirable, unintended event and would not have occurred had the operation gone as could be reasonably hoped.”¹³  In this review, definitions of the primary outcome, namely, “any major complication” were informed by the American College of Surgeons National Quality Improvement Program and/or the Clavien Classification system¹⁴  or however defined by the primary study authors. Secondary outcomes included individual complications such as mortality, surgical site infection (SSI), pneumonia, wrong-site surgery, unplanned return to OR, and blood loss more than 500 ml (i.e., preoperative blood loss of 500 ml more than expected for a given case)¹⁵  or however defined by the primary study authors. A surgical checklist was defined as a cognitive tool for delineating team tasks according to preinduction, before skin incision, and before leaving the OR and used to prompt interdisciplinary verification of surgical and patient details to increase teamwork.^2,4,9  Standard care was defined as the usual practices undertaken by the interdisciplinary surgical teams before checklist implementation or however defined by primary study authors.

Systematic review and meta-analysis of studies other than RCTs is challenging because combining observational studies of heterogeneous quality may be highly biased.^12,13,16  Therefore, we required a quality assessment tool that allowed us to compare observational studies with wider heterogeneity. For this reason, quality was assessed using a modified version of the previously validated Methodological Index for Nonrandomized Studies.¹⁶  The original version of the Methodological Index for Nonrandomized Studies is a 12-item index. Each item is scored as 0 = not reported, 1 = reported but inadequate, or 2 = reported and adequate, with the combined score ranging from 0 to 24 for observational studies.¹⁶  For the purposes of this review, we made modifications to the index. Specifically, items 7 and 10¹⁶  were not considered as these items are better suited to evaluate studies using contemporaneous groups, that is, control and intervention groups that are managed during the same time period (no historical controls). Therefore, in this study, the highest possible score was 20. Two review authors (B.M.G. and M.J.) independently assessed study quality, and the proportion of agreement was measured using the intraclass correlation coefficient. A coefficient of 0.70 or greater was considered adequate.¹⁷ 

The software package Comprehensive Meta-Analysis Version 2.0 (Biostat Solutions Inc., Englewood, NY) was used to estimate the overall pooled effect size with fixed or random effects models for each patient outcome. Statistical heterogeneity among the studies was assessed using I-squared (I²) and Cochrane Q statistic (chi-square test) for heterogeneity. The significance level of the chi-square test was set at P value less than 0.05. The degree of statistical heterogeneity was determined if the I² was 50% or greater. In cases of statistical heterogeneity, only random effects models were used. Results of the meta-analysis were presented using risk ratio and corresponding 95% CI and the z test for pooled effect size estimates. For each outcome, the number needed to treat (NNT) and absolute risk reduction (ARR) were calculated from the pooled risk ratio and the event rates to further place risk estimates into context. Forrest plots were used for graphical display of results.

To assess the robustness of the results, sensitivity analyses were performed.^12,18  We estimated patient outcome measures after excluding studies with lower methodological quality to check whether the results had changed. If the results did not change significantly after excluding low-quality studies, then they were considered to be robust. If the results had changed or the conclusions differed, then they had low stability. We assessed potential publication bias by performing informal visual inspection of funnel plot symmetry¹²  based on the primary outcome, that is, any major complication.

Figure 1 summarizes the number of observational studies identified in the systematic review and meta-analysis according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.¹⁹  Our search yielded 207 articles. Of these, seven (3.4%) studies^20–26  with a total of 37,339 patients met eligibility criteria. The included studies were published between 2009 and 2012. The characteristics of each study are summarized in table 1. All studies were observational in nature, and most employed a prospective cohort design using historical controls, with the exception of the study by van Klei et al.²⁴  who used a retrospective cohort design. Most studies (six of seven, 86%) used the WHO Surgical Safety Checklist²⁷ ; however, the study by Bliss et al.²¹  used the Association of peri-Operative Registered Nurses Comprehensive Surgical Checklist,^§ which includes items based on the WHO Checklist^* and the Joint Commission Universal Protocol. The studies by Haynes et al.²²  and Weiser et al.,²⁵  although based on the same multinational study, focused on different subanalyses of patients. Weiser et al.²⁵  examined checklist use in urgent surgeries, whereas Haynes et al.²²  included all patient groups in their analyses. Consequently, where both of these studies examined the same patient outcome, we analyzed those outcomes (i.e., any major complication, mortality, SSI) based on the results reported by Haynes et al.²²  The study by Weiser et al.²⁵  was included in the analysis of only one outcome, expected blood loss more than 500 ml (an outcome not measured by Haynes et al.²² ).

The quality assessments of the studies by two independent reviewers were in good agreement (intraclass correlation coefficient, 0.80; CI, 0.27 to 0.96; P = 0.008). Out of a possible score of 20 on the modified 10-item Methodological Index for Nonrandomized Studies tool,¹⁶  the mean score for the seven included studies was 12.6 (±1.3; range, 11 to 15). Four studies (57%) achieved a score of 13/20 or greater.

Five published studies^20–23,26  with 9,747 patients compared checklist use relative to this outcome that combined single complications that included up to 18 individual complications (pulmonary embolism, deep vein thrombosis, SSI, pneumonia, unplanned return to OR, blood loss, death, wound dehiscence, cerebrovascular accident, myocardial infarction, vascular graft failure, coma, sepsis, unplanned intubation, systematic inflammatory response syndrome, septic shock, cardiac arrest, and acute renal failure). As there was no significant heterogeneity among studies (chi-square = 4.37, df = 4, P = 0.357, I² = 8.6%), we used the fixed effect model (fig. 2). Pooling of individual studies indicated that 381 of 4,922 (7.7%) patients in the intervention group and 592 of 4,835 (12.2%) patients in the control group developed any complication. The ARR was 3.7% (95% CI, 2.6 to 4.8%), and the NNT was 27 (95% CI, 21 to 38). That is, one less major complication may be prevented for every 27 patients when the checklist is used.

Four published studies^22–24,26  with 34,642 patients compared checklist use in relation to this outcome. Pooled analysis revealed that 356 of 15,840 (2.3%) patients in the intervention group and 520 of 18,807 (2.8%) patients in the control group died. As there was significant heterogeneity (chi-square = 5.58, df = 3, P = 0.039, I² = 46.3%) among studies, a random effects model was used (fig. 3). The ARR was 0.44% (95% CI, 0.12 to 0.76%), and the NNT was 229 (95% CI, 131 to 867), suggesting that one less death may be prevented for every 229 patients when the checklist is used. However, in three of four included studies, the CIs cross the line of no difference, which suggests that the use of a checklist is not associated with a reduction in mortality.

Five published studies^20–23,26  with 9,747 patients assessed this outcome. As there was significant heterogeneity between studies (chi-square = 6.50, df = 4, P = 0.0001, I² = 38.4%), a random effects model was used (fig. 4). Pooling of individual studies showed that 188 of 4,912 (3.8%) patients in the intervention group and 344 of 4,835 (7.1%) patients in the control group developed a SSI. The ARR was 2.9% (95% CI, 2.1 to 3.8%), and the NNT was 34 (95% CI, 26 to 47). That is, one less SSI may be prevented for every 34 patients when the checklist is used.

Four published studies^20–23  with 9,266 patients evaluated this outcome. As there was no significant heterogeneity between studies (chi-square = 11.4, df = 4, P = 0.022, I² = 64.3%; fig. 5), a fixed effects model was used. Pooled analysis revealed that 69 of 4,663 (1.5%) patients in the intervention group and 70 of 4,603 (1.5%) patients in the control group developed pneumonia. The ARR was 0.04% (95% CI, −0.4 to 5.0%), and the NNT was 2,512. In this instance, the number of patients required in the checklist group is greater than 190, compared with that in the standard care group, which exceeds 224 (95% confidence). In these four studies, the CIs cross the line of no difference, suggesting that the use of a checklist is not associated with a reduction in pneumonia.

Two published studies^21,25  with 2,069 patients assessed this outcome. As there was no significant heterogeneity between studies (chi-square = 9.38, df = 1, P =, I² = 0.0%; fig. 6), a fixed effects model was used. Pooling of individual studies showed that 122 of 981 (12.4%) patients in the intervention group and 182 of 1,088 (16.7%) patients in the control group had experienced bleeding requiring 4 units of blood or greater within 72 h after surgery. The ARR was 3.8% (95% CI, 0.6 to 5.9%), and the NNT was 33 (95% CI, 17 to 168). Therefore, one less patient may have blood loss more than 500 ml for every 31 patients when the checklist is used.

Two published studies^22,23  with 8,653 patients assessed this outcome. As there was no significant heterogeneity between studies (chi-square = 0.193, df = 1, P = 0.661, I² = 0.0%; fig. 7), we used a fixed effects model. Pooling of individual studies showed that 76 of 4,440 (1.7%) patients in the intervention group and 95 of 4,213 (2.3%) patients in the control group had an unplanned return to the OR within 72 h after surgery. The ARR was 0.52% (95% CI, −0.5 to 1.1%), and the NNT was 192, suggesting that unplanned return to OR may be avoided for every 192 patients when the checklist is used. The number of patients required in the checklist group exceeds 91, compared with that in the standard care group, which exceeds 1,825 (95% confidence). In the two included studies for this outcome, the CIs cross the line of no difference, suggesting that the use of a checklist is not associated with a reduction in unplanned return to OR.

The funnel plot (fig. 8) for the primary outcome, any major complication, was relatively symmetrical, suggesting that publication bias was not present. Begg test (z = 0.939, P = 0.347) and Egger test (P = 0.452) did not support publication bias.

We found seven observational studies that examined the effect of using surgical checklists on any major postoperative complication and individual complications (i.e., mortality, SSI, pneumonia, blood loss, and unplanned return to OR). The main finding from this analysis is that checklists appear to be associated with the reduction in the risk of “any major complication,” which was measured as a composite variable. Our results are similar to those of an earlier meta-analysis by Borchard et al.²⁸  that included three studies that examined the outcome “any complication.” Yet, given the observational nature of the research, we are unable to establish causality. In both our and Borchard’s²⁸  reviews, the primary authors measured “any major complication” based on previous definitions of the American College of Surgeons National Quality Improvement Program and/or the Clavien Classification system.¹⁴  That the authors^14–17,20  combined all of the complications they measured in the primary studies as a single outcome (i.e., a composite measure) more likely demonstrated significant before-and-after changes in this outcome.

Notably, there were significant risk reductions for the outcomes of SSI^20–23,26  and expected blood loss.^12,16  These results may reflect the specific items that target these aspects of intraoperative care. For example, item 7 (sign-in) of the WHO SSC²⁷  addresses the risk of blood loss, whereas items 12 of 13 (time out) require confirmation of equipment availability/instrument sterility and antibiotic prophylaxis. Hospital-acquired infections are considered preventable—and therefore, human error may contribute to SSI.⁴  Using a checklist in surgery is considered effective in assisting individuals and teams to remember key information or actions that would otherwise be overlooked, thereby reducing the potential for human error.^24,29 

In our analyses, the nonsignificant results relative to individual complications of pneumonia,^20–23  patient mortality, and unplanned return to OR^22,23  may be attributed to several factors: First, the numbers of patients who developed pneumonia or who died were relatively lower for each of these individual complications, representing less than 3% patients who received the checklist intervention. Given the low percentages of patients who developed pneumonia or died, a significant reduction would have required a much larger number of patients, which is unlikely given the potential effect size (i.e., low numbers, small effect size). Second, all studies included herein individually reported superiority of checklist use over standard practice—but “standard practice” was never defined in any of these studies. Written correspondence with study authors (Mehrdad Askarian, M.D., M.P.H., Department of Community Medicine, Shiraz University, Shiraz, Iran, February 27, 2013; Thomas Weiser, M.D., M.P.H., Department of Surgery, General Hospital, Boston, Massachusetts, United States, February 29, 2013) indicated that “timeout” was used in some hospitals, but in most cases, there was no routine or standardized practice before checklist implementation. Thus, there was no stable baseline from which to draw comparisons. It may be that standard practice is naturally improving because of contemporary trends and organizational initiatives around patient safety.^2,3  There is little doubt that the culture of surgery is changing, and the importance of human factors is increasingly being recognized.³⁰  Finally, there is a danger in treating the checklist as a “tick box” exercise, rather than emphasizing the need for clinicians to engage in the process.^,4,11,29 

We acknowledge some limitations. Although meta-analysis is powerful—it is also controversial because small violations in meeting critical assumptions can give misleading results.³¹  Critique of observational studies is not an exact science, and the quality assessment of these studies is essentially subjective. However, we used a tool that has established validity¹⁶  and that is appropriate and has demonstrated an acceptable level of agreement in the appraisal of the included studies. Our search methods were exhaustive and robust, but it is possible that we may have missed other important studies. An additional concern was the modest number of studies available for inclusion.

A significant methodological limitation to the review studies was the lack of a control and a comparison group—none were RCTs. All included studies used a cohort design; thus, selection and reporting biases cannot be eliminated. Before-and-after studies are problematic in estimating the efficacy of an intervention.³¹  In some of the reviewed studies,^20,26  insufficient sample size, selection bias, and surgical team and institutional factors were also potential confounders. Moreover, there is the likely disparity in patient characteristics due to nonrandomization, which may have potentially increased variability in observed heterogeneity effects among studies.¹²  Where appropriate we used random effects models to account for the presence of other unpublished studies not included herein. Patient factors that may have confounded the results of the primary studies include comorbidities, procedural complexity, and whether the surgery was elective or emergency—all of which impact on postoperative outcomes. Patients needing emergency surgery are at greater risk of developing postoperative complications.³  Further, organizational factors, differences in hospital sites, and temporal effects (i.e., organizational initiatives, cycles of surgical training) may have been responsible for reduced postoperative complications, rather than intervention effects. Finally, team culture and communication practices may have led to differences in checklist implementation and the level of adherence.

In all of the reviewed studies, the WHO SSC or adapted version was used,^§ and there was relative uniformity in the number of checklist items, with either 19^20,22,23,26  or 22²⁴  items. Yet, across studies there was disparity in adherence in relation to (1) the number of occasions the checklist was used (per patient) after its introduction, and (2) the number of checklist items that were fully completed. For instance, of the review studies that examined checklist adherence post implementation, overall rates varied from 18.6²⁶  to 96.9%.²³  The earlier systematic review by Borchard et al.²⁸  examined checklist implementation as their major outcome. In that study, the mean overall compliance rate was 75%, and the compliance for the “timeout” component of the checklist was much higher, with an average of 91%. In our review, variability in compliance rates may have confounded the results of these studies. Follow-up periods in the review studies were up to 30 postoperative days or discharge, whichever came first.^20–24,26  The brief duration of the follow-up period may have only captured the initial flush of enthusiasm following the introduction of a new project, biasing the results. As data collection was limited to this 30-day period, late complications may have been underestimated. However, it is also plausible that there were more events at the initiation of the study than during the entire study period.

Despite these limitations, the results of the current meta-analyses are meaningful because of the large number (≥40,000) and heterogeneity of patients analyzed. While we are cautionary in our judgment of external validity, the review studies included were conducted with patients drawn from global populations. As such our findings suggest that the WHO SSC has wide application.

Randomized controlled trial methods are likely to have increased internal validity but they cannot answer all important questions for a given intervention.³¹  The “sterility” offered by RCTs is likely untenable in assessing the effectiveness of checklist use in real-world clinical environments, where pragmatic approaches to practice improvements are often required.^29,32  Checklists facilitate an exchange or clarification of information among team members to avert sentinel events in surgery.^1,3,5  Consequently, checklist use supports a safety culture,³³  and it is viewed as best practice.^2,4,29  As such, it would be inappropriate to withhold the use of checklists in clinical practice rendering RCT methods unfeasible. Although statistical significance is important when considering the efficacy of an intervention, the intervention implemented must be practical, sustainable, cost-effective, and clinically important. In some instances, there were minor modifications made to the checklists to ensure that they were contextually responsive to the local hospitals where the review studies were conducted.

The strengths of this meta-analysis include the rigor used to identify and evaluate available studies. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines¹⁹  were adhered to, and we utilized an appropriate, validated tool to assess study quality.¹⁶  The reviewed studies incorporated 11 countries (i.e., Canada, India, Jordan, New Zealand, Philippines, Tanzania, England, the United States, Iran, The Netherlands, and Liberia) and varied in patient case-mix; consequently, our results may have global applicability. There was reasonable homogeneity in the outcome measures and the intervention evaluated. Despite some methodological limitations and the caveats around the interpretation of findings, the meta-analyses presented herein represent the best available current evidence.

These results suggest that checklists are associated with a reduction in overall complications in surgical patients. Surgical safety checklists provide a means to safeguard patients and minimize risk through increased team cohesion and coordination. Importantly checklists should be used to augment, and not replace, other initiatives that contribute to a safety culture.

The authors thank the financial assistance of the Australian Research Council, Early Career Discovery Fellowship Scheme, Canberra, ACT, Australia.

The authors declare no competing interests.