Quality and Safety Indicators in Anesthesia: A Systematic Review

We searched PubMed (1970–December 2005), EMBASE (1970–December 2005), CINAHL (1980–December 2005), and the Cochrane library–DARE (1990–December 2005) for all English-language articles relating to the development or use of clinical indicators. We also looked for clinical indicator programs available on governmental, private, and professional organizations’ publications and Web sites, if organizations were established at national level and had information in English. Some examples in the United States include the American Medical Association’s Physician Consortium for Performance Improvement Program, the National Committee for Quality Assurances Indicators Program, the Centers for Medicare and Medicaid Services, the Surgical Care Improvement Project, the American Society of Anesthesiologists’ Committee on Performance and Outcome program, and the Maryland Hospital Association Quality Indicator Program. In Australia, these include the Australian Council on Healthcare Standards–Care Evaluation Program, the Australian Commission on Safety and Quality in Healthcare Initiative, and the National Hospital Quality Management Program; in Canada, the Canadian Health Department indicator program; in New Zealand, the Health System Performance Indicators from the National Department of Health; and in the United Kingdom, the National Health Service Indicator Program and the International Quality Indicator Project. For the rest of the world, these include the World Health Organization and the Organization for Economic Cooperation and Development technical papers and other publications. Finally, expert opinion was sought for unpublished indicators developed by quality initiatives and professional organizations. The detailed search strategy is available in Supplemental Digital Content 1 (see table, https://links.lww.com/A1006).

For the purpose of this systematic review, a clinical indicator was defined as an explicit measure (defined by the developer) of some aspect of patient clinical care used to judge a particular clinical situation and indicate whether the care delivered was appropriate.21We included all publications (peer-reviewed and non–peer-reviewed) and Web sites describing the development and validation of indicators. To be included in the review, clinical indicators had to be also directly related to anesthesia. Indicators relating only to intensive care or surgical care were excluded. We also excluded indicators used for medicoeconomic purposes. Publications reporting raw figures of adverse events, incidents, and complications were also excluded because these did not comply with the definition of indicators. We did not consider studies on patient satisfaction because the metrics developed for its measurement rely on questionnaires and not on clinical indicators.

For published articles, inclusion and exclusion criteria were extracted either from the abstract or from the full article. For indicator programs, the information provided by indicator summary tables was used. If incomplete, the rest of the Web site or manual of use was searched. Selected articles and indicator programs were then carefully examined, and information about the indicators was extracted. The overall process was performed by one of the authors and verified by the second author.

We used a standardized coding template#adapted to a unifying taxonomy system for classification of clinical indicators.20–22**The template included the indicator’s abbreviated/standard name, the developer’s definition, the level of specificity for anesthetic care, the area of care (i.e. , structure, process, outcome), and the measured dimension of quality (i.e. , continuity, effectiveness, safety). The template also integrated the type of indicator (descriptive, prescriptive, proscriptive) and the recommended method to assess the presence of an anesthesia-related quality issue.

We classified and incorporated indicators into tables according to their original developer or promoter specifications (hospital, accreditation body, quality initiatives, professional organizations, other). Therefore, if a clinical indicator had been developed through collaborative work of different organizations, only the primary promoters and not all of the organizations involved were considered.

If there was more than one primary promoter for the same indicator, we chose to select the one providing the most information and excluded the others. This was, for example, the case for indicators developed by the Health Administration program of the US Department of Veterans Affairs, such as acute myocardial infarct 2 days after anesthesia or cardiac arrest rate of patients receiving anesthesia, which were similar to clinical indicators developed by the American Society of Anesthesiologists.23††

If indicators were closely related, although not similar in their definition, they were individually cited in the first summary table but aggregated in the second and third tables, where indicators were analyzed. This was the case, for example, for the indicators of the Australian Council on Healthcare Standards analyzing preoperative documentation and patient consent and the American Society of Anesthesiologists indicator related to postoperative care unit stay.

When indicators defined different time periods for the same measurement, even if closely related, they were all considered. For example, both death within 48 h of a procedure involving anesthesia and death rate associated with procedures involving anesthesia were selected and described.

The full data abstraction and classification form is available in Supplemental Digital Content 2 (see table, https://links.lww.com/A1007).

Whenever information on specificity for anesthetic care, area, and dimension of quality measured was not provided by the developer, we used standardized definitions and classification procedures. We defined an indicator as specific to anesthetic care if it referred specifically to the practice of anesthesia. If it could also relate to surgical or postoperative ward care, it was defined as general. To classify indicators into the different areas of care, we referred to the model and definitions for quality assessment developed by Donabedian24: the structure, process, and outcome model. This model sees health care as a cyclic transformation mechanism. Patients are inputs  entering a healthcare organization’s structure  to undergo a process  of care through which they will become outcomes/outputs , which will further inform the feedback loop  back to inputs . In our case, indicators of processes  referred to measures assessing the implementation of program activities, and indicators of outcomes  referred to those measuring patient-related end results of anesthesia care. Structure  referred to hospital staff, material, and overall organization.

The classification template also included the dimension of quality measured by indicators. Because there is no single definition of quality and well-known definitions such as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge”25cannot be easily used to discriminate indicators, we chose the approach developed by the Joint Commission in its extensive review of clinical indicators, the National Library of Healthcare indicators.26It describes quality of care through its different attributes rather than through a single definition. It identifies 10 different attributes of quality: appropriateness, availability, continuity, effectiveness, efficacy, efficiency, prevention, respect and caring, safety, and timeliness. Details regarding these attributes’ definitions are available in Supplemental Digital Content 3 (see https://links.lww.com/A1008).

We also classified clinical indicators into three main distinct category types.27Descriptive indicators were those that provided descriptive information on unusual situations of patient care that could reveal, if further investigated, potential defects in the quality of care provided. An unplanned overnight admission of day surgery patients for anesthetic reasons was, for example, considered a descriptive indicator. Prescriptive indicators were defined as indicators representing recommendations or desired targets. Prophylactic antibiotic selection for surgical patients according to current recommendations, for example, was considered a prescriptive indicator. Proscriptive indicators were defined as measures of actions that “should not be performed,” such as, for example, medication error with the wrong medication being given.

To classify developers’ recommended methods for the identification of an anesthesia-related quality issue, we used the five major models described in the literature8,11–17,21,22:

1. Peer review of medical charts and other data sources: Individual cases flagged by the indicator are reviewed by a committee of experts to identify possible quality and safety issues.

2. Hospital internal benchmarking or comparison: The current rate or frequency of the indicator is compared with previous average rates for the indicator, and an abnormally high rate or frequency (usually > 2 SDs) indicates possible quality and safety issues.

3. External benchmarking or comparison with other hospitals’ average rates: The hospital current rate or frequency of the indicator is compared with other hospitals’ average rates, and an abnormally high rate or frequency (usually > 2 SDs) indicates possible quality and safety issues.

4. Noncompliance with accepted and widely recognized standards (± based on scientific evidence): This noncompliance identified by the indicator indicates possible quality and safety issues.

5. Risk adjustment for comorbidities and other factors: Risk-adjusted rates for the indicator higher than the reference population indicate possible quality and safety issues.

Because there were often, for the same indicator, several methods recommended to identify anesthesia-related quality issues, we chose to report all of them in the summary tables. For example, the American Society of Anesthesiologists indicators could be used both as flags to guide a peer review process (method 1)††and as rates plotted on a statistical process control chart for hospital internal benchmarking (method 2).28 

Finally, we extracted and assessed the level of validity of each indicator (the extent to which it measures what it supposed to measure) based on its psychometric properties. This is the approach most widely used by developers and expert committees. It recognizes four different types of validity: face, content, construct, and criterion. Face validity  refers to the extent to which, on the face of it, an indicator seems to be measuring what it is intended to measure. This is usually a subjective judgment based on the review of the measure itself by one or more experts.29,Content validity  is the extent to which an indicator samples all relevant subdimensions of the domain under study. For example, an indicator assessing quality of life needs to integrate all of the different domains composing quality of life: physical functioning, bodily pain, mental health, general health, social functioning, emotional/physical role, and vitality.30An instrument assessing only pain, for example, would not be considered to have content validity as a measure of quality of life. Construct validity  is the demonstration that there is a significant convergence between a new measurement tool and previously validated measures of the same attribute (construct). A good example is the assessment of the convergence between the Glasgow coma score and the cerebral metabolic rate or visual evoked potentials to determine the validity of the score as a measure of brain damage and coma.31 Criterion validity  is the extent to which an indicator agrees with a gold standard of the domain or phenomenon being measured. The later is not used for indicators’ validation because there is no consensus gold standard to measure quality of care.

To assess indicators’ validity, we used documentation provided by the developer, expert committees, or any publication related to the formal validation procedure of indicators. If publications reported only examples of use of the measure, they were not considered to be formal validation studies. When indicators were prescriptive (recommendations of best practices), we assessed whether these practices had a scientifically proven value. We searched the literature for published evidence on the scientific soundness of the proposed recommendations. For example, for the prescriptive indicator of patients with a history of postoperative nausea and vomiting (PONV) to whom a prophylactic antiemetic has been administered, we assessed whether there was any published evidence that prescription of prophylactic antiemetic treatment would benefit patients with a history PONV. If publications were found, we rated the reliability of the evidence they provided using the widely publicized scale developed by the Oxford Centre for Evidence-based Medicine.‡‡

This scale enables studies to be ranked from 1a to 5 according to the validity of their findings. Therefore, a prescriptive indicator based on results from a large randomized controlled trial or systematic review would be rated as 1, whereas a recommendation supported exclusively by expert opinion would be rated as 5. A simplified version of the scale is provided in fig. 1, and a more detailed one can be found in Supplemental Digital Content 4 (see table, https://links.lww.com/A1009). When this was available, we used the level of evidence mentioned by the developer. In all other cases, we performed a literature search to find all publications related to the recommendation and rated prescriptive indicators according to the level of evidence provided by these publications.

The overall classification and evaluation process of indicators was performed by two assessors. Both were medical doctors with health services research and anesthetic training, respectively. Any disagreement between the two reviewers was solved by discussion until a consensus between the two assessors was reached.

We identified 834 articles, of which a total of 22 met all inclusion criteria, particularly regarding indicators’ definition and relevance to the specialty of anesthesia. The search strategy also included private and governmental quality organizations’ Web sites and publications, including technical papers and manuals. We found 37 indicator programs, of which 11 were excluded because they did not relate to the specialty of anesthesia, were irrelevant, or did not correspond to selected definitions. Figure 2provides a detailed description of the selection process of articles and indicator programs. Overall, 22 articles and 26 Web sites, manuals, and technical papers were selected for full data abstraction and assessment, resulting in the identification of 108 different clinical indicators related to the practice of anesthesia.10–16,22,23,26,28,32–52#**§§∥∥##***†††‡‡‡§§§∥∥∥###****††††‡‡‡‡§§§§

An overview of all selected clinical indicators is provided in table 1.

A large number of indicators were closely related to each other, such as total perioperative mortality for all American Society of Anesthesiologists classes, perioperative mortality for American Society of Anesthesiologists classes I–V, or death rate associated with procedures involving anesthesia. Some indicators, such as mortality within 30 days of surgery, although developed by different quality initiatives in different countries (United Kingdom and United States), were identical to each other. These were consequently aggregated in table 2, which describes indicators’ characteristics and validity.

We identified 55 indicators related specifically to the practice of anesthesia and 53 that were also related to the practice of surgery, emergency medicine, or postoperative ward care. Only 26 indicators measured exclusively one dimension of quality of anesthetic care (usually safety and appropriateness). All others assessed several dimensions of quality at the same time. The combination we most often identified was appropriateness (relevance to patient’s clinical need), effectiveness (correct care according to current state of knowledge), and patient safety (absence of avoidable iatrogenic complications). Safety was the most common dimension measured by anesthetic clinical indicators (83%), followed by effectiveness (68%). More than half of the indicators (57%) were measures of outcome, and the remaining ones were process (42%) or structure indicators (1%).

Most outcome measures related to patient safety and effectiveness. To identify possible anesthesia-related issues, the most frequently suggested method was external benchmarking (comparison with other hospitals) to flag abnormal variations that required expert-based assessment to determine causes and improvement strategies. A number of indicators were also considered direct markers of quality and performance to be integrated into report cards (e.g. , death within 30 days of surgery) or pay for performance programs (e.g. , surgical patients for whom administration of a prophylactic antibiotic has been initiated within 1 h before the surgical incision, noncardiac surgical patients who received prophylactic antibiotics and who have an order for discontinuation within 24 h of surgical end time).

All indicators identified in the review had face validity and had been assessed by expert committee. Only 40% had undergone an additional content and/or construct validation process through patient case note review or other related methods (table 2).

Table 3reports the level of scientific soundness of the literature on which prescriptive indicators (recommendations of best practices) are based.10,22–23,32–63We found that only 38% of recommendations were based on high levels of scientific evidence (1a–2b). Most of the remaining ones (62%), particularly Australian Council on Healthcare Standards prescriptive indicators, relied nearly exclusively on expert opinion or limited case series.

Following this systematic review, we identified 108 clinical indicators developed for anesthesia. Nearly half of them (53) also measured surgical or postoperative ward care. Most were either outcome (57%) or process (42%) indicators. Only 1% related to the structure (setting attributes) of care. Patient safety (83%) and effectiveness (68%) were the two dimensions of quality of anesthetic care most often addressed, usually by outcome indicators. To identify possible quality issues, external benchmarking (comparison with other hospitals) and peer review by healthcare professionals were the primary methods in use. Only a few indicators were considered direct markers of quality and performance to be used for accountability purposes. Only 40% of the clinical indicators had been validated beyond face validity (expert opinion). The scientific evidence on which 38% of prescriptive indicators (recommendations) were based was globally high (1a–1b). However, a significant number of prescriptive indicators, particularly Australian Council on Healthcare Standards measures, were nearly exclusively based on expert opinion.

Systematic reviews are a highly recommended part of new indicators’ development processes.64,65However, this is the first time a systematic review of anesthesia-related clinical indicators has been performed, and no direct comparison with published or unpublished work in anesthesia could be done. To our knowledge, the only other review performed to identify clinical indicators was the one completed by researchers of the Agency for Healthcare Research and Quality for the development of its hospital-wide patient safety indicators program.#The authors reviewed more than 2,600 articles to find only 27 relevant articles representing altogether 37 different measures. Only 2 indicators were related to anesthesia: the rate of preventable adverse events and death rate associated with procedures involving anesthesia. This is far less than the 108 indicators found in this review. One of the main reasons for such discrepancies is that researchers from the Agency for Healthcare Research and Quality did not consider unpublished work or existing indicator programs.

However, most existing clinical indicators are only available from specific registries of performance measures or from clinical indicator programs.66The exclusive use of published peer-reviewed literature to identify available clinical indicators may significantly decrease the sensitivity of the overall review process.

Few clinical indicators are specifically defined, evaluated, and reported in the literature besides descriptive information on the process of development. Therefore, this review required a systematic and extensive search of the “gray” literature such as information from user’s manuals, books, accreditation bodies, Web sites, and quality initiatives programs.

There seems to be limited academic interest in clinical indicators, which tend to be viewed more as “quality improvement tools” than as true measures of clinical practice and patient outcome in anesthesia. It is therefore not surprising that the validation of anesthetic clinical indicators is largely limited to expert opinion (face validity). This should be considered carefully when governmental bodies or multiprovider organizations use clinical indicators to make external judgments or to apply rewards (i.e. , financial gain) or sanctions (i.e. , star ratings).67Even when process indicators based on sound scientific evidence are used, there has been insufficient demonstration that compliance with evidence-based best practice systematically results in better patient outcome.68,69The conflicting findings of two recent studies examining the impact on patient outcomes of an evidence-based guidelines for the treatment of acute myocardial infarction is a good example of this uncertainty.70,71 

Another aspect of the literature on clinical indicators is the lack of standardized and consensus definitions across systems and countries. For example, perioperative anesthesia-related mortality can be measured by three different indicators: death within 48 h of a procedure involving anesthesia, death rate associated with procedures involving anesthesia, or deaths within 30 days of surgery. The first indicator (death within 48 h) is considered to be a flag that can alert to possible problems in individual patient care, but it can also be used like the second one (death rate associated with procedures involving anesthesia) in the collection and analysis of monthly or annual rates for internal comparisons with previous values of the indicator. On the other hand, the third indicator (deaths within 30 days of surgery) is used for external benchmarking to compare hospitals between each other and build report cards for accountability purposes. These issues make attempts at classification quite complex and require the use of generic definitions and standardized tools. We used instruments developed by the Agency for Healthcare Research and Quality for the development of its indicators’ program and definitions commonly used in the literature.20,24,26 

One of the findings of this study is that most clinical indicators are used as indirect measures. Once an indicator is present, further steps need to be taken (usually a formal peer-review process) to confirm a potential quality issue of anesthetic care. This may require significant resources to set up professional peer-review committees. However, there are examples in the literature of integrated quality assurance programs where indicator measurement and peer-review are part of departmental routine activity.28 

Another finding is the predominance of outcome indicators related to patient safety. This may be due to a longstanding tradition, following the pioneer studies on ether and perioperative mortality, to systematically monitor adverse outcomes in anesthesia.72 

However, the number of process indicators for anesthesia care is currently growing. These process indicators are mainly prescriptive (“how things should be done”). They offer great promises as quality improvement tools. They define targets that have to be reached to ensure quality of anesthesia care. Despite their potential impact on anesthetists’ behavior, they become convincing improvement tools only if a causal link to important outcomes can be demonstrated.5Only if it can be demonstrated, for example, that a documented preanesthetic consultation leads to a decrease in postoperative morbidity, can this indicator be considered a valid quality improvement tool. As mentioned previously, this link is not always straightforward. Furthermore, there is an unsolved debate in the literature as to whether process or outcome measures should be favored.73Outcome measures have been accused of misuse and overinterpretation, leading to erroneous judgments or gambling behaviors.67,74However, outcome measures are deeply embedded in clinicians’ culture and occupy a prominent position in quality improvement efforts. Clinical autopsies to reveal diagnostic errors, or root-cause analyses of iatrogenic injuries to uncover system errors, have been widely recognized as effective methods to improve quality of care.

Outcome studies such as the Harvard Medical Practice Study and the Quality in Australian Healthcare Study have raised substantial interest in the medical community and have led to many improvement initiatives.42,43The nature itself of clinical practice largely relies on measures of outcome. This is why, despite the rapid development of process indicators, outcome measures will remain a significant part of the quality improvement measurement set for anesthesia.

Another finding of this review is that existing indicators developed and defined as such for anesthesia care focus largely on complications, medical errors, and adverse events from a physician’s perspective. Patients’ perspectives on the quality of anesthetic care are rarely incorporated into clinical indicators’ definitions. Further research should aim at developing clinical indicators that are based on patients’ perceptions and perspective over quality of anesthesia care.

Although established methods for the systematic retrieval, appraisal, and synthesis of the literature were used, a number of limitations of this study should be mentioned. First, to maximize the likelihood of identifying all relevant work on anesthesia-related clinical indicators, unpublished literature and information available on quality initiatives and accreditation bodies’ Web sites were also integrated in the search process. We also integrated a number of clinical indicators under development at the time of the literature review. If this contributed to enhance the sensitivity of our search strategy, it led to including information that had not been peer reviewed, the true quality and reliability of which was unclear. To limit this weakness, we systematically cross-checked Web site information with data published in articles, newsletters, expert committees’ technical papers, or users’ manuals.

The second limitation is that only work published in English was included in the study. A number of articles on clinical indicators, published predominantly in Spanish and Portuguese, were not considered. This might have introduced a publication bias in this review and a number of clinical indicators may have been missed. However, because the United States, Australia, Canada, and the United Kingdom have played a leading role in the development of clinical indicators, it can be assumed that only a minimal number of indicators were missed in this way.

Third, the identification of relevant themes from text descriptions in quality indicators’ programs and publications is highly subjective. Neither of the two reviewers performing data abstraction and synthesis was blinded. The selection and assessment of indicators’ validity could have been influenced by the reviewers’ understanding and beliefs about clinical indicators. This may have influenced the classification of dimensions of care measured and, more particularly, the rating of evidence of prescriptive indicators when not specified by developers. To minimize the latter, we chose to report ranges rather than individual scores for the level of evidence supporting prescriptive indicators (recommendations). For example, we attributed a score between 1a and 4 to the indicator of patients with a history of PONV to whom a prophylactic antiemetic has been administered, because it had a level 1a for the evidence that prophylaxis for nausea and vomiting is effective.53It had a level 2b for the evidence that patients with a history of PONV/travel sickness are at higher risk of nausea and vomiting after anesthesia.63It had a level 4 for the demonstration that following the whole algorithm and providing prophylactic antiemetic to patients with a history of PONV decreased the incidence of postoperative nausea and vomiting when compared with a control group of patients without past history of PONV and not receiving any antiemetic medication.61Reporting ranges rather than individual scores of evidence also allowed accounting for the heterogeneity of the literature on which prescriptive indicators are based. Studies supporting the validity of recommendations are often a mixture of efficacy and effectiveness studies. For example, studies supporting the recommendation to administer antibiotic prophylaxis before surgery include both studies analyzing antibiotic penetration in tissues/wound infection75,76(efficacy) and studies looking at comparative mortality, duration of stay, and cost in patients receiving preoperative antibiotics (effectiveness).77 

To minimize reviewers’ bias, we also chose to use standardized definitions from the National Library of Clinical Indicators and commonly used in the literature.20,24,26We based data retrieval on content analysis methodology and used a standardized extraction coding template. We considered only definitions, dimensions, and other indicator characteristics for which both reviewers had a consensus.

Because no previous systematic review of clinical indicators in anesthesia has been performed previously, the assessment and classification methodology had to be developed specifically for this study or borrowed from other areas, such as evidence-based medicine. Therefore, although based on previously published work, none of the tools used in this study had been validated for the specific assessment of anesthesia-related clinical indicators. Further validation could have been performed, for example, by using qualitative methodologies, to ensure that themes and definitions identified by the standardized tools in this study were reproducible with other data sources.78,79 

We also limited our review of quality measures of anesthetic care to clinical indicators. Therefore, we did not include patient satisfaction questionnaires80,81or patients’ complaints,82which are also important measures of quality. These measures reflect more specifically patients’ perspective on quality than clinical indicators. These measures should be further investigated in future research.

Finally, we chose to cite only primary promoters cited in the published literature, and not all organizations involved in indicators’ development. This was done because several professional and governmental organizations are often involved in clinical indicators’ development. The extent (and limits) of these collaborations are difficult to describe accurately. As a result, we may have failed to acknowledge significant contributions to the development of some of these indicators.

Despite these limitations, it has been possible to perform an extensive systematic review of existing clinical indicators available in anesthesia and to identify indicators’ characteristics and validity.

In summary, our systematic review identified 108 clinical indicators developed for quality and safety measurement in anesthesia. Approximately half of them could be considered specific to the specialty, whereas the remaining measures also assessed surgical or postoperative ward care at the same time. Nearly all indicators were indirect measures designed to trigger further peer review of quality and safety issues occurring during anesthesia. Most indicators measured several dimensions of anesthetic care, and safety was the dimension that was most commonly addressed. Only 40% of clinical indicators had been validated beyond face validity. The level of scientific evidence on which prescriptive measures were based was found to be high (1a–1b) for 38% and low (4–5) for 62% of indicators. Efforts to develop and further validate clinical indicators specific to the practice of anesthesia should be strongly promoted.

The authors thank Dagmar Haller-Hester, M.D., Ph.D. (Research Fellow, Department of Community Medicine and Primary Care, Geneva University Hospitals, Geneva, Switzerland), for her editorial advice.