Clinical indicators are increasingly developed and promoted by professional organizations, governmental agencies, and quality initiatives as measures of quality and performance. To clarify the number, characteristics, and validity of indicators available for anesthesia care, the authors performed a systematic review. They identified 108 anesthetic clinical indicators, of which 53 related also to surgical or postoperative ward care. Most were process (42%) or outcome (57%) measures assessing the safety and effectiveness of patient care. To identify possible quality issues, most clinical indicators were used as part of interhospital comparison or professional peer-review processes. For 60% of the clinical indicators identified, validity relied on expert opinion. The level of scientific evidence on which prescriptive indicators ("how things should be done") were based was high (1a-1b) for 38% and low (4-5) for 62% of indicators. Additional efforts should be placed into the development and validation of anesthesia-specific quality indicators.

QUALITY and safety in anesthesia is usually monitored by analysis of perioperative mortality–morbidity and incidents.1–4However, these methods have limited sensitivity and specificity for quality and safety issues. Patient perioperative mortality and morbidity are not always related to anesthesia. Incidents largely rely on the willingness of staff members to report them. As a consequence, a number of additional measurement tools are increasingly promoted, particularly clinical indicators.5 

Indicators are primarily measures of a nonquantifiable construct: quality of care. Developed initially in the manufacturing industry, indicators were first introduced into the healthcare industry in 1982 by the Federal Government of the United States for Medicare beneficiaries, as part of its Professional Review Organization program.6,7Original clinical indicators were therefore designed as Generic Quality Screens or “flags,” which required individual case review to identify undesirable occurrences related to problems with the quality of care provided.

Clinical indicators were soon adopted by the Maryland Hospital Association and the Joint Commission on Accreditation of Healthcare Organizations, and several experts from professional organizations were involved in their development.6,8,9Between 1987 and 1993, the anesthesia care task force assembled by the Joint Commission developed 14 anesthesia-related indicators to continuously monitor organizational performance across hospitals through a national Indicator Measurement System. Indicators were divided into two categories: (1) sentinel event indicators—unusual and isolated occurrences involving death or serious physical or psychological injury (e.g. , perioperative cardiac arrest); and (2) rate-based indicators—abnormal trends in a particular type of process or outcome of care recorded on a regular basis and reported as a ratio comprising numerator and denominator (e.g. , yearly rate of unplanned intensive care admissions after anesthesia).8 

This initial development was followed by field testing for reliability assessment in a number of volunteer US hospitals. The number of anesthesia-related indicators in the Indicator Measurement System program was consequently reduced to five (perioperative central nervous system complications, peripheral nervous system complications, acute myocardial infarction, cardiac arrest, and death). Anesthesia-related indicators were also renamed into perioperative indicators as experts concluded that these measures were not specific to anesthesia care.10The Indicator Measurement System program is one of the many indicator programs developed in the United States and in other countries, such as Australia, Canada, and the United Kingdom.11–16

In recent years, the increasing involvement of healthcare professional organizations, governmental bodies, and managed care organizations into the running of the healthcare system has boosted the development of indicator programs, particularly in the United States.17–19To respond to stakeholders’ pressures to measure both professional and organizational performance, various meanings and definitions of clinical indicators have also proliferated. As a result, the number and complexity of these measures have greatly increased.20This is particularly true for anesthesia, where the nature, characteristics, and validity of clinical indicators are unclear. The aim of this systematic review is to assess the number of clinical indicators currently available for patient quality and safety measurement in anesthesia, the level of evidence for their validity, and the recommended method for their use.

Materials and Methods

Search Strategy and Selection Criteria

We searched PubMed (1970–December 2005), EMBASE (1970–December 2005), CINAHL (1980–December 2005), and the Cochrane library–DARE (1990–December 2005) for all English-language articles relating to the development or use of clinical indicators. We also looked for clinical indicator programs available on governmental, private, and professional organizations’ publications and Web sites, if organizations were established at national level and had information in English. Some examples in the United States include the American Medical Association’s Physician Consortium for Performance Improvement Program, the National Committee for Quality Assurances Indicators Program, the Centers for Medicare and Medicaid Services, the Surgical Care Improvement Project, the American Society of Anesthesiologists’ Committee on Performance and Outcome program, and the Maryland Hospital Association Quality Indicator Program. In Australia, these include the Australian Council on Healthcare Standards–Care Evaluation Program, the Australian Commission on Safety and Quality in Healthcare Initiative, and the National Hospital Quality Management Program; in Canada, the Canadian Health Department indicator program; in New Zealand, the Health System Performance Indicators from the National Department of Health; and in the United Kingdom, the National Health Service Indicator Program and the International Quality Indicator Project. For the rest of the world, these include the World Health Organization and the Organization for Economic Cooperation and Development technical papers and other publications. Finally, expert opinion was sought for unpublished indicators developed by quality initiatives and professional organizations. The detailed search strategy is available in Supplemental Digital Content 1 (see table,

For the purpose of this systematic review, a clinical indicator was defined as an explicit measure (defined by the developer) of some aspect of patient clinical care used to judge a particular clinical situation and indicate whether the care delivered was appropriate.21We included all publications (peer-reviewed and non–peer-reviewed) and Web sites describing the development and validation of indicators. To be included in the review, clinical indicators had to be also directly related to anesthesia. Indicators relating only to intensive care or surgical care were excluded. We also excluded indicators used for medicoeconomic purposes. Publications reporting raw figures of adverse events, incidents, and complications were also excluded because these did not comply with the definition of indicators. We did not consider studies on patient satisfaction because the metrics developed for its measurement rely on questionnaires and not on clinical indicators.

Data Extraction

For published articles, inclusion and exclusion criteria were extracted either from the abstract or from the full article. For indicator programs, the information provided by indicator summary tables was used. If incomplete, the rest of the Web site or manual of use was searched. Selected articles and indicator programs were then carefully examined, and information about the indicators was extracted. The overall process was performed by one of the authors and verified by the second author.

Indicator Classification Procedure

We used a standardized coding template#adapted to a unifying taxonomy system for classification of clinical indicators.20–22**The template included the indicator’s abbreviated/standard name, the developer’s definition, the level of specificity for anesthetic care, the area of care (i.e. , structure, process, outcome), and the measured dimension of quality (i.e. , continuity, effectiveness, safety). The template also integrated the type of indicator (descriptive, prescriptive, proscriptive) and the recommended method to assess the presence of an anesthesia-related quality issue.

We classified and incorporated indicators into tables according to their original developer or promoter specifications (hospital, accreditation body, quality initiatives, professional organizations, other). Therefore, if a clinical indicator had been developed through collaborative work of different organizations, only the primary promoters and not all of the organizations involved were considered.

If there was more than one primary promoter for the same indicator, we chose to select the one providing the most information and excluded the others. This was, for example, the case for indicators developed by the Health Administration program of the US Department of Veterans Affairs, such as acute myocardial infarct 2 days after anesthesia or cardiac arrest rate of patients receiving anesthesia, which were similar to clinical indicators developed by the American Society of Anesthesiologists.23††

If indicators were closely related, although not similar in their definition, they were individually cited in the first summary table but aggregated in the second and third tables, where indicators were analyzed. This was the case, for example, for the indicators of the Australian Council on Healthcare Standards analyzing preoperative documentation and patient consent and the American Society of Anesthesiologists indicator related to postoperative care unit stay.

When indicators defined different time periods for the same measurement, even if closely related, they were all considered. For example, both death within 48 h of a procedure involving anesthesia and death rate associated with procedures involving anesthesia were selected and described.

The full data abstraction and classification form is available in Supplemental Digital Content 2 (see table,

Definitions for Classification

Areas of Care.

Whenever information on specificity for anesthetic care, area, and dimension of quality measured was not provided by the developer, we used standardized definitions and classification procedures. We defined an indicator as specific to anesthetic care if it referred specifically to the practice of anesthesia. If it could also relate to surgical or postoperative ward care, it was defined as general. To classify indicators into the different areas of care, we referred to the model and definitions for quality assessment developed by Donabedian24: the structure, process, and outcome model. This model sees health care as a cyclic transformation mechanism. Patients are inputs  entering a healthcare organization’s structure  to undergo a process  of care through which they will become outcomes/outputs , which will further inform the feedback loop  back to inputs . In our case, indicators of processes  referred to measures assessing the implementation of program activities, and indicators of outcomes  referred to those measuring patient-related end results of anesthesia care. Structure  referred to hospital staff, material, and overall organization.

Dimension of Quality.

The classification template also included the dimension of quality measured by indicators. Because there is no single definition of quality and well-known definitions such as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge”25cannot be easily used to discriminate indicators, we chose the approach developed by the Joint Commission in its extensive review of clinical indicators, the National Library of Healthcare indicators.26It describes quality of care through its different attributes rather than through a single definition. It identifies 10 different attributes of quality: appropriateness, availability, continuity, effectiveness, efficacy, efficiency, prevention, respect and caring, safety, and timeliness. Details regarding these attributes’ definitions are available in Supplemental Digital Content 3 (see

Descriptive, Prescriptive, or Proscriptive Type of Indicator.

We also classified clinical indicators into three main distinct category types.27Descriptive indicators were those that provided descriptive information on unusual situations of patient care that could reveal, if further investigated, potential defects in the quality of care provided. An unplanned overnight admission of day surgery patients for anesthetic reasons was, for example, considered a descriptive indicator. Prescriptive indicators were defined as indicators representing recommendations or desired targets. Prophylactic antibiotic selection for surgical patients according to current recommendations, for example, was considered a prescriptive indicator. Proscriptive indicators were defined as measures of actions that “should not be performed,” such as, for example, medication error with the wrong medication being given.

Recommended Methods for the Identification of an Anesthesia-related Quality Issue.

To classify developers’ recommended methods for the identification of an anesthesia-related quality issue, we used the five major models described in the literature8,11–17,21,22:

  1. Peer review of medical charts and other data sources: Individual cases flagged by the indicator are reviewed by a committee of experts to identify possible quality and safety issues.

  2. Hospital internal benchmarking or comparison: The current rate or frequency of the indicator is compared with previous average rates for the indicator, and an abnormally high rate or frequency (usually > 2 SDs) indicates possible quality and safety issues.

  3. External benchmarking or comparison with other hospitals’ average rates: The hospital current rate or frequency of the indicator is compared with other hospitals’ average rates, and an abnormally high rate or frequency (usually > 2 SDs) indicates possible quality and safety issues.

  4. Noncompliance with accepted and widely recognized standards (± based on scientific evidence): This noncompliance identified by the indicator indicates possible quality and safety issues.

  5. Risk adjustment for comorbidities and other factors: Risk-adjusted rates for the indicator higher than the reference population indicate possible quality and safety issues.

Because there were often, for the same indicator, several methods recommended to identify anesthesia-related quality issues, we chose to report all of them in the summary tables. For example, the American Society of Anesthesiologists indicators could be used both as flags to guide a peer review process (method 1)††and as rates plotted on a statistical process control chart for hospital internal benchmarking (method 2).28 

Level of Validity.

Finally, we extracted and assessed the level of validity of each indicator (the extent to which it measures what it supposed to measure) based on its psychometric properties. This is the approach most widely used by developers and expert committees. It recognizes four different types of validity: face, content, construct, and criterion. Face validity  refers to the extent to which, on the face of it, an indicator seems to be measuring what it is intended to measure. This is usually a subjective judgment based on the review of the measure itself by one or more experts.29,Content validity  is the extent to which an indicator samples all relevant subdimensions of the domain under study. For example, an indicator assessing quality of life needs to integrate all of the different domains composing quality of life: physical functioning, bodily pain, mental health, general health, social functioning, emotional/physical role, and vitality.30An instrument assessing only pain, for example, would not be considered to have content validity as a measure of quality of life. Construct validity  is the demonstration that there is a significant convergence between a new measurement tool and previously validated measures of the same attribute (construct). A good example is the assessment of the convergence between the Glasgow coma score and the cerebral metabolic rate or visual evoked potentials to determine the validity of the score as a measure of brain damage and coma.31 Criterion validity  is the extent to which an indicator agrees with a gold standard of the domain or phenomenon being measured. The later is not used for indicators’ validation because there is no consensus gold standard to measure quality of care.

To assess indicators’ validity, we used documentation provided by the developer, expert committees, or any publication related to the formal validation procedure of indicators. If publications reported only examples of use of the measure, they were not considered to be formal validation studies. When indicators were prescriptive (recommendations of best practices), we assessed whether these practices had a scientifically proven value. We searched the literature for published evidence on the scientific soundness of the proposed recommendations. For example, for the prescriptive indicator of patients with a history of postoperative nausea and vomiting (PONV) to whom a prophylactic antiemetic has been administered, we assessed whether there was any published evidence that prescription of prophylactic antiemetic treatment would benefit patients with a history PONV. If publications were found, we rated the reliability of the evidence they provided using the widely publicized scale developed by the Oxford Centre for Evidence-based Medicine.‡‡

This scale enables studies to be ranked from 1a to 5 according to the validity of their findings. Therefore, a prescriptive indicator based on results from a large randomized controlled trial or systematic review would be rated as 1, whereas a recommendation supported exclusively by expert opinion would be rated as 5. A simplified version of the scale is provided in fig. 1, and a more detailed one can be found in Supplemental Digital Content 4 (see table, When this was available, we used the level of evidence mentioned by the developer. In all other cases, we performed a literature search to find all publications related to the recommendation and rated prescriptive indicators according to the level of evidence provided by these publications.

Fig. 1. Levels of evidence (simplified version of the Oxford Center for Evidence-based Medicine Scale) .

Fig. 1. Levels of evidence (simplified version of the Oxford Center for Evidence-based Medicine Scale) .

Procedure for Classification and Evaluation

The overall classification and evaluation process of indicators was performed by two assessors. Both were medical doctors with health services research and anesthetic training, respectively. Any disagreement between the two reviewers was solved by discussion until a consensus between the two assessors was reached.


We identified 834 articles, of which a total of 22 met all inclusion criteria, particularly regarding indicators’ definition and relevance to the specialty of anesthesia. The search strategy also included private and governmental quality organizations’ Web sites and publications, including technical papers and manuals. We found 37 indicator programs, of which 11 were excluded because they did not relate to the specialty of anesthesia, were irrelevant, or did not correspond to selected definitions. Figure 2provides a detailed description of the selection process of articles and indicator programs. Overall, 22 articles and 26 Web sites, manuals, and technical papers were selected for full data abstraction and assessment, resulting in the identification of 108 different clinical indicators related to the practice of anesthesia.10–16,22,23,26,28,32–52#**§§∥∥##***†††‡‡‡§§§∥∥∥###****††††‡‡‡‡§§§§

Fig. 2. Selection process for articles and indicator programs (including Web sites, manuals, and technical papers) .

Fig. 2. Selection process for articles and indicator programs (including Web sites, manuals, and technical papers) .

An overview of all selected clinical indicators is provided in table 1.

Table 1. Overview of Studies and Programs for Anesthetic Clinical Indicators 

Table 1. Overview of Studies and Programs for Anesthetic Clinical Indicators 
Table 1. Overview of Studies and Programs for Anesthetic Clinical Indicators 

Table 1. Continued 

Table 1. Continued 
Table 1. Continued 

Table 1. Continued 

Table 1. Continued 
Table 1. Continued 

A large number of indicators were closely related to each other, such as total perioperative mortality for all American Society of Anesthesiologists classes, perioperative mortality for American Society of Anesthesiologists classes I–V, or death rate associated with procedures involving anesthesia. Some indicators, such as mortality within 30 days of surgery, although developed by different quality initiatives in different countries (United Kingdom and United States), were identical to each other. These were consequently aggregated in table 2, which describes indicators’ characteristics and validity.

Table 2. Characteristics and Validity of Clinical Indicators for Anesthesia Care 

Table 2. Characteristics and Validity of Clinical Indicators for Anesthesia Care 
Table 2. Characteristics and Validity of Clinical Indicators for Anesthesia Care 

Table 2. Continued 

Table 2. Continued 
Table 2. Continued 

Table 2. Continued 

Table 2. Continued 
Table 2. Continued 

Table 2. Continued 

Table 2. Continued 
Table 2. Continued 

Table 2. Continued 

Table 2. Continued 
Table 2. Continued 

We identified 55 indicators related specifically to the practice of anesthesia and 53 that were also related to the practice of surgery, emergency medicine, or postoperative ward care. Only 26 indicators measured exclusively one dimension of quality of anesthetic care (usually safety and appropriateness). All others assessed several dimensions of quality at the same time. The combination we most often identified was appropriateness (relevance to patient’s clinical need), effectiveness (correct care according to current state of knowledge), and patient safety (absence of avoidable iatrogenic complications). Safety was the most common dimension measured by anesthetic clinical indicators (83%), followed by effectiveness (68%). More than half of the indicators (57%) were measures of outcome, and the remaining ones were process (42%) or structure indicators (1%).

Most outcome measures related to patient safety and effectiveness. To identify possible anesthesia-related issues, the most frequently suggested method was external benchmarking (comparison with other hospitals) to flag abnormal variations that required expert-based assessment to determine causes and improvement strategies. A number of indicators were also considered direct markers of quality and performance to be integrated into report cards (e.g. , death within 30 days of surgery) or pay for performance programs (e.g. , surgical patients for whom administration of a prophylactic antibiotic has been initiated within 1 h before the surgical incision, noncardiac surgical patients who received prophylactic antibiotics and who have an order for discontinuation within 24 h of surgical end time).

All indicators identified in the review had face validity and had been assessed by expert committee. Only 40% had undergone an additional content and/or construct validation process through patient case note review or other related methods (table 2).

Table 3reports the level of scientific soundness of the literature on which prescriptive indicators (recommendations of best practices) are based.10,22–23,32–63We found that only 38% of recommendations were based on high levels of scientific evidence (1a–2b). Most of the remaining ones (62%), particularly Australian Council on Healthcare Standards prescriptive indicators, relied nearly exclusively on expert opinion or limited case series.

Table 3. Level of Scientific Evidence on Which Prescriptive Indicators (Recommendations) Are Based 

Table 3. Level of Scientific Evidence on Which Prescriptive Indicators (Recommendations) Are Based 
Table 3. Level of Scientific Evidence on Which Prescriptive Indicators (Recommendations) Are Based 


Following this systematic review, we identified 108 clinical indicators developed for anesthesia. Nearly half of them (53) also measured surgical or postoperative ward care. Most were either outcome (57%) or process (42%) indicators. Only 1% related to the structure (setting attributes) of care. Patient safety (83%) and effectiveness (68%) were the two dimensions of quality of anesthetic care most often addressed, usually by outcome indicators. To identify possible quality issues, external benchmarking (comparison with other hospitals) and peer review by healthcare professionals were the primary methods in use. Only a few indicators were considered direct markers of quality and performance to be used for accountability purposes. Only 40% of the clinical indicators had been validated beyond face validity (expert opinion). The scientific evidence on which 38% of prescriptive indicators (recommendations) were based was globally high (1a–1b). However, a significant number of prescriptive indicators, particularly Australian Council on Healthcare Standards measures, were nearly exclusively based on expert opinion.

Systematic reviews are a highly recommended part of new indicators’ development processes.64,65However, this is the first time a systematic review of anesthesia-related clinical indicators has been performed, and no direct comparison with published or unpublished work in anesthesia could be done. To our knowledge, the only other review performed to identify clinical indicators was the one completed by researchers of the Agency for Healthcare Research and Quality for the development of its hospital-wide patient safety indicators program.#The authors reviewed more than 2,600 articles to find only 27 relevant articles representing altogether 37 different measures. Only 2 indicators were related to anesthesia: the rate of preventable adverse events and death rate associated with procedures involving anesthesia. This is far less than the 108 indicators found in this review. One of the main reasons for such discrepancies is that researchers from the Agency for Healthcare Research and Quality did not consider unpublished work or existing indicator programs.

However, most existing clinical indicators are only available from specific registries of performance measures or from clinical indicator programs.66The exclusive use of published peer-reviewed literature to identify available clinical indicators may significantly decrease the sensitivity of the overall review process.

Few clinical indicators are specifically defined, evaluated, and reported in the literature besides descriptive information on the process of development. Therefore, this review required a systematic and extensive search of the “gray” literature such as information from user’s manuals, books, accreditation bodies, Web sites, and quality initiatives programs.

There seems to be limited academic interest in clinical indicators, which tend to be viewed more as “quality improvement tools” than as true measures of clinical practice and patient outcome in anesthesia. It is therefore not surprising that the validation of anesthetic clinical indicators is largely limited to expert opinion (face validity). This should be considered carefully when governmental bodies or multiprovider organizations use clinical indicators to make external judgments or to apply rewards (i.e. , financial gain) or sanctions (i.e. , star ratings).67Even when process indicators based on sound scientific evidence are used, there has been insufficient demonstration that compliance with evidence-based best practice systematically results in better patient outcome.68,69The conflicting findings of two recent studies examining the impact on patient outcomes of an evidence-based guidelines for the treatment of acute myocardial infarction is a good example of this uncertainty.70,71 

Another aspect of the literature on clinical indicators is the lack of standardized and consensus definitions across systems and countries. For example, perioperative anesthesia-related mortality can be measured by three different indicators: death within 48 h of a procedure involving anesthesia, death rate associated with procedures involving anesthesia, or deaths within 30 days of surgery. The first indicator (death within 48 h) is considered to be a flag that can alert to possible problems in individual patient care, but it can also be used like the second one (death rate associated with procedures involving anesthesia) in the collection and analysis of monthly or annual rates for internal comparisons with previous values of the indicator. On the other hand, the third indicator (deaths within 30 days of surgery) is used for external benchmarking to compare hospitals between each other and build report cards for accountability purposes. These issues make attempts at classification quite complex and require the use of generic definitions and standardized tools. We used instruments developed by the Agency for Healthcare Research and Quality for the development of its indicators’ program and definitions commonly used in the literature.20,24,26 

One of the findings of this study is that most clinical indicators are used as indirect measures. Once an indicator is present, further steps need to be taken (usually a formal peer-review process) to confirm a potential quality issue of anesthetic care. This may require significant resources to set up professional peer-review committees. However, there are examples in the literature of integrated quality assurance programs where indicator measurement and peer-review are part of departmental routine activity.28 

Another finding is the predominance of outcome indicators related to patient safety. This may be due to a longstanding tradition, following the pioneer studies on ether and perioperative mortality, to systematically monitor adverse outcomes in anesthesia.72 

However, the number of process indicators for anesthesia care is currently growing. These process indicators are mainly prescriptive (“how things should be done”). They offer great promises as quality improvement tools. They define targets that have to be reached to ensure quality of anesthesia care. Despite their potential impact on anesthetists’ behavior, they become convincing improvement tools only if a causal link to important outcomes can be demonstrated.5Only if it can be demonstrated, for example, that a documented preanesthetic consultation leads to a decrease in postoperative morbidity, can this indicator be considered a valid quality improvement tool. As mentioned previously, this link is not always straightforward. Furthermore, there is an unsolved debate in the literature as to whether process or outcome measures should be favored.73Outcome measures have been accused of misuse and overinterpretation, leading to erroneous judgments or gambling behaviors.67,74However, outcome measures are deeply embedded in clinicians’ culture and occupy a prominent position in quality improvement efforts. Clinical autopsies to reveal diagnostic errors, or root-cause analyses of iatrogenic injuries to uncover system errors, have been widely recognized as effective methods to improve quality of care.

Outcome studies such as the Harvard Medical Practice Study and the Quality in Australian Healthcare Study have raised substantial interest in the medical community and have led to many improvement initiatives.42,43The nature itself of clinical practice largely relies on measures of outcome. This is why, despite the rapid development of process indicators, outcome measures will remain a significant part of the quality improvement measurement set for anesthesia.

Another finding of this review is that existing indicators developed and defined as such for anesthesia care focus largely on complications, medical errors, and adverse events from a physician’s perspective. Patients’ perspectives on the quality of anesthetic care are rarely incorporated into clinical indicators’ definitions. Further research should aim at developing clinical indicators that are based on patients’ perceptions and perspective over quality of anesthesia care.

Although established methods for the systematic retrieval, appraisal, and synthesis of the literature were used, a number of limitations of this study should be mentioned. First, to maximize the likelihood of identifying all relevant work on anesthesia-related clinical indicators, unpublished literature and information available on quality initiatives and accreditation bodies’ Web sites were also integrated in the search process. We also integrated a number of clinical indicators under development at the time of the literature review. If this contributed to enhance the sensitivity of our search strategy, it led to including information that had not been peer reviewed, the true quality and reliability of which was unclear. To limit this weakness, we systematically cross-checked Web site information with data published in articles, newsletters, expert committees’ technical papers, or users’ manuals.

The second limitation is that only work published in English was included in the study. A number of articles on clinical indicators, published predominantly in Spanish and Portuguese, were not considered. This might have introduced a publication bias in this review and a number of clinical indicators may have been missed. However, because the United States, Australia, Canada, and the United Kingdom have played a leading role in the development of clinical indicators, it can be assumed that only a minimal number of indicators were missed in this way.

Third, the identification of relevant themes from text descriptions in quality indicators’ programs and publications is highly subjective. Neither of the two reviewers performing data abstraction and synthesis was blinded. The selection and assessment of indicators’ validity could have been influenced by the reviewers’ understanding and beliefs about clinical indicators. This may have influenced the classification of dimensions of care measured and, more particularly, the rating of evidence of prescriptive indicators when not specified by developers. To minimize the latter, we chose to report ranges rather than individual scores for the level of evidence supporting prescriptive indicators (recommendations). For example, we attributed a score between 1a and 4 to the indicator of patients with a history of PONV to whom a prophylactic antiemetic has been administered, because it had a level 1a for the evidence that prophylaxis for nausea and vomiting is effective.53It had a level 2b for the evidence that patients with a history of PONV/travel sickness are at higher risk of nausea and vomiting after anesthesia.63It had a level 4 for the demonstration that following the whole algorithm and providing prophylactic antiemetic to patients with a history of PONV decreased the incidence of postoperative nausea and vomiting when compared with a control group of patients without past history of PONV and not receiving any antiemetic medication.61Reporting ranges rather than individual scores of evidence also allowed accounting for the heterogeneity of the literature on which prescriptive indicators are based. Studies supporting the validity of recommendations are often a mixture of efficacy and effectiveness studies. For example, studies supporting the recommendation to administer antibiotic prophylaxis before surgery include both studies analyzing antibiotic penetration in tissues/wound infection75,76(efficacy) and studies looking at comparative mortality, duration of stay, and cost in patients receiving preoperative antibiotics (effectiveness).77 

To minimize reviewers’ bias, we also chose to use standardized definitions from the National Library of Clinical Indicators and commonly used in the literature.20,24,26We based data retrieval on content analysis methodology and used a standardized extraction coding template. We considered only definitions, dimensions, and other indicator characteristics for which both reviewers had a consensus.

Because no previous systematic review of clinical indicators in anesthesia has been performed previously, the assessment and classification methodology had to be developed specifically for this study or borrowed from other areas, such as evidence-based medicine. Therefore, although based on previously published work, none of the tools used in this study had been validated for the specific assessment of anesthesia-related clinical indicators. Further validation could have been performed, for example, by using qualitative methodologies, to ensure that themes and definitions identified by the standardized tools in this study were reproducible with other data sources.78,79 

We also limited our review of quality measures of anesthetic care to clinical indicators. Therefore, we did not include patient satisfaction questionnaires80,81or patients’ complaints,82which are also important measures of quality. These measures reflect more specifically patients’ perspective on quality than clinical indicators. These measures should be further investigated in future research.

Finally, we chose to cite only primary promoters cited in the published literature, and not all organizations involved in indicators’ development. This was done because several professional and governmental organizations are often involved in clinical indicators’ development. The extent (and limits) of these collaborations are difficult to describe accurately. As a result, we may have failed to acknowledge significant contributions to the development of some of these indicators.

Despite these limitations, it has been possible to perform an extensive systematic review of existing clinical indicators available in anesthesia and to identify indicators’ characteristics and validity.

In summary, our systematic review identified 108 clinical indicators developed for quality and safety measurement in anesthesia. Approximately half of them could be considered specific to the specialty, whereas the remaining measures also assessed surgical or postoperative ward care at the same time. Nearly all indicators were indirect measures designed to trigger further peer review of quality and safety issues occurring during anesthesia. Most indicators measured several dimensions of anesthetic care, and safety was the dimension that was most commonly addressed. Only 40% of clinical indicators had been validated beyond face validity. The level of scientific evidence on which prescriptive measures were based was found to be high (1a–1b) for 38% and low (4–5) for 62% of indicators. Efforts to develop and further validate clinical indicators specific to the practice of anesthesia should be strongly promoted.

The authors thank Dagmar Haller-Hester, M.D., Ph.D. (Research Fellow, Department of Community Medicine and Primary Care, Geneva University Hospitals, Geneva, Switzerland), for her editorial advice.


Aitkenhead AR: Injuries associated with anesthesia: A global perspective. Br J Anaesth 2005; 95:95–109
Lee A, Lum ME: Measuring anaesthetic outcomes. Anaesth Intensive Care 1996; 24:685–93
Cooper JB, Newbower RS, Kitz RJ: An analysis of major errors and equipment failures in anesthesia management: Considerations for prevention and detection. Anesthesiology 1984; 60:34–42
Runciman WB, Sellen A, Webb RK, Williamson JA, Currie M, Morgan C, Russell WJ: The Australian Incident Monitoring Study: Errors, incidents and accidents in anaesthetic practice. Anaesth Intensive Care 1993; 21:506–19
Pronovost PJ, Nolan T, Zeger S, Miller M, Rubin H: How can clinicians measure safety and quality in acute care? Lancet 2004; 363:1061–7
Dodson BA: Quality assurance/quality improvement, Anesthesia, volume 2,5th edition. Edited by Miller RD. Philadelphia, Churchill Livingstone, 2000, pp 597–612Miller RD
Churchill Livingstone
Genovich-Richards J: Quality management organizational structures: History and trends. J Healthc Qual 1994;16: 22–5, 28–9
Nadzam DM, Turpin R, Hanold LS, White RE: Data-driven performance improvement in health care: The Joint Commission’s Indicator Measurement System (IMSystem). Jt Comm J Qual Improv 1993; 19:492–500
Institute of Medicine, Committee to Design a Strategy for Quality Review and Assurance in Medicare: Medicare: A Strategy for Quality Assurance Volume II: Source & Methods. Washington, D.C., National Academy Press, 1990, pp 140–291
Washington, D.C.
National Academy Press
Gabel RA: Evolution of Joint Commission anesthesia clinical indicators. ASA Newsletter 1994; 58:24–9
Department of Health: Quality and Performance in the NHS: NHS Performance Indicators. NHS catalogue No. 21946. Leeds, Department of Health Publishing Services, 2000, pp 1–119
Department of Health Publishing Services
Collopy BT, Ansari MZ, Booth JL, Brosi JA: The Australian Council on Healthcare Standards Care Evaluation Program. Med J Aust 1995; 163: 477–80
Barbour GL: Development of a quality improvement checklist for the Department of Veterans Affairs. Jt Comm J Qual Improv 1994; 20:127–39
Kazandjian VA, Matthes N, Wicker KG: Are performance indicators generic? The international experience of the Quality Indicator Project. J Eval Clin Pract 2003; 9:265–76
Ballard DJ: Indicators to improve clinical quality across an integrated health care system. Int J Qual Health Care 2003; 15(suppl 1):i13–23
Broyce N, McNeil J, Graves D, Dunt D: Quality and Outcome Indicators for Acute Health Care Services. Canberra, Commonwealth Department of Health and Family Services, Australian Government Publishing Service, 1997, pp 17-29
Commonwealth Department of Health and Family Services, Australian Government Publishing Service
Brook RH, McGlynn EA, Shekelle PG: Defining and measuring quality of care: A perspective from US researchers. Int J Qual Health Care 2000; 12:281–95
McLoughlin V, Leatherman S, Fletcher M, Owen JW: Improving performance using indicators: Recent experiences in the United States, the United Kingdom, and Australia. Int J Qual Health Care 2001; 13:455–62
Arah OA, Klazinga NS: How safe is the safety paradigm? Qual Saf Health Care 2004; 13:226–32
Mainz J: Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 2003; 15:523–30
Ibrahim JE, Majoor JW, Boyce NW, McNeil JJ: Pilot Hospital-wide Clinical Indicators Project: Final Report. Canberra, Commonwealth Department of Health and Family Services, Health Services Outcomes Branch, 1998, pp 8–22
Commonwealth Department of Health and Family Services, Health Services Outcomes Branch
Kast JE, Rosenzweig JE: General systems theory: Applications of organizations & management. Acad Management J 1972; 15:447–65
US Department of Veterans Affairs: Final Report—Quality Management in the Department of Veterans Affairs’ Veterans Health Administration, Final report—Report No. 8HI-A28-072. Washington, D.C., Department of Veterans Affairs, Office of Inspector General, 1998, pp 1–37
Donabedian A: The quality of care: How can it be assessed? JAMA 1988; 260:1743–8
Institute of Medicine, Committee on Quality of Health Care in America: Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, D.C., National Academy Press,, 2001, pp 232
Institute of Medicine, Committee on Quality of Health Care in America
Washington, D.C.
National Academy Press
Joint Commission on Accreditation of Healthcare Organizations: National Library of Healthcare Indicators. Oakbrook Terrace, Illinois, 1997, pp 193–409
Joint Commission on Accreditation of Healthcare Organizations
Oakbrook Terrace, Illinois
Freeman T: Using performance indicators to improve health care quality in the public sector: A review of the literature. Health Serv Manage Res 2002; 15:126–37
Lagasse RS, Steinberg ES, Katz RI, Saubermann AJ: Defining quality of perioperative care by statistical process control of adverse outcomes. Anesthesiology 1995; 82:1181–8
Streiner DL, Norman GR: Health Measurement Scales: A Practical Guide to Their Development and Use. 3rd edition. Oxford Oxford University Press,, 2003, pp 4–13
Oxford University Press
Ware J Jr, Kosinski M, Keller SD: A 12-item Short-Form Health Survey: Construction of scales and preliminary tests of reliability and validity. Med Care 1996; 34:220–33
Prasad K: The Glasgow Coma Scale: A critical appraisal of its clinimetric properties. J Clin Epidemiol 1996; 49:755–63
Australian Council on Healthcare Standards: Australasian Clinical Indicator Report: 1998–2006: Determining the Potential to Improve Quality of Care, 8th edition. Newcastle, Australian Council on Healthcare Standards Health Service, 2007, pp 56–129
Australian Council on Healthcare Standards
Australian Council on Healthcare Standards Health Service
Australian Council on Healthcare Standards, Australian and New Zealand College of Anaesthetists: Clinical Indicators: A Users’ Manual, version 2 for use after January 1998. Melbourne, Australian Council on Healthcare Standards, Australian and New Zealand College of Anaesthetists, 1997, pp 3–26
Margovsky A: Unplanned admissions in day-case surgery as a clinical indicator for quality assurance. Aust N Z J Surg 2000; 70:216–20
Morales R, Esteve N, Casas I, Blanco C: Why are ambulatory surgical patients admitted to hospital? Prospective study. Ambulatory Surg 2002; 9:197–205
Collopy BT: Clinical indicators in accreditation: An effective stimulus to improve patient care. Int J Qual Health Care 2000; 12:211–6
Romano PS, Geppert JJ, Davies S, Miller MR, Elixhauser A, McDonald KM: A national profile of patient safety in U.S. hospitals. Health Affairs (Millwood) 2003; 22:154–66
Rosen AK, Rivard P, Zhao S, Loveland S, Tsilimingras D, Christiansen CL, Elixhauser A, Romano PS: Evaluating the patient safety indicators: How well do they perform on Veterans Health Administration data? Med Care 2005; 43:873–84
US Department of Veterans Affairs, Veterans Health Administration: Quality Management (QM) and Patient Safety Activities That Can Generate Confidential Documents, VHA Directive 2004-051. Washington, D.C., 2004, pp 1–7
US Department of Veterans Affairs, Veterans Health Administration
Washington, D.C.
Halpern J: The measurement of quality of care in the Veterans Health Administration. Med Care 1996; 34:(suppl 3):55–68
Kizer KW: The “new VA”: A national laboratory for health care quality management. Am J Med Qual 1999; 14:3–20
Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, Newhouse JP, Weiler PC, Hiatt HH: Incidence of adverse events and negligence in hospitalized patients: Results of the Harvard Medical Practice Study I. N Engl J Med 1991; 324:370–6
Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD: The Quality in Australian Health Care Study. Med J Aust 1995; 163:458–71
Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, Howard KM, Weiler PC, Brennan TA: Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000; 38:261–71
Brennan TA, Localio RJ, Laird NL: Reliability and validity of judgments concerning adverse events suffered by hospitalized patients. Med Care 1989; 27:1148–58
Cohen J: Quality improvement and peer review in anesthesiology, Manual for Anesthesia Department Organization and Management, edition 2003–2004. Edited by the American Society of Anesthesiologists. Park Ridge, Illinois, American Society of Anesthesiologists, 2003, pp 1–28the American Society of Anesthesiologists
Park Ridge, Illinois
American Society of Anesthesiologists
Thomson R, Taber S, Lally J, Kazandjian V: UK Quality Indicator Project (UK QIP) and the UK independent health care sector: A new development. Int J Qual Health Care 2004; 16(suppl 1):i51–6
Kazandjian VA, Thomson RG, Law WR, Waldron K: Do performance indicators make a difference? Jt Comm J Qual Improv 1996; 22:482–91
Gibberd R, Hancock S, Howley P, Richards K: Using indicators to quantify the potential to improve the quality of health care. Int J Qual Health Care 2004; 16(suppl 1):i37–43
Weeks AM: Draft clinical indicators in anaesthesia. Faculty of Anaesthetists Royal Australasian College of Surgeons Bulletin 1991; 4:28–9
Australian and New Zealand College of Anaesthetists: Safety of Anaesthesia in Australia: A Review of Anaesthesia Related Mortality 1997–1999. Edited by Mackay P. Melbourne, Australian and New Zealand College of Anaesthetists, 2002, pp 8–17
Australian and New Zealand College of Anaesthetists: Safety of Anaesthesia in Australia
Mackay P
Australian and New Zealand College of Anaesthetists
Haller G, Myles P, Wolfe R, Weeks T, Stoelwinder J, McNeil J: Validity of unplanned intensive care unit admission as an indicator of patient safety in surgical patients. Anesthesiology 2005; 103:1121–9
Gan TJ, Meyer T, Apfel CC, Chung F, Davis PJ, Eubanks S, Kovac A, Philip BK, Sessler DI, Temo J, Tramer MR, Watcha M: Consensus guidelines for managing postoperative nausea and vomiting. Anesth Analg 2003; 97:62–71
Negus B, Markovic S: A surgical antiemetic protocol-implementation and evaluation. Acute Pain 2003; 5:63–8
Geerts WH, Pineo GF, Heit JA, Bergqvist D, Lassen MR, Colwell CW, Ray JG: Prevention of venous thromboembolism: The Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004;126: 338S–400S
Hirsh J, Raschke R: Heparin and low-molecular-weight heparin: The Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126:188S–203S
Engelman R, Shahian D, Shemin R, Guy TS, Bratzler D, Edwards F, Jacobs M, Fernando H, Bridges C: The Society of Thoracic Surgeons practice guideline series: Antibiotic prophylaxis in cardiac surgery, part II: Antibiotic choice. Ann Thorac Surg 2007; 83:1569–76
Garcia-Miguel FJ, Serrano-Aguilar PG, Lopez-Bastida J: Preoperative assessment. Lancet 2003; 362:1749–57
Agency for Healthcare Research and Quality: Making Health Care Safer: A Critical Analysis of Patient Safety Practices—Evidence Report/Technology Assessment No. 43. AHRQ Publication No. 01-E058. Rockville, Agency for Healthcare Research and Quality, 2001, pp 214–53
Agency for Healthcare Research and Quality
Agency for Healthcare Research and Quality
Devereaux PJ, Beattie WS, Choi PT, Badner NH, Guyatt GH, Villar JC, Cinà CS, Leslie K, Jacka MJ, Montori VM, Bhandari M, Avezum A, Cavalcanti AB, Giles JW, Schricker T, Yang H, Jakobsen CJ, Yusuf S: How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. BMJ 2005; 331:313–21
Pierre S, Corno G, Benais H, Apfel CC: A risk score-dependent antiemetic approach effectively reduces postoperative nausea and vomiting: A continuous quality improvement initiative. Can J Anaesth 2004; 51:320–5
Apfel CC, Greim CA, Haubitz I, Goepfert C, Usadel J, Sefrin P, Roewer N: A risk score to predict the probability of postoperative vomiting in adults. Acta Anaesthesiol Scand 1998; 42:495–501
Apfel CC, Korttila K, Abdalla M, Kerger H, Turan A, Vedder I, Zernak C, Danner K, Jokela R, Pocock SJ, Trenkler S, Kredel M, Biedler A, Sessler DI, Roewer N, IMPACT Investigators: A factorial trial of six interventions for the prevention of postoperative nausea and vomiting. N Engl J Med 2004; 350:2441–51
IMPACT Investigators
Mainz J: Developing evidence-based clinical indicators: A state of the art methods primer. Int J Qual Health Care 2003;15(suppl 1):i5–11
Campbell SM, Braspenning J, Hutchinson A, Marshall M: Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 2002; 11:358–64
McNeil JJ, Ibrahim J, Majoor J, Cicuttini F: Performance and outcome measures. Med J Aust 1999; 170:507
Lilford R, Mohammed MA, Spiegelhalter D, Thomson R: Use and misuse of process and outcome data in managing performance of acute medical care: Avoiding institutional stigma. Lancet 2004; 363:1147–54
Bahtsevani C, Uden G, Willman A: Outcomes of evidence-based clinical practice guidelines: A systematic review. Int J Technol Assess Health Care 2004; 20:427–33
Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: A systematic review of rigorous evaluations. Lancet 1993; 342:1317–22
Bradley EH, Herrin J, Elbel B, McNamara RL, Magid DJ, Nallamothu BK, Wang Y, Normand SL, Spertus JA, Krumholz HM: Hospital quality for acute myocardial infarction: Correlation among process measures and relationship with short-term mortality. JAMA 2006; 296:72–8
Peterson ED, Roe MT, Mulgund J, DeLong ER, Lytle BL, Brindis RG, Smith SC Jr, Pollack CV Jr, Newby LK, Harrington RA, Gibler WB, Ohman EM: Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA 2006; 295:1912–20
Blake JB: An examination of some recent statistics in regard ether, and a consideration of some present methods of administration. Boston Med Surg J 1895; 132:590
Rubin HR, Pronovost P, Diette GB: The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care 2001; 13:469–74
Pitches D, Burls A, Fry-Smith A: How to make a silk purse from a sow’s ear: A comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians. BMJ 2003; 327:1436–9
Gascón AR, Campo E, Hernández RM, Calvo B, Errasti J, Pedraz JL: Pharmacokinetics and tissue distribution of intravenous ofloxacin for antibiotic prophylaxis in biliary surgery. Clin Drug Investig 1998; 15:491–6
Platt R, Muñoz A, Stella J, VanDevanter S, Koster JK Jr: Antibiotic prophylaxis for cardiovascular surgery: Efficacy with coronary artery bypass. Ann Intern Med 1984; 101:770–4
Schilling J, Michalopoulos A, Geroulanos S: Antibiotic prophylaxis in gastroduodenal surgery. Hepatogastroenterology 1997; 44:116–20
Cooper JB, Gaba DM, Liang B, Woods D, Blum LN: The National Patient Safety Foundation agenda for research and development in patient safety. Med Gen Med 2000; 2:E38
Liamputtong P, Ezzy D: Qualitative Research Methods: A Health Focus. Melbourne, Oxford University Press, 1999, pp 29–50
Oxford University Press
Myles PS, Weitkamp B, Jones K, Melick J, Hensen S: Validity and reliability of a postoperative quality of recovery score: The QoR-40. Br J Anaesth 2000; 84:11–5
Auquier P, Pernoud N, Bruder N, Simeoni MC, Auffray JP, Colavolpe C, François G, Gouin F, Manelli JC, Martin C, Sapin C, Blache JL: Development and validation of a perioperative satisfaction questionnaire. Anesthesiology 2005; 102:1116–23
Cheney FW: The American Society of Anesthesiologists Closed Claims Project: What have we learned, how has it affected practice, and how will it affect practice in the future? Anesthesiology 1999; 91:552–6