The call ... is twofold: create readily comprehensible, valid interpretations of anesthesia performance metrics … and develop meaningful metrics that can be tracked and improved.

Image: ©Thinkstock.

Image: ©Thinkstock.

“Try – not! Do! Or do not! There is no try!”

—Yoda

Anesthesiology has striven to improve its performance since its inception as a medical specialty, perhaps because the evidence of failure in our specialty’s infancy was often death or grievous injury. We are beyond the era when survival was considered evidence for success and high quality in most anesthesia subspecialties. The focus now is on value, seeking to maximize the ratio of quality over cost. How to measure quality? Especially, how do we measure quality attributable specifically to the anesthesia provider? In this issue of Anesthesiology, Bayman et al.1  take on the quality question, focusing on an outcome that might map cleanly to the anesthesia team’s decision-making (the time from end of surgery to extubation, hereinafter called time to extubation). This outcome is certainly visible to the eyes in the operating room, surgery center, and hospital administration trained upon “Anesthesia.”

The article by Bayman et al. contains several implicit warnings for anesthesiology as a field. First, beware of simplistic statistical approaches to comparing quality. Means, medians, and 95th centiles as well as multiple comparisons abound on dashboards of healthcare quality. Probability dictates that someone will be on the wrong side of the cutoff line, and simplistic approaches may sweep up clinicians whose practice is normative, but simplistic analyses without thoughtful follow-up identify them as outliers.2,3  Anesthesiology as a field can commend itself for bringing rigor to many operational measurements,4–7  where simplistic analyses initially lead to inappropriate conclusions. It is important to make these rigorous approaches intellectually accessible to administrators, policy makers, and bureaucrats. The gold standard of simplicity should be the mean, median, or target decile. Observed-to-expected ratios have arguably entered this class of readily understood metrics. Moving Bayesian analyses into the common lexicon of quality and value measurement is an important, unfinished task. Frequent, clear reinforcement is an important pedagogical objective.

Second, beware of simplistic indicators of quality. The authors provide explanations why 15-min time to extubation is the limit of acceptability, due to the cost of operating room time and the inconvenience to the surgical team, but so many extubation times exceeded 15 min that this cutoff seems debatable. Moreover, is time to extubation a metric that we want to use for valuing our specialty? In a patient-centered health system, we should attend to the outcomes that patients care about. Moreover, are anesthesiologists so subsidiary to every other part of the perioperative process that time waiting for emergence from anesthesia (arguably a pharmacologic–physiologic process that we indirectly control) is something that we should actively minimize any more than surgeons try to manage the time to accomplish the operation? We certainly should diligently work to minimize unnecessary anesthetic exposure and inefficient resource consumption, but should our perioperative colleagues be led to believe that anesthesia is something we should be able to turn on and off like water from a faucet? If this is true, are we sure there is a basis for a medical specialty built around anesthesia, or can it be turned over to technicians? (This rhetorical question is intended to be provocative.) It seems more likely (or more hopeful) that there are more meaningful outcomes of anesthesia to measure and improve than time to extubation. Bayman et al. deflect this simplistic measure by revealing its minimal discriminant power, but at the risk of demonstrating that yet another potential indicator of anesthesia quality actually has no utility.

A third warning is the unintended but distressing message arising from this work, along with several other recent publications, specifically, that few of the process-oriented quality or value measures anesthesiologists have focused on can be readily improved. Some elements of performance in control of the anesthesiologist can be improved (apparently),2,8–12  but more careful statistical analyses indicate that the changes are small at best at the individual provider level, and that there are few, if any, provider outliers,3  or that the observed clinical effects of performance variations are minimal.13  Anesthesiologists run the risk of painting a picture to regulators, policy makers, and payers of not being able or willing to improve anything.

Bayman et al. provide a more subtle message about quality and value improvement in anesthesiology that attending anesthesiologists alone are rarely the main drivers of the anesthetic outcome of interest, in this instance, time to extubation. Here is the important conclusion of this and many other studies of anesthesia process measures: Most cases were performed by two providers (at least), both the attending anesthesiologist and the in-room clinician, either a resident or a certified registered nurse anesthetist. The in-room provider exerts varying, but unmeasured control over decision-making about each case. Programs are training their residents to become autonomous, and certified registered nurse anesthetists enjoy limited, although unquantified, autonomy in the care team model. Therefore, the impact of attending anesthesiologists on such a shared outcome as time to extubation is likely to be muted.

This observation highlights an important constraint in the design of prospective whole-population studies of anesthesia outcomes or efforts to improve anesthesia performance. Because there are almost always two providers (at least in North American sites performing outcomes research), because the influence of each provider on the conduct of the case is substantial (probably larger than faculty like to admit), and because provider pairings change almost every day,* it is problematic to randomize individual clinicians when studying an intervention. The effect of intervention is likely to be modulated (likely downward, but possibly carried across to faculty randomized to the alternate group) by the participation of in-room clinicians who are not randomized and change daily. Similarly, randomizing both groups creates a 2 × 2 table (even before considering hand-offs for shift relief) in which half of the pair combinations are discordant with respect to randomization. Both possible randomization designs quickly expose the study allocations (and hence, probably the hypotheses) to the study participants, further degrading the methodologic strength of randomization.

What then is the solution? Studies of anesthetic interventions must be larger than single departments. For example, with a few departments, it is possible to construct a modified case control design with “case” departments making a before and after intervention with some departments serving as a control.11  Even better, with multiple departments or units, separated by enough geography to prevent exchange of personnel, but serving similar populations, it is possible to achieve cluster randomization in pragmatic trials where clinicians are aware of the study hypotheses.14,15  Given that these studies are likely to be very expensive, what should we study to measure and potentially improve anesthesia quality and value?

Recent literature provides some interesting examples: Evaluation of Nitrous Oxide in the Gas Mixture for Anaesthesia (ENIGMA II)16  and Perioperative Ischemic Evaluation (POISE) 117  and 218,19  are “classic” drug versus drug designs that led to surprising and interesting conclusions and raised as many questions as they answered. However, they are not designed to assess the impact of anesthesia efficiency or operational interventions. Anesthesiologists, as system-embedded physicians, practice in an environment where most interventions are linked to multiple outcomes, and vice versa, and where multiple actors influence outcomes simultaneously. Given the difficulty of separating even the attending anesthesiologist vis-a-vis in-room clinician dyadic impacts on study outcomes, it seems virtually impossible to design an incisive study of anesthesia interventions on relevant patient outcomes. One final complexity is the imperative to move to holistic functional outcome measures of medical interventions rather than process outcomes.20  This adds to the complexity of teasing out what contribution, if any, that anesthesiology performance makes to overall patient outcomes, although there have been recent attempts to do so.21 

It is incumbent on anesthesiologists to engage in the performance improvement and value maximization conversation, despite the difficulty in attributing the observed effects. If we do not, bureaucrats who understand simplistic analyses will define and analyze performance metrics for us. The call, then, is twofold: create readily comprehensible, valid interpretations of anesthesia performance metrics that, for better or worse, have gained traction, and develop meaningful metrics that can be tracked and improved.

Funding to find innovative approaches to defining, tracking, and analyzing meaningful quality metrics that map to anesthesia seems on its face to be difficult to find. In an apparent vacuum, departments, anesthesiology groups, payers, and regulators are self-funding grass roots efforts, which is commendable. The Patient-Centered Outcomes Research Institute and the Agency for Healthcare Research and Quality are potential Federal sources, and the American Society of Anesthesiologists itself is another potential source of substantial support. In the current resource-constrained environment, substantial, unrestricted funding is required to support dedicated, scientific efforts in search of important metrics and valid methods that focus on meaningful, patient-centered opportunities to measure performance and improve value. Challenging remains this question of funding, for now.

Acknowledgments

Support was provided solely from institutional and/or departmental sources.

Competing Interests

The author is not supported by, nor maintains any financial interest in, any commercial activity that may be associated with the topic of this article.

*

Creating fixed pairings of attending anesthesiologists and in-room clinicians, especially with trainees who must progress through a series of rotations over three Clinical Anesthesia training years, is a potential but impractical solution.

References

1.
Bayman
EO
,
Dexter
F
,
Todd
MM
:
Prolonged operative “time to extubation” is not a useful metric for comparing the performance of individual anesthesia providers.
Anesthesiology
2016
;
124
:
322
38
2.
Ehrenfeld
JM
,
Henneman
JP
,
Peterfreund
RA
,
Sheehan
TD
,
Xue
F
,
Spring
S
,
Sandberg
WS
:
Ongoing professional performance evaluation (OPPE) using automatically captured electronic anesthesia data.
Jt Comm J Qual Patient Saf
2012
;
38
:
73
80
3.
Bayman
EO
,
Dexter
F
,
Todd
MM
:
Assessing and comparing anesthesiologists’ performance on mandated metrics using a Bayesian approach.
Anesthesiology
2015
;
123
:
101
15
4.
Jacques
PS
,
France
DJ
,
Pilla
M
,
Lai
E
,
Higgins
MS
:
Evaluation of a hands-free wireless communication device in the perioperative environment.
Telemed J E Health
2006
;
12
:
42
9
5.
Ledolter
J
,
Dexter
F
,
Epstein
RH
:
Analysis of variance of communication latencies in anesthesia: Comparing means of multiple log-normal distributions.
Anesth Analg
2011
;
113
:
888
96
6.
Kynes
JM
,
Schildcrout
JS
,
Hickson
GB
,
Pichert
JW
,
Han
X
,
Ehrenfeld
JM
,
Westlake
MW
,
Catron
T
,
Jacques
PS
:
An analysis of risk factors for patient complaints about ambulatory anesthesiology care.
Anesth Analg
2013
;
116
:
1325
32
7.
Wanderer
JP
,
Shi
Y
,
Schildcrout
JS
,
Ehrenfeld
JM
,
Epstein
RH
:
Supervising anesthesiologists cannot be effectively compared according to their patients’ postanesthesia care unit admission pain scores.
Anesth Analg
2015
;
120
:
923
32
8.
Kheterpal
S
,
Gupta
R
,
Blum
JM
,
Tremper
KK
,
O’Reilly
M
,
Kazanjian
PE
:
Electronic reminders improve procedure documentation compliance and professional fee reimbursement.
Anesth Analg
2007
;
104
:
592
7
9.
Spring
SF
,
Sandberg
WS
,
Anupama
S
,
Walsh
JL
,
Driscoll
WD
,
Raines
DE
:
Automated documentation error detection and notification improves anesthesia billing performance.
Anesthesiology
2007
;
106
:
157
63
10.
Sandberg
WS
,
Sandberg
EH
,
Seim
AR
,
Anupama
S
,
Ehrenfeld
JM
,
Spring
SF
,
Walsh
JL
:
Real-time checking of electronic anesthesia records for documentation errors and automatically text messaging clinicians improves quality of documentation.
Anesth Analg
2008
;
106
:
192
201
11.
Ehrenfeld
JM
,
Epstein
RH
,
Bader
S
,
Kheterpal
S
,
Sandberg
WS
:
Automatic notifications mediated by anesthesia information management systems reduce the frequency of prolonged gaps in blood pressure documentation.
Anesth Analg
2011
;
113
:
356
63
12.
O’Reilly
M
,
Talsma
A
,
VanRiper
S
,
Kheterpal
S
,
Burney
R
:
An anesthesia information system designed to provide physician-specific feedback improves timely administration of prophylactic antibiotics.
Anesth Analg
2006
;
103
:
908
12
13.
Epstein
RH
,
Dexter
F
:
Mean arterial pressures bracketing prolonged monitoring interruptions have negligible systematic differences from matched controls without such gaps.
Anesth Analg
2011
;
113
:
267
71
14.
Huang
SS
,
Septimus
E
,
Kleinman
K
,
Moody
J
,
Hickok
J
,
Avery
TR
,
Lankiewicz
J
,
Gombosev
A
,
Terpstra
L
,
Hartford
F
,
Hayden
MK
,
Jernigan
JA
,
Weinstein
RA
,
Fraser
VJ
,
Haffenreffer
K
,
Cui
E
,
Kaganov
RE
,
Lolans
K
,
Perlin
JB
,
Platt
R
;
CDC Prevention Epicenters Program; AHRQ DECIDE Network and Healthcare-Associated Infections Program
:
Targeted versus universal decolonization to prevent ICU infection.
N Engl J Med
2013
;
368
:
2255
65
15.
Schweizer
ML
,
Chiang
HY
,
Septimus
E
,
Moody
J
,
Braun
B
,
Hafner
J
,
Ward
MA
,
Hickok
J
,
Perencevich
EN
,
Diekema
DJ
,
Richards
CL
,
Cavanaugh
JE
,
Perlin
JB
,
Herwaldt
LA
:
Association of a bundled intervention with surgical site infections among patients undergoing cardiac, hip, or knee surgery.
JAMA
2015
;
313
:
2162
71
16.
Myles
PS
,
Leslie
K
,
Chan
MT
,
Forbes
A
,
Peyton
PJ
,
Paech
MJ
,
Beattie
WS
,
Sessler
DI
,
Devereaux
PJ
,
Silbert
B
,
Schricker
T
,
Wallace
S
;
ANZCA Trials Group for the ENIGMA-II Investigators
:
The safety of addition of nitrous oxide to general anaesthesia in at-risk patients having major non-cardiac surgery (ENIGMA-II): A randomised, single-blind trial.
Lancet
2014
;
384
:
1446
54
17.
Devereaux
PJ
,
Yang
H
,
Yusuf
S
,
Guyatt
G
,
Leslie
K
,
Villar
JC
,
Xavier
D
,
Chrolavicius
S
,
Greenspan
L
,
Pogue
J
,
Pais
P
,
Liu
L
,
Xu
S
,
Málaga
G
,
Avezum
A
,
Chan
M
,
Montori
VM
,
Jacka
M
,
Choi
P
;
POISE Study Group
:
Effects of extended-release metoprolol succinate in patients undergoing non-cardiac surgery (POISE trial): A randomised controlled trial.
Lancet
2008
;
371
:
1839
47
18.
Devereaux
PJ
,
Mrkobrada
M
,
Sessler
DI
,
Leslie
K
,
Alonso-Coello
P
,
Kurz
A
,
Villar
JC
,
Sigamani
A
,
Biccard
BM
,
Meyhoff
CS
,
Parlow
JL
,
Guyatt
G
,
Robinson
A
,
Garg
AX
,
Rodseth
RN
,
Botto
F
,
Lurati Buse
G
,
Xavier
D
,
Chan
MT
,
Tiboni
M
,
Cook
D
,
Kumar
PA
,
Forget
P
,
Malaga
G
,
Fleischmann
E
,
Amir
M
,
Eikelboom
J
,
Mizera
R
,
Torres
D
,
Wang
CY
,
VanHelder
T
,
Paniagua
P
,
Berwanger
O
,
Srinathan
S
,
Graham
M
,
Pasin
L
,
Le Manach
Y
,
Gao
P
,
Pogue
J
,
Whitlock
R
,
Lamy
A
,
Kearon
C
,
Baigent
C
,
Chow
C
,
Pettit
S
,
Chrolavicius
S
,
Yusuf
S
;
POISE-2 Investigators
:
Aspirin in patients undergoing noncardiac surgery.
N Engl J Med
2014
;
370
:
1494
503
19.
Devereaux
PJ
,
Sessler
DI
,
Leslie
K
,
Kurz
A
,
Mrkobrada
M
,
Alonso-Coello
P
,
Villar
JC
,
Sigamani
A
,
Biccard
BM
,
Meyhoff
CS
,
Parlow
JL
,
Guyatt
G
,
Robinson
A
,
Garg
AX
,
Rodseth
RN
,
Botto
F
,
Lurati Buse
G
,
Xavier
D
,
Chan
MT
,
Tiboni
M
,
Cook
D
,
Kumar
PA
,
Forget
P
,
Malaga
G
,
Fleischmann
E
,
Amir
M
,
Eikelboom
J
,
Mizera
R
,
Torres
D
,
Wang
CY
,
Vanhelder
T
,
Paniagua
P
,
Berwanger
O
,
Srinathan
S
,
Graham
M
,
Pasin
L
,
Le Manach
Y
,
Gao
P
,
Pogue
J
,
Whitlock
R
,
Lamy
A
,
Kearon
C
,
Chow
C
,
Pettit
S
,
Chrolavicius
S
,
Yusuf
S
;
POISE-2 Investigators
:
Clonidine in patients undergoing noncardiac surgery.
N Engl J Med
2014
;
370
:
1504
13
20.
Shulman
MA
,
Myles
PS
,
Chan
MT
,
McIlroy
DR
,
Wallace
S
,
Ponsford
J
:
Measurement of disability-free survival after surgery.
Anesthesiology
2015
;
122
:
524
36
21.
Glance
LG
,
Kellermann
AL
,
Hannan
EL
,
Fleisher
LA
,
Eaton
MP
,
Dutton
RP
,
Lustik
SJ
,
Li
Y
,
Dick
AW
:
The impact of anesthesiologists on coronary artery bypass graft surgery outcomes.
Anesth Analg
2015
;
120
:
526
33