Detection of Intraoperative Incidents by Electronic Scanning of Computerized Anesthesia Records: Comparison with Voluntary Reporting

In 1993, a computerized anesthesia record-keeping system (CompuRecord, ARI, Pittsburgh, PA) was installed in all operating rooms of one of this institution's clinical sites (St. Luke's division). The system recorded physiologic data during anesthesia for all nonobstetric, surgical procedures for a period of 10 months (9/1/93-6/30/94). At the time data collection for this study began, the computerized record-keeping system had been in use for more than 3 months, and manual anesthesia record-keeping had been eliminated completely. Every 15 s, the record-keeping computer recorded the most recently measured values for systolic arterial pressure (SAP), heart rate (HR), arterial oxygen saturation measured by finger pulse oximetry (SpO²) and core body temperature (Temp) during all anesthetics (core temperature was not measured during some monitored anesthesia care (MAC) and regional anesthesia cases). Most SAP measurements were made using an automated noninvasive blood pressure cuff programmed to measure blood pressure every 3-5 min, whereas HR, SpO², and Temp were measured continuously.

After excluding the records of patients who underwent surgical procedures that required cardiopulmonary bypass, 5,454 records were retained for further analysis. For each of the 5,454 records, the patient's age, gender, American Society of Anesthesiologists (ASA) physical status, nature of the procedure, type of anesthetic, and attending anesthesiologist were noted.

The record-keeping system in each operating room was networked to a file server that stored the data for every case as a database file (Paradox, Borland International, Scotts Valley, CA) on an optical disk. One optical disk stored the data for all 5,454 cases. Using an analysis program provided with the CompuRecord system, the stored physiologic data were scanned electronically for intraoperative incidents, defined as deviations from specified limits for SAP, HR, SpO sub 2, and Temp (Table 1). For a deviation to be identified as an incident, values for SAP, HR, SpO², and Temp needed to be detected outside the specified range, continuously, without any interruption, for the entire specified time period.

When a record was identified as containing an intraoperative incident, the complete anesthesia record, including both the electronic and printed forms, was first examined independently, then together, by two of the investigators (KVS and JC). By consensus, they eliminated case records in which the detected incident was judged to be the result of artifact or in which the clinical context suggested that the incident was not relevant to the objectives of the study. The authors' own incidents were reviewed for context or artifact by a third anesthesiologist, who was neither an author nor a member of the CQI committee of our department. Cases excluded for clinical context were those in which the variables deviated outside of the specified limits, but were not considered relevant within the context of a CQI program. The exclusions were usually based on a patient's clinical condition or on an anesthesiologist's decision to accept the deviations (Table 2).

Subsequently, the investigators accessed a separate database that contained the reports that anesthesiologists completed at the end of each case to record untoward events. The report consisted of a list of questions related to the patient's clinical course while under anesthesia care. It had been in use in our department for several years and, before implementation of computerized record-keeping, had been printed on the back of the anesthesia record. With the introduction of automated record-keeping, the same set of questions was converted to an electronic format (Table 1). Anesthesiologists completed the report by entering a "yes" or "no" answer to questions displayed on a computer terminal located in the postanesthesia care unit. For all questions, the default answer was "no." The resulting reports were then sent to the record-keeping system's file server and stored on the same optical disk used to store physiologic data. Based on electronic verification, compliance with completing the computerized questionnaire was 100%. When an intraoperative incident was detected by electronic scanning, the anesthesiologist's answers to the questions listed in Table 1were compared with the results of the scan of physiologic data for deviations from defined limits.

In addition, the database that contained the 5,454 anesthesia records was compared with our institution's "in-hospital mortality" database. The hospital's mortality database was consulted for a time period that extended to December 31, 1994, 6 months beyond the anesthesia record-keeping database. For any patient that appeared in both databases, the complete medical record was reviewed. In-hospital mortality was classified into one of four categories: (A) Death in the operating room; (B) Death within 48 h of anesthesia (but not in the operating room); (C) Death during hospitalization in which the intraoperative incident occurred but > 48 h after anesthesia; (D) Death after discharge from hospitalization in which the intraoperative incident occurred and > 48 h after anesthesia. Only patients who died in our hospital between September 1, 1993, and December 31, 1994, could be identified in category D. The number of patients in this study who died outside the hospital or at another hospital is unknown.

After examination of the methods used in this study, the Institutional Review Board of St. Luke's-Roosevelt Hospital Center determined that these methods belonged to a category that did not require its review or approval.

An anesthesia record could contain more than one incident. Patients with more than one incident were only counted once in the analyses of patient characteristics, anesthesia technique, and mortality.

As discussed earlier, anesthesiologists could report or not report the various types of intraoperative incidents as independent selections. Therefore, to take type of incident into account, analyses of the reporting behavior of anesthesiologists were based on intraoperative incidents, regardless of the number of incidents per record. Using the same reasoning, the accuracy of the electronic scanning system to identify anesthesia records with intraoperative incidents was assessed by calculating its sensitivity and specificity rate regardless of the number of incidents per record.

The accuracy with which the electronic scanning program detected intraoperative incidents in the database of physiologic variables was assessed by reviewing 30 days in the period covered by this study. Each printed automated record in this randomly selected 10% sample (30 of 303 days) was examined by one of the authors for intraoperative incidents and artifacts. Sensitivity and specificity rates were calculated on the 470 records in this 10% sample. However, because it was not possible to analyze, in retrospect, whether the physiologic data recorded by the AIMS were always true representations of the actual quantities displayed on the physiologic monitors used in the operating room, the calculation of sensitivity and specificity rates was based on the assumption that all physiologic data were transferred faithfully from physiologic monitors to the computer database during the recording process in the operating room.

Because the variables in this study were categorical, the method of statistical inference was predominantly chi square analysis (chi squared). Bonferroni-adjusted P values were reported when multiple comparisons were made. The chi squared for trend (chi squared^trend) tested the significance of the association for variables that had ordered levels. When expected frequencies were low (< 5), the Fisher's exact test provided the P values. Unadjusted odds ratios (OR) were used to estimate the univariate relative risks for factors associated with intraoperative incidents; 95% confidence intervals (CI) were used as a measure of the statistical precision of each OR. To evaluate the independent contributions of the study variables to risk of intraoperative incidents, adjusted ORs and their 95% CIs were obtained from multivariate logistic regression. All statistical analyses were performed with the Statistical Package for the Social Sciences (SPSS for Windows) version 5.0.2 (Chicago, IL, 1993).

Electronic scanning of 5,454 anesthesia records identified 494 defined intraoperative incidents in 473 anesthesia records. Among the 473 anesthesia records, 455 (96.2%) contained a single intraoperative incident. Sixteen additional records contained two intraoperative incidents, one contained three intraoperative incidents and another contained four. After screening both the electronic and printed forms of the anesthesia record, 60 intraoperative incidents on 60 anesthesia records were eliminated. Incidents were eliminated due to artifact in 25 cases and due to clinical context suggesting irrelevance in 35 cases (Table 2). The remaining 434 incidents distributed by type of incident, patient, and case characteristics are shown in Table 3, Table 4, and Table 5.

Risk of an intraoperative incident was increased for patients undergoing emergency rather than elective surgery (OR = 1.4, 95% CI 1.1-1.8). As a patient's ASA physical status increased, the proportion of intraoperative incidents increased (chi squared^trend= 35.1, P <0.001). Male patients were at greater risk for intraoperative incidents than female patients (OR = 1.4, 95% CI 1.1-1.7). Intraoperative incidents were not more likely to occur among inpatients than among ambulatory patients.

The distribution of electronically detected intraoperative incidents by type of anesthesia is also shown in Table 4. Risk of intraoperative incident was increased for general anesthesia cases as compared with either regional or MAC cases (OR = 1.9, 95% CI 1.4-2.6; OR = 2.1, 95% CI 1.5-2.8; respectively). There was no difference in frequency of incidents between regional and MAC cases. Because children and adolescents (age < 20 yr) were rarely given regional anesthesia or MAC, this analysis was repeated for adults (age greater or equal to 20 yr). For adults, there were no differences in the risk of intraoperative incidents among the three anesthesia types.

The distribution of electronically detected intraoperative incidents by patient age category is presented in Table 5. Intraoperative incidents were more frequent among children and adolescents (age < 20 yr) than among adults (age greater or equal to 20 yr) (chi squared = 217.8, P < 0.001). Among adults, the frequency of incidents was greater for elderly patients (age greater or equal to 70 yr) than for younger adults (age 20-69 years) (chi squared = 39.3, P <0.001). Three trends (P values Bonferroni-adjusted) were noted: 1) high SAP incidents increased with increasing patient age (chi squared^trend= 108.7, P < 0.01), 2) high HR incidents decreased with increasing patient age (chi squared^trend= 282.5, P < 0.01), and 3) low SpO sub 2 incidents increased with increasing patient age (chi squared^trend= 13.8, P < 0.01). Because tachycardia is common among children and adolescents, the analysis for high HR incidents was repeated for adults (age greater or equal to 20 yr). Although attenuated, the trend of decreasing high HR incidents with increasing patient age remained (chi squared^trend= 13.4, P < 0.01).

A multivariate analysis largely confirmed the univariate analyses. In a logistic regression that only included patients older than 20 yr, the following factors were associated with increased risk of an intraoperative incident: emergency surgical status, age greater than 70 yr, and PS 3 or more. Adjusted odds ratios for these variables were 1.6 (95% CI 1.2-2.2), 1.8 (95% CI 1.3-2.4) and 2.6 (95% CI 2.0-3.6), respectively. General anesthesia was associated with a slightly increased risk of an intraoperative incident as compared with MAC (adjusted odds ratio = 1.4, 95% CI 1.0-2.0), whereas regional anesthesia was not associated with increased risk of an intraoperative incident as compared with MAC. Male gender, significantly associated with intraoperative incidents in the univariate analysis, was not a risk factor in the multivariate analysis. As in the univariate analysis, inpatient status remained noncontributory to risk of intraoperative incidents in the multivariate analysis.

When the 434 intraoperative incidents were checked for voluntary reporting, 18 (4.1%) matching voluntary reports were found. The distribution of intraoperative incidents by type of incident and their voluntary reporting rates are shown in Table 3. Electronically detected hypotension (SAP < 70 mmHg for at least 10 min) and hypothermia (Temp < 34 degrees Celsius for at least 30 min) were most frequently reported. The number of reported incidents was too small to conduct all pairwise comparisons of percent reported among the types of incidents. However, among the few electronically detected hypotension incidents, the percent reported (37.5%) appeared to be higher than that for the other types of incidents (2%-12%).

All of the intraoperative incidents that had been reported voluntarily also were detected by electronic scanning. Each printed automated record in a randomly selected 10% sample (30 of 303 days) was examined by one of the authors for intraoperative incidents and artifacts. No record in this subgroup contained more than one type of incident. Seven false positive records contained artifacts. In the single false negative record, the system overlooked a valid intraoperative incident (tachycardia) that had an age criterion (> 4 yr) because the patient's date of birth was missing from the AAR, but could be estimated by the examiner from other information on the printed record. Based on this randomly selected sample of records, the sensitivity rate of electronic scanning was 97.2% (35/36), and the specificity rate was 98.4% (427/434). This calculation assumes that all physiologic data were transferred faithfully from physiologic monitors to the computer database during the recording process in the operating room.

The proportion of electronically detected intraoperative incidents that was reported voluntarily was higher for inpatients than for ambulatory patients (chi squared = 7.8, P < 0.01), for emergency than for elective surgery (Fisher's exact P < 0.001) and for male than for female patients (chi squared = 5.1, P < 0.05) (Table 4).

Although few incidents were reported, ASA physical status appeared to be strongly associated with voluntary reporting by anesthesiologists. As shown in Table 4, less than 2% of the incidents detected in the AARs of PS 1 and 2 patients were reported. Compared with PS 1 patients, the proportion of reported incidents doubled for PS 3 patients, doubled again for PS 4 patients, and quadrupled for PS 5 patients.

The distribution of reporting by age category is also shown in Table 5. Due to small numbers in each category, all possible pairwise comparisons were not conducted. It does appear, however, that intraoperative incidents among those aged 30-39 yr were more likely to be reported than incidents in the other age categories.

Type of anesthesia (general, regional, and MAC) was not associated with voluntary reporting of intraoperative incidents (Table 4). The proportion of incidents reported varied widely among anesthesiologists. Although some anesthesiologists reported more incidents than others, none of the anesthesiologists reported more than 17% of their incidents. In addition, of the 18 anesthesiologists who had electronically detected intraoperative incidents, 11 (61%) reported none of their incidents.

Among 413 cases with electronically detected intraoperative incidents, there were 29 deaths (7.0%), compared with 79 deaths (1.6%) among the 5,041 cases without electronically detected intraoperative incidents (Table 6). The proportion of deaths among cases with intraoperative incidents was significantly higher than among cases without intraoperative incidents (chi squared = 58.85, P < 0.001). The distribution of deaths among the four categories for the timing of death is also shown in Table 6.

The findings of this study indicate that, using an AIMS, AAR with specific intraoperative incidents can be identified with high sensitivity and specificity rates. The study further shows that only a small fraction (4.1%) of those defined intraoperative incidents were voluntarily reported by anesthesiologists after anesthesia despite the fact that a questionnaire that elicited such reporting was completed after every case. These results support the assumptions of several authors concerning the usefulness of AAR and AIMS in CQI and risk management activities. [1-11]The study also illustrates some of the difficulties and limitations encountered when AAR and AIMS are used for such purposes.

Only four process variables were examined in this study. The clinical importance of SAP, HR, SpO², and Temp as study variables is obvious. Other reasons for selecting them were that: 1) the variables were measured continuously, or very frequently, during almost every anesthetic; 2) the measured variables were quantitative in nature and ideally suited for digital recording and analysis; 3) the variables were recorded with minimal input from the anesthesiologists; and 4) the accuracy of AIMS for recording such variables had been demonstrated previously. [5] 

After these four variables had been selected for analysis, significant difficulties were encountered in the definition of a normal range of values for each variable. Similar problems were noted by others. [12,13]Van Oostrom et al. [13]observed that even when residents were asked to state their own desirable ranges for each patient's vital signs, transgressions of these ranges were common. Although the chosen ranges were often quite generous, 305 deviations were counted in 50 cases, and not all transgressions were treated. We relied on the definitions of hypertension, hypotension, tachycardia, bradycardia, hypoxia, and hypothermia that had been used for several years by our departmental CQI program. The anesthesiologist members of the CQI committee chose these limits by consensus. In our study, the limits of normal selected for hypertension, hypotension, tachycardia, bradycardia, hypoxia, and hypothermia were more extreme than some of those published in the literature. [14,15]When the data were screened using more narrow ranges of normal, approximately 2/3 of all AARs were noted to have at least one intraoperative incident.

Berger et al. [12]suggested that, for blood pressure and HR, the limits of normal may be better defined by using average values of preoperative measurements. Average values of these variables were not measured routinely by the anesthesiologists participating in the current study and, therefore, were not available for analysis. In addition, one of the perceived advantages of computerized AIMS is that it allows one to rapidly scan a large database for deviations from predefined limits. If individualized limits were to be used, the definition of these individual limits would need to be part of the automated recording process. The computerized AIMS used in the current study was not programmed for that task.

Most analyses of critical intraoperative incidents include the examination of many more variables than the four studied in this investigation. [16-18]Whereas events such as cardiac arrhythmias, repeated attempts at intubation, mechanical failures, nerve injury, intraoperative awareness, failed regional block, bronchospasm, vomiting, and many others may be important to patient outcome, they are not readily detectable by electronic scanning of an AAR database. Many of these events are available for electronic scanning only if they have been recorded systematically by the anesthesiologist in a manner that allows their automated retrieval. If an incident is reported, however, it can be analyzed in great detail using the data stored in an AAR. A review of these data may contribute to an understanding of the pathophysiology and timing of the untoward event. The analysis can also point to elements of an incident that may be missed in a hand-written anesthesia record. [11] 

Several authors have expressed concern that AIMS will record artifacts as intraoperative incidents and that such artifacts will be taken as evidence of poor anesthetic management in the event of adverse outcome. [5,10]The type and frequency of artifacts recorded on AARs have been documented previously. [1]In our study, 25 (5%) of 494 intraoperative incidents that were detected electronically were identified as artifacts of monitoring when the AAR was visually inspected by two senior anesthesiologists. This suggests that current technology is not totally accurate in avoiding the recording of artifacts or in discerning them once they have been recorded. It also suggests that visual inspection of the record allows one to differentiate artifacts from true deviations in physiologic variables. Advances in software, like artificial intelligence, and advances in monitoring technology may reduce problems with artifacts in the near future. [5]Similarly, although an anesthesiologist may, based on a patient's condition or other considerations, need to accept deviations of physiologic variables beyond specified limits, such deviations will, nonetheless, be identified as intraoperative incidents during electronic scanning. Our review of AARs that contained electronically detected intraoperative incidents revealed that 35 intraoperative incidents occurred in a clinical context that would render the incident irrelevant to members of a CQI committee assigned to review such incidents.

Other authors have documented that there are sizable differences between the values measured by automatic devices and values recorded in hand-kept anesthesia records. [14,19,20]In 50 patients, Cook et al. [19]found significant differences between automatic and handwritten maximum and minimum recorded blood pressures. They attributed the discrepancies to faulty reconstruction from memory and to bias in favor of less controversial values. In a similar comparison, Lerou et al. [20]noted a high incidence of missing or erroneous blood pressure values in the manual record. The AAR does not suffer from the "smoothing" of vital sign data that tends to occur with hand-kept records. [14]During critical events, anesthesiologists may be too busy to record vital sign data accurately. After resolution of the critical event, data recorded by hand are likely to be inaccurate and may reflect the anesthesiologist's concern about the consequences of abnormal vital sign data.

In the future, the renewal of credentials may require a review of an anesthesiologist's performance, including measures of process and outcome. If electronic scanning of AARs is part of this review, it is essential that intraoperative incidents that are insignificant due to artifact or context be excluded. Our study shows that, through visual inspection, some of these events can be identified. However, their recognition is greatly facilitated when the anesthesiologist incorporates descriptive comments in the AAR.

The patients anesthetized during the study period were primarily from a poor, inner-city population. When compared with other large populations in which intraoperative incidents were studied, our population was characterized by high ratios of PS 3-5 versus PS 1-2 (ratio = 0.269), emergency versus elective surgery (ratio = 0.176), elderly (age > 70 yr) versus younger patients (ratio = 0.183), and females versus males (ratio = 0.569). [21]Despite the high-risk characteristics of the population, the number of intraoperative incidents detected electronically was low. In Cohen et al.'s [22]study, intraoperative hypotension was more frequent than hypertension, whereas in the current study, the reverse was true. In addition, the frequency of both of these incidents was lower in our study. Our study also found a lower incidence of arterial desaturation than other studies. [23,24]These discrepancies can be due to many factors, including differences in inclusion criteria, patient populations, anesthetic techniques, and many others. To what extent the electronic recording of physiologic variables and the automatic selection of incidents contributed to these discrepancies is unknown. It should be emphasized that, in the current study, the definition of intraoperative incidents mandated that a deviation from limits be continuously present for a specified time period. Therefore, brief and transient events that may have been counted in other studies were unlikely to be labeled as intraoperative incidents by our analysis. The duration of physiologic disturbances is seldom mentioned in other studies.

Although there was no difference in the frequency of intraoperative incidents between PS 1 and 2 patients, there was a significant trend of increasing frequency of intraoperative incidents with advancing PS from 3 to 4 (chi squared^trend= 35.1, P < 0.001). ASA physical status also was associated with the incidence of perioperative death.

Tachycardia was the most commonly observed incident. It was particularly frequent in patients younger than age 20 yr, and one wonders whether its inclusion as an intraoperative incident in our CQI questionnaire was meaningful for this age group. There is little evidence to suggest that tachycardia is harmful in children or adolescents.

The level of compliance with voluntary self-reporting by the anesthesiologists included in the study was strikingly low. Reporting an intraoperative incident required that the anesthesiologist spend only 1-3 min at the keyboard of the postanesthesia care unit computer dedicated to this purpose. A questionnaire was completed in all cases, ruling out inconvenience of location or difficulty with operating the reporting computer. The default answer on the screen for each question about intraoperative incidents was "no," which required positive action by the anesthesiologist to change the answer to "yes," thereby reporting the incident. There was no history in our department of disciplinary action, financial harm, or political embarrassment related to voluntary (or involuntary) reporting of intraoperative incidents. During the period of the study, anesthesiologists were aware that their reports were used for the department's CQI program. They also were aware that all physiologic variables and voluntary reports of intraoperative incidents were recorded permanently by the AIMS. However, they were not aware that a study would be performed to electronically analyze intraoperative incidents and voluntary reporting.

Possible explanations for this low compliance in voluntary reporting include "human factors" that lead to minimizing the importance of reporting an event or fear of consequences of reporting one's own shortcomings. [25]Concerns about the use of a checklist for self-reporting have been raised by other investigators. [26]In other fields, several studies demonstrated that physicians tend to be poorly compliant with published recommendations. [27-32]** In a study of house officers at a major teaching hospital, only 1.5% of adverse events that occurred on a medical service resulted in a written incident report, but the rate of voluntary reporting of the same adverse events by electronic mail ranged from 32% to 62.5%, depending on the type of adverse event. [33]In a well controlled study, Cullen et al.*** reported that, of 54 adverse drug events found by independent nurse-investigators, only 3 (6%) resulted in the filing of an incident report by either the unit personnel or the pharmacy. [34]Headrick et al. [29]also noted that a physician's own estimate of his or her compliance with voluntary reporting was much greater than actual compliance.

One can speculate on the factors that influence an anesthesiologist's self-reporting behavior. Hypotension and hypothermia were more likely to be reported than other incidents, suggesting that the anesthesiologists in this study may have considered those deviations more important than others. Why the intraoperative incidents of male patients, inpatients, high-risk patients and those having emergency surgery were reported more frequently than their counterparts is also unexplained. Perhaps anesthesiologists have greater expectation of incidents in those patients and, therefore, are less reluctant to report incidents. Other explanations may include enhanced time pressures when caring for outpatients, or a perception that outpatient surgery is less likely to be associated with poor outcome. Both male and female anesthesiologists tended to report the intraoperative incidents of male patients more frequently than those of female patients. There was no significant difference among anesthesiologists in the overall frequency of reporting intraoperative incidents based on their emergency caseload, level of American Board of Anesthesiology certification, their gender, or their seniority status.

When this study was completed, the results were made available to our department's CQI committee and then to the entire staff. Our CQI committee now performs a monthly electronic scan for the six intraoperative incidents analyzed in this study and compares the results of the scan to the voluntary reports. A followup study is planned to determine whether our staff's rate of voluntary reporting of intraoperative incidents has improved since the announcement of the results of this study.

The current study was not designed as an outcome study, but the availability of an inhospital mortality database encouraged us to explore the association between intraoperative incidents and inhospital mortality. A highly significant association was observed between the occurrence of electronically detected intraoperative incidents and mortality. Although these findings do not point to a cause-and-effect relation between anesthetic management and mortality, they nonetheless indicate that process-related measurements may provide a link to patient outcome. Previous studies attempting to relate intraoperative hemodynamic or respiratory incidents to outcome were hampered by the small numbers of cases in which actual measurements of physiologic variables could be recorded accurately [35]or were flawed by reliance on hand-written anesthesia records, which are notoriously inaccurate. [10,35] 

Using a computerized AIMS, six types of quantitative intraoperative incidents were identified in a large group of patients. Electronic scanning detected these incidents with high sensitivity and specificity rates. Only a small percentage of these intraoperative incidents were voluntarily self-reported by the responsible anesthesiologist. There was a strong association between intraoperative incidents and in-hospital mortality. The handling of artifact and clinical context presented a problem in the use of the AIMS for the detection of intraoperative incidents. Despite that problem, the study demonstrates that a computerized AIMS can measure certain anesthesia process variables with great accuracy.

The authors thank undergraduate students Kristin Sanborn, Natan Khishenko, and Dave Ham, for their help with organizing patient records.

* Improving organizational performance, 1995 Comprehensive Accreditation Manual for Hospitals. Oakbrook Terrace, Illinois, Joint Commission on Accreditation of Healthcare Organizations, 1994, pp 219-66.

** Dajczman E, Dascal A, Orenstein P, Frank H: Survey of infection control precautions: A comparison to recommended guidelines. Can J Infect Control 1992; 7:7-12.

*** Cullen D, Bates D, Small S, Cooper J, Nemeskal A, Leape L: The incident reporting system does not detect adverse drug events: A problem for quality improvement. Journal on Quality Improvement 1995; 21:541-8.