Validity of Unplanned Admission to an Intensive Care Unit as a Measure of Patient Safety in Surgical Patients

Click on the links below to access all the ArticlePlus for this article.

Please note that ArticlePlus files may launch a viewer application outside of your web browser.

• https://links.lww.com/ALN/A372

• https://links.lww.com/ALN/A373

THE risks associated with anesthetic practice have long been recognized. Monitoring adverse outcomes, particularly anesthetic-related mortality and morbidity, has become a widely used method to guide patient safety improvement initiatives.2,,3,,4,,5,,6,,7 Because these outcomes provide little information about the process of care, a number of parallel measurement tools, largely borrowed from other industry settings, have been introduced in anesthesia and surgery: critical incident analysis techniques, incident reporting, and organizational safety culture assessment.8,,9,,10,,11,,12 Based on qualitative methodologies, these tools are designed to provide a more comprehensive understanding of critical patient safety issues and related work or organizational practices. However, because of their intrinsic qualitative nature, they cannot be used to quantify these patient safety issues or measure their variation over time. For example, measuring anesthesia-related adverse events or the benefits of a new monitoring device by counting incidents reported is unreliable. Voluntary incident reporting systems have been shown to undercount incidents by 77–94%.13,,14 They can be significantly biased by the reporters’ perceptions and expectations of the system.15,,16 There is a strong need for more quantifiable measures of adverse events and patient safety. To date, no such measure has been validated in anesthesia.

Indicators offer interesting perspectives. Well-known in social sciences, they are used to measure directly or indirectly noncountable concepts, such as intelligence. When these indicators stand reliably for the concept they represent, they can become a direct quantifiable measure of the concept.

Clinical indicators have recently been introduced in anesthesia by accreditation bodies and quality initiatives.18,,19 Under the influence of industrial engineering theory, they have mainly been used to measure abnormal variations in patient care to guide further peer review analysis.21,,22 However, it has never been demonstrated that they could directly stand for a noncountable concept such as patient safety and become, as a consequence, a reliable quantifiable measure of it. For such purpose, it would need to be shown that an indicator strongly relates to patient safety and that this can be formally validated.

We chose to assess an unplanned admission to the intensive care unit (ICU) within 24 h of a procedure with an anesthesiologist in attendance (UIA). This indicator is related to the early generic occurrence tools used in large programs and studies such as the Harvard medical practice study to uncover adverse events and medical errors.23,,24 It has been refined to measure unexpected adverse events occurring during the perioperative period. Therefore, it has great appeal as a patient safety indicator in surgical patients.

To demonstrate the validity of UIA as a measure of adverse events and patient safety, we chose to use a methodology developed in social sciences: construct validation. This method is based on the demonstration that there is a significant convergence between a new measurement tool and previously validated measures of the same attribute (construct). A good example is the assessment of the convergence between the Glasgow coma score and the cerebral metabolic rate or visual evoked potentials to determine the validity of the score as a measure of brain damage and coma.

We used intraoperative incidents/near misses, risk-adjusted mortality and risk-adjusted length of hospital stay (LOS) as constructs of adverse events and patient safety.27,,28 

Our hypothesis was that if UIA is a valid measure of patient safety in surgical patients, it would be significantly associated with intraoperative incidents, increased hospital LOS, and mortality. Therefore, the study purpose was to provide support for the construct validity of UIA as a measure of patient safety in surgical patients.

The Alfred Hospital (Melbourne, Victoria, Australia) is an adult university-affiliated hospital with all types of specialties. As part of an ongoing quality assurance (QA) program starting in 1993, all data of patients who undergo a medical or surgical invasive procedure are prospectively collected and recorded on a Microsoft Access database (Microsoft Corp., Seattle, WA). Medical and QA staff are instructed in the collection of data and provided with written instructions and item definitions.

Recorded information includes patient demographic characteristics, medical history, current health status, medication usage, and American Society of Anesthesiologists (ASA) physical status classification. Details of the procedures, including timing, emergency status, and associated incidents are also measured. A follow-up at 24 h is routinely performed by the QA coordinator or an anesthesiology resident. Information collected includes complications and UIA.

After institutional (Alfred Hospital) and university (Monash University, Melbourne, Victoria, Australia) ethics approval, we retrieved all procedure data collected between October 1995 and December 2000. After reorganizing and processing data, we performed a prospective cohort study. We excluded all patients who were routinely scheduled at our institution for postoperative admission to the ICU, such as cardiac surgery and lung transplantation, and those for which emergency circumstances did not allow the preoperative booking of an ICU bed (e.g. , patients with a ruptured aorta, with multitrauma, or with an ASA V E status admitted after hours).

We compared patients with and without UIA. To be considered valid, UIA patients had to be patients who were not scheduled for postoperative admission to the ICU at the beginning of the procedure. This had to be confirmed during the 24-h follow-up period by the QA coordinator or anesthesiology resident. To be considered valid for the study, the patient’s unplanned admission status to the ICU had to be also subsequently confirmed by a second data source, either the ICU database or medical records.

Incidents were defined as an unintended event or outcome which could have reduced or did reduce the safety margin for the patient. They were recorded using a predetermined list of 44 items routinely completed during the procedure by the anesthesiologist in attendance. This list resulted from a consensus of expert opinion in the department on common intraoperative incidents. We reviewed items before the beginning of the study and reclassified into near misses those that implied for patients acts of commission or omission that could have harmed but did not cause harm, as a result of chance, prevention, or mitigation.

Length of stay was measured in days, from admission date to discharge date. Same-day procedures such as respiratory or digestive diagnostic endoscopies were assigned a 0.5-day value. A number of patients were identified as having more than one procedure performed during their overall LOS. Because the first procedure is most often the reason why the patient was admitted to the hospital and subsequent interventions are often those performed to complete the initial procedure or to deal with consecutive complications, the overall value for LOS was attributed to the first procedure performed and classified according to the International Classification of Disease , 10th revision, Australian modification (ICD-10-AM).

Between 1995 and 2000, data were collected on a scannable form. They were systematically checked for coherence and completeness by the QA coordinator or a senior staff member before being entered into the database. Data consistency was also ensured by consistency filters and plausibility rules implemented in the computerized system.

Before the analysis, we checked files for double entries and illogical values, using specific structured query language clauses. When identified, these were corrected using a separate ICU electronic patient record, the hospital administrative database, or direct referral to medical charts. We further transformed the files into a statistically analyzable spreadsheet using Stat/transfer (version 7.01; Circles Systems Inc., Seattle, WA) and performed a random cross-check with the initial Access (version 2003; Microsoft Inc., Redmond, WA) files. We finally tested and corrected the spreadsheet for missing or out-of-range values. To confirm the validity of the incident measurement system, a systematic cross-check with medical charts was performed for all patients with an unscheduled admission to the ICU. This was performed by three independent assessors. The computerized measurement system was found to have a sensitivity to detect incidents recorded in medical charts of 81.4% and a specificity of 90.6%.

We recoded and aggregated comorbidities and Australian Medicare procedures into the ICD-10-AM international standardized classification system for diseases and procedures. To decrease the risk of transcription and interpretation errors, procedures were recoded electronically, using validated tables of equivalence. Of the 19 initial procedure categories, we combined 9 that were closely related either anatomically or clinically, resulting in 14 distinct blocks. This was done to increase power in the stratified statistical analyses. For the purpose of clarity, we also collapsed the 44-item list of intraoperative incidents and near misses into 18 broader categories. Incidents were classified according to the injury mechanism or the organ involved.

The ICU electronic patient record was used to confirm the status of all patients with a UIA during the study period. In cases of missing or unclear information in the ICU information system, a direct medical chart review was performed. An admission chart to the ICU and evidence of a scheduled ward before the procedure were considered as proof of UIA status.

Length of stay and mortality data were abstracted from the hospital central administrative database. Because this data source is used for funding purposes, its accuracy is ensured by a close collaboration between the department of informatics and the medical documentation unit, and so we limited our cross-checking with medical records to out-of-range values.

A unique medical record number assigned to each patient on hospital admission, with the patient’s name and date of birth, allowed accurate cross-checking of the different data sources.

We performed three separate analyses, measuring the association between UIA and (1) intraoperative incidents/near misses, (2) 30-day mortality, and (3) LOS. We used the STATA statistical software (version 8.2 SE 2003; StataCorp, College Station, TX). Patient age was grouped into equal tertiles, 10–40 yr, 41–64 yr, and older than 64 yr.

We divided the time of procedure into three separate categories: in-hours procedures (7:00 am to 6:59 pm), after-hours procedures (7:00 pm to 6:59 am), and late-hours procedures (starting in hours but finishing after 6:59 pm). We chose the ASA physical status classification as a surrogate comorbidity score because it is routinely used and validated as a predictor of postoperative complications, increased LOS, and mortality.

In analysis 1, we first determined univariate significant risk factors for UIA. For categorical variables, we used chi-square, binary logistic regression, and odds ratio (OR) with 95% confidence interval (CI). After testing for linearity, we reduced the five categories of the ASA physical status into three compiled groups (cASA). After testing ICD-10-AM procedure categories for linearity and homogeneity, we generated four homogeneous categories of surgical procedures, according to their respective power to predict an unplanned ICU admission in the univariate analysis. All multivariate regression analyses were stratified according to these different categories, except procedures on the eye, procedures on the mastoid process, and miscellaneous procedures that were not analyzed, because no UIA occurred.

To control for confounding factors, we also adjusted all analyses for patient age, sex, comorbidities, emergency status, and type and timing of procedure. For comorbidities, we tested two alternative approaches. In one, we adjusted for a single comorbidity index, the cASA score; in the other, we incorporated each ICD-10-AM disease category that was found significant in the univariate analysis into the multivariate stratified model.

In both approaches, we built multivariate models using a forward selection technique, considering only univariate risk factors with a P  value less than 0.10 and an expected count of at least 5 in the contingency tables. Variables that maintained significance at P  < 0.05 or that had a strong clinical significance (age, sex, emergency status) were retained. Because using the cASA score of comorbidity showed relatively similar performances to using individual diagnoses of comorbidities, we chose to keep the cASA score to adjust all analyses. The significance of the Hosmer-Lemeshow goodness-of-fit test ranged between 0.75 and 0.96. C indices were all between 0.72 and 0.89 across different analyses. Final results are expressed as unadjusted and adjusted or 95% CI and P  values.

For analysis 2, we determined risk factors for 30-day mortality using the same approach as for analysis 1, except that the original ASA physical status classification instead of the compiled score was used.

In analysis 3, we summarized LOS using median and interquartile range. We used log-rank tests, Kaplan-Meier plots, and Cox proportional hazards regression models to compare time to discharge among patient groups. Patients were censored at time of death or at December 2000, if discharged after the end of our data collection period.

We used two different tests of proportionality of hazards and subsequently stratified our analysis by groups of homogeneous surgical procedures, with or without an additional procedure performed during the patient’s stay. Groups were similar to those described in the first two analyses, except for procedures on the musculoskeletal and digestive systems, which were analyzed separately.

As for previous analyses, we used a forward selection technique to enter variables into the proportional hazards model and chose the cASA score as a measure of comorbidity. Results are expressed as adjusted hazards ratio with 95% CI.

During the study period, 48,467 patients had perioperative data recorded. We excluded 4,336 (9%) of them according to our study criteria and a further 659 (0.02%) without information on type of procedure, comorbidities, and identification number. The final study cohort consisted on 44,130 patients. Demographic characteristics, type and duration of procedures, ASA physical status, and comorbidities are described in .

There were 201 patients with a UIA. Univariate risk factors for UIA are reported in . Male patients older than 64 yr and with multiple comorbidities were at increased risk of UIA. Procedures on the nervous, cardiovascular, and respiratory systems, as well as prolonged, after-hours, and emergency procedures were significant risk factors.

One hundred five patients (52.2%) with a UIA had an incident or near miss during their procedure (). In nearly half (42%) of the cases, there was more than one event reported. Patients with a UIA had a significant increased risk of intraoperative incident or near misses (OR 7.94; 95% CI, 5.90–10.71; P  < 0.001).

After adjusting for patient age, sex, comorbidities, emergency status, and type and timing of surgery, the association between UIA and intraoperative incidents/near misses remained significant (). This was particularly apparent in endocrine system; ear, nose, and throat; female genital organs and breast; musculoskeletal system; urinary system and male genital organs; and dermatologic and plastic procedures (category 1), where patients with a UIA had a 12-fold increased risk of an incident and near miss to have occurred during the procedure (OR 12.21; 95% CI, 6.33–23.58; P  < 0.001).

For the purpose of clarity and brevity, we report only final results of the multivariate analyzes of 30-day hospital mortality and LOS. Additional information regarding univariate analyzes are available on the Anesthesiology Web site at http://www.anesthesiology.org (Web tables 1 and 2).

The overall 30-day mortality was 1.2% (n = 529). In the UIA group, the proportion was significantly higher, 10.9% (n = 22), than in the non-UIA group, 1.1% (n = 507) (). The unadjusted association with 30-day mortality for patients with UIA was OR 10.47 (95% CI, 6.34–16.51; P  < 0.001). After correcting the 30-day mortality risk for age, sex, comorbidities, and other confounding factors, the association was still clearly statistically significant in procedures on blood and blood-forming organs and procedures on the cardiovascular, digestive, and respiratory system, with an OR of 3.89 (95% CI, 2.14–7.04; P  < 0.001) ().

The overall median LOS for the study cohort was 2 days (interquartile range, 0.5–9) (). When UIA occurred, the LOS increased to a median of 16 days (interquartile range, 10–31; P  < 0.001).

After adjusting for confounding factors, patients with a UIA had a significant increase in LOS or decrease in chance of hospital discharge in nearly all types of surgery (). When an additional procedure was performed during the same hospital stay, the increase was still significant in several categories of procedures.

This study provides substantial support for the construct validity of UIA as a measure of patient safety in surgical patients. Developed by social scientists to validate measures of complex phenomenon such as intelligence, satisfaction, or pain, construct validation is based on the demonstration that there is a convergence between the new measure and previously validated measures of the same attribute (construct). In this study, we used intraoperative incidents, risk-adjusted mortality, and risk-adjusted LOS as constructs (or markers) of adverse events and patient safety. We performed three different analyses comparing UIA and non-UIA patients, and we adjusted for patients’ comorbidities, age, sex, emergency status, and type and timing of surgery.

After risk adjustment, we found that UIA was associated with a 2- to 12-fold increased likelihood of incidents occurring during anesthesia, a 2- to 3-fold increased risk of 30-day mortality, and a substantial increase in hospital LOS. The reproducibility of the pattern within these different constructs strongly supports the validity of UIA as a measure of patient safety.

Although UIA is available in different countries as part of hospital accreditation processes, it has never been formally assessed as an indicator to measure patient safety. However, there are a number of studies that report the use of ICU admissions as a screen to detect adverse events. In one, authors found that a transfer from a general ward to the ICU was associated with adverse events in 3.1% of cases. In two early publications, authors identified intraoperative complications as the main reason for ICU care in 2% and 5.7% of cases.38,,39 

More recently, it was found that 23% of patients with an unplanned admission or readmission to the ICU had an adverse event. Finally, Swann et al.  found that of 34 unplanned ICU admissions immediately after surgery, 16 (47%) were predictable and 7 (20%) were preventable.

Others have expressed reservations about the validity and usefulness of reviewing ICU admissions. In one study, the authors found no clear safety concern among patients with an unplanned transfer to the ICU from the operating room. In another, transfer from a general care to a special care unit was considered as ineffective because it uncovered only 16% of potentially compensable events.

These discrepant findings can be explained in several ways. First, methodologies varied widely between studies. Definitions of unplanned transfers to the ICU combined a broad range of conditions, and none applied the restrictive definition we used in this study. Second, such studies lacked a contemporaneous control group of patients not admitted to the ICU and so had limited ability to test the true effectiveness of UIA as an indicator of patient safety. Third, to uncover safety issues, they relied on a peer review process of selected cases of ICU admissions. For many years, peer review has been considered as the accepted standard to measure injuries and complications caused by healthcare management. However, there is increasing evidence to show that reviewers’ opinions regarding the occurrence of an adverse event and the appropriateness of care can be significantly biased, particularly when the outcome is known.44,,45 

Furthermore, the level of agreement between reviewers is also often only slightly better than chance (κ values from 0.2 to 0.6). Finally, most limited their definition to compensable adverse events.

These limitations were addressed in our study because we analyzed a large cohort of patients routinely admitted to a university-affiliated hospital and largely reflecting everyday clinical practice. To avoid reviewers’ bias and to measure a complex phenomenon such as patient safety, we used a validated methodology developed by psychologists and social scientists. In a rigorous statistical analysis, we assessed our new measure of patient safety, UIA, against three different constructs of patient safety: intraoperative incidents, risk-adjusted mortality, and LOS. The use of three different constructs allowed us to minimize biases that would stem from the use of one single construct.

This is related to the concept of triangulation, widely used in qualitative research and also familiar to anesthesiologists. The concluding clinical evidence that a patient is significantly bleeding intraoperatively is often obtained by using the convergence of related information such as severe hypotension, tachycardia, and an increasing amount of blood suctioned in the collecting bag.

This approach was strengthened by the fact that we were able to adjust our analyses for the confounding effects of comorbid conditions, age, sex, emergency status, and timing and type of procedure, which could be by themselves independent predictors of an unscheduled admission to the ICU.

However, a number of limitations of our study should be considered. Data collected during routine practice may not be as accurate as that seen in clinical trials. They can only be double-checked after patients have left the hospital, relying mainly on medical records.

Incidents were identified and recorded by staff members directly involved in the care of the patient, and the sensitivity of this incident measurement system in UIA patients was 81.4%. Furthermore, the definition of UIA includes unplanned transfers within 24 h, but only incidents occurring during the procedure were systematically collected and recorded. A number of incidents occurring in other hospital locations (postanesthesia care unit, surgical wards) could not be formally assessed. As a consequence, a significant number of patients with a UIA had no recorded incident occurring as a reason for their admission to the ICU.

If this affected the power of the study to demonstrate a difference between UIA and non-UIA patients, it had no impact on the validity of the study results, because a control group was used with intraoperative incidents measured the same way at the same time.

A number of these unplanned admissions could also reflect miscommunication or organizational problems in our hospital rather than safety issues. To avoid these confounding factors and increase the sensitivity of the indicator, we excluded all patients who were routinely scheduled at our institution for postoperative admission to the ICU and those for whom emergency circumstances did not allow preoperative booking of an ICU bed. Therefore, there is strong evidence to suggest that the remaining unanticipated admissions truly reflected adverse events rather than other factors.

The association between UIA and 30-day mortality and LOS was not statistically significant in some of the procedures (categories 1 and 3). This could be due to a lack of power resulting from the stratification process, particularly because 30-day mortality was low in this category of procedures. Furthermore, the strength of the association varied across surgical strata and is lower for procedures on the nervous system. This is due to the method we used to account for the influence of surgical risk. We stratified multivariate models by homogeneous groups of surgical procedures to account for the impact of surgical risk itself. The difference between strata reflects this impact. This is why there is, for example, a reduction after risk adjustment of the OR for intraoperative incidents occurring during neurosurgical procedures compared with plastic procedures. Because of the nature of the surgery or patient comorbidity in neurosurgical procedures, the occurrence of an intraoperative incident as a reason for an unscheduled admission to the ICU is lower. Therefore, the value of UIA as a measure of patient safety should be considered with more caution in patients undergoing procedures on the nervous system.

Finally, the total sum of intraoperative incidents was considered as a unique variable in the analysis. This method did not allow us to determine what type of intraoperative incidents would be more likely to be the reason for the admission to the ICU. Therefore, minor events such as technical failures or a temporarily disconnected intravenous line seemed to account for an unplanned ICU admission. However, these near misses and minor events were systematically concomitant with more major events such as complications after insertion of a central venous line or uncontrolled hypotension (). This is in accord with previous observational analyses of surgical procedures and performance that showed that successive minor and mitigated events are often precursors of major events and can be by themselves strong predictors of major adverse outcomes.

Incidents such as hypotension could be also due to surgical factors, such as uncontrolled bleeding. This reflects the difficulty in separating events related to anesthetic care from those related to surgical management. Therefore, UIA should be considered more as a global measure of patient safety in surgical patients than as an exclusive measure of patient safety in anesthesia.

Because a UIA is a relatively rare event (201 cases recorded in our hospital between October 1995 and December 2000), it should not be considered as a unique measure of patient safety. Efforts should also be directed toward the development of additional indicators and the improvement of existing tools, such as voluntary incident reporting systems or computer-based analysis systems.

Compared with traditional measures such as adverse outcomes, UIA is highly specific to the safety dimension of patient care. Therefore, it does not depend on the use of complex risk-adjusted models or peer review committees to determine whether an adverse outcome is related to the patient’s condition or to a patient’s safety issue. It can be used in hospitals that do not record data on extensive information systems, because the information can be extracted from medical charts, if necessary. It can provide hospitals and departments with useful information about the variability of the safety dimension of patient care, on a quarterly or an annual basis. For this, the exact process leading to an unplanned admission to the ICU must be well understood to remove from the definition those patients admitted after miscommunication or organizational issues.

In our hospital, these were mainly related to extreme emergency admissions to the operating room and cardiac patients, but this may vary from one hospital to another and should always be taken into account. If hospitals can agree on a definition of UIA and similar processes leading to an UIA can be identified, UIA has potential as a cost-effective benchmarking tool.

The specificity of UIA for the patient safety dimension of care makes it also particularly suitable for root cause analyses. This extensive analysis process used to uncover system errors can only be effective if obvious organizational safety issues have been detected. UIA could also be used to monitor activity at a departmental level or to assess the benefits of a new guideline or monitoring system.

In conclusion, our results show that UIA is a valid indicator to measure patient safety in surgical patients. It can be clearly defined and readily identified in basic hospital information systems. It provides a new perspective on the use of clinical indicators as measures of patient safety.

The authors thank Olga Bourlak and Bernie Hair (Research and Information Technologies Officers, Alfred Hospital, Melbourne, Australia) for their technical support; Adam Meehan and Steven White (Data Managers, Monash University, Melbourne, Australia) for their expert computing advice and support; Dr. Hugh Anderson (Fellow, Australian and New Zealand College of Anesthetists, Melbourne, Australia, and Staff Anesthesiologist, Alfred Hospital) for his tireless efforts to ensure accuracy and completeness of the data; and Dagmar Haller-Hester, Ph.D. (Research Fellow, Murdoch Research Institute, Melbourne, Australia), for her helpful comments and suggestions. The authors also thank the staff of the Department of Anesthesia, Alfred Hospital, for their participation and enthusiasm.