Achieving computer vision for drug delivery events includes several steps: detecting a syringe, recognizing the text, measuring dose, and reporting the information back to the clinician and EMR.

Achieving computer vision for drug delivery events includes several steps: detecting a syringe, recognizing the text, measuring dose, and reporting the information back to the clinician and EMR.

Computer vision uses digital images in conjunction with powerful computing algorithms to analyze the visual world. This technology has frequently been associated with preventing accidents in self-driving cars or, most notably in medicine, with radiological breast screening exams, where they are at least equivalent to expert radiologists (Nature 2020;577:89-94). As these tools are constantly improving and becoming easier to implement, their utility across the hospital, including ICUs and the OR, is becoming more widespread, with the potential to improve patient care and safety while decreasing documentation burden for clinicians.

In the OR, some computer vision studies have focused on developing tools to assess the skill level of surgeons (Sci Rep 2021;11:5197). This is of importance not only for education and training but because skill level has been associated with patient outcomes (JAMA Surg 2020;155:590-8). For example, it is possible to separate surgeons into highly skilled and less skilled groups by applying a neural network to videos of laparoscopic surgeries with 87% accuracy. A neural network architecture mimics the neural architecture of the human brain, with a series of interconnected nodes that all have different weights. Through a training data set, machine learning algorithms assign weights to individual nodes and then update these weights by comparing the output from the algorithm to the training data set. After training is complete, the outcome is compared with a different testing data set to assess the algorithm's accuracy.

Looking beyond the visual field of the surgeon, larger room-based camera systems are being used to predict the current phase of patient care in an OR (Minim Invasive Ther Allied Technol 2019;28:82-90). From this information, there are a lot of potential benefits, from real-time personalized radiation dosimetry data for each team member, to automatically adjusting music volume during critical portions of the case, to predicting the remaining duration of surgery (IEEE Trans Med Imaging 2019;38:1069-78). This information could decrease patient wait times and prioritize instrument cleaning based on what is next needed across all ORs, minimizing delays.

These strategies are useful outside the OR in ICUs as well, where handwashing events can be characterized with greater than 95% accuracy (J Am Med Inform Assoc 2020;27:1316-20). The computer algorithm was equivalent in performance to a human observer, at a much lower cost over time when compared to a trained observer. Using depth cameras rather than traditional camera technology can preserve information about distance from the camera and the external shape of an object while also providing individual anonymity, a frequent concern in video- or image-based research on human behaviors.

Until recently, computer vision in the OR directly pertaining to anesthesiologists has been lacking. Despite anesthesiology being at the forefront of other aspects of artificial intelligence such as machine learning and predictive analytics, most computer vision research in our field pertains to analysis of ultrasound images for neuraxial or intravascular procedures (IEEE Trans Med Imaging 2018;37:81-92; Deep Learning and Data Labeling for Medical Applications Springer; 2016:30-8). Through a FAER Mentored Research Training Grant (MRTG) at the University of Washington, my colleagues and I are applying computer vision techniques to anesthesiologists in the OR, aiming to provide a second set of eyes for syringe selection and automating drug delivery documentation.

In a study of anesthesia records, drugs were omitted from the electronic medical record (EMR) in 15% of instances, and the documented drug matched the administered drug name and dose only 83% of the time (Can J Anaesth 2014;61:979-85). Many of the manual provider inputs in the anesthesia record are documentation of actions, such as drug delivery, vascular access, and airway management, areas where computer vision has the potential to augment the process or documentation. Accurate documentation assumes increased importance with the growing use of predictive analytics and machine learning models for health care.

In addition, drug delivery errors in anesthesia drug administration are common, can result in morbidity or mortality, and should be preventable. Numerous attempts to minimize drug errors and improve documentation have resulted in incremental improvement, but these efforts generally require additional steps by providers (Br J Anaesth 2018;121:1338-45; Anesth Analg 2017;125:458-66; Int J Med Inform 2008;77:448-60). These are steps that could be eliminated by incorporating smart eyewear with neural networks to automatically identify a syringe at the time it is picked up by a provider, prior to injection. Direct recording of drug delivery frees physicians from having to break up their workflow to chart drugs, which allows them to transition smoothly between tasks and ultimately provide greater surveillance of the patient.

The work at the University of Washington my colleagues and I are conducting involves several computer vision tasks. Specifically, object detection of a syringe in the hand of a provider, followed by optical character recognition to “read” the name and concentration of the drug, and lastly volume assessment, which measures the syringe volume before and after drug administration is performed. These tasks need to be performed quickly, and there are typically just a few seconds between drug selection and delivery. After end-to-end algorithm development and comparison with human labelled video as well as the EMRs, a comparison of simulated errors in which providers intentionally mismatch drug vials and syringes to test the algorithm will be performed.

This is just one example of the potential for computer vision in the OR specifically for anesthesiologists. From narcotic waste documentation to difficult airway prediction, there are many productive areas for study in the future. This early work would not be possible without the protected time and money offered by the FAER MRTG and will hopefully serve as a springboard for further innovation in this new and exciting field.

Join us on Twitter and Facebook

Read up and share your favorite ASA Monitor articles on our brand-new Twitter and Facebook accounts! Follow us to take advantage of timely, informative, and leading-edge resources. Don't miss out, join us here on Twitter: @ASAMonitor and Facebook: @ASAMonitor.

Kelly Michaelsen, MD, PhD, Acting Assistant Professor, University of Washington, Seattle.

Kelly Michaelsen, MD, PhD, Acting Assistant Professor, University of Washington, Seattle.