Patient Safety Tip of the Week

March 5, 2013

Underutilized Safety Tools: The Observational Audit



We’ve frequently recommended performing audits on a variety of processes in healthcare. To name just a few, we’ve discussed audits in MRI safety, PCA (patient-controlled analgesia) safety, hi-alert medication safety, and many others. But most of those are retrospective audits and involve reviewing information that has been recorded in the chart or electronic medical record. Often there are processes in healthcare that are very necessary, even crucial, yet never get recorded anywhere.


One of the most underutilized tools in patient safety is the observational audit. Other industries (eg. aviation, banking, pharmaceuticals, steel making, etc.) have used audit techniques successfully for many years. We’ve previously talked about the line operations safety audits (LOSA) audits in aviation where an independent observer sits in the cockpit and monitors and assesses multiple operations and procedures, then critiques the crew. An audit such as the LOSA is resource-intense, requires a well-trained auditor, and behavior may be quite different when a known audit is being undertaken. Nevertheless, the findings are often eye-opening. The power of real-time observational methodologies has largely been untapped in healthcare today.


We have previously discussed one form of observational audit, the real-time random safety audit (see our May 18, 2010 Patient Safety Tip of the Week “Real-Time Random Safety Audits”).


Perioperative processes are an area ripe for auditing and two recent studies in the surgical literature demonstrate how auditing using observational tools can be used to identify areas for improvement. They illustrate 2 slightly different methodologies that are helpful in identifying processes of care not well done.


First is a Metric for Evaluating Task Execution in the Operating Room, better known as the acronym METEOR (Russ 2013). Their underlying concept is that there are certain key tasks by the OR team that are critical for patient safety and OR efficiency, coordination and teamwork. Another key principle they considered is that safety and efficiency in the OR are not often compromised by single, large, catastrophic failures but rather more often by small failures that go unnoticed and accumulate to impact safety or efficiency.


They note that the list of such tasks goes well beyond what would typically be included in a working checklist. You’ll recall from our previous discussions about checklists that the number of items in a checklist should be small and should include things you might otherwise forget (see our July 6, 2010 Patient Safety Tip of the Week “Book Reviews: Pronovost and Gawande” and our September 23, 2008 Patient Safety Tip of the Week “Checklists and Wrong Site Surgery”). Your typical checklist used for patient safety purposes may have 10 or fewer items. An exception is the SURPASS (SURgical PAtient Safety System) checklist (see our November 30, 2010 Patient Safety Tip of the Week SURPASS: The Mother of All Checklists”).


But Russ and colleagues note that there are far more processes and tasks involved in surgery than you would include in a checklist. So they developed an audit tool that contains items for the preoperative, intraoperative, and postoperative stages and breaks the tasks down roughly into categories of patient tasks, communication tasks and equipment tasks. They first developed a list of 106 discrete OR tasks felt to be important in “getting things done”. They then had 2 general surgeons trained in use of the list observe 50 general surgical procedures in real-time and compute task completion rates for each of the 106 tasks in the list. If tasks were completed less than 70% of the time they considered that as either a signal of poor OR team performance or that the task was not really relevant for safety or efficiency. Those tasks with <70% completion rates were then evaluated by a panel of OR experts to be rated for the tasks’ contribution to OR safety or efficiency. The ratings were then reviewed again by a smaller select group of OR experts and 2 practicing clinicians. This iterative process ultimately led to the final list, METEOR, consisting of 80 OR tasks (34 preoperative, 18 intraoperative, and 28 postoperative). METEOR theoretically can be used to identify patient safety vulnerabilities and to identify factors contributing to OR inefficiencies. It serves as a good data collection tool that obviously needs to be combined with constructive feedback to the OR team members. Because of its size and the fact that a real-time observer is needed it is not something that would be applied on a daily basis in your OR suite. However, it has considerable potential as a tool for periodic auditing of compliance with good OR practices and safety measures.


The second study used an observational methodology to assess failures in postoperative care (Symons 2013). Many of our columns have focused on problematic care of postoperative patients. A variety of system failures have contributed to adverse outcomes in such patients. These have included poor handoffs, failure to identify patients at risk for specific complications, failure to identify early clinical deterioration and failure to rescue, a variety of problems related to pain management, and medication-related problems.


Most of those have stemmed from specific case examples. But the Symons study used observational tools on a population of general surgical patients to identify and quantify such process failures. The population included 50 adults undergoing major upper or lower gastrointestinal surgery (most often for cancer) at a large urban teaching hospital. Two surgical residents, trained in patient safety research, observed the care of these patients from the first postoperative day through discharge. They attended rounds, reviewed patient charts and records, and did structured interviews with care providers and identified any event that was not considered routine in the postoperative period. They further broke those events down into those with or without process failures and assessed the likely preventability of these events. They also determined whether harm resulted from the events


A total of 352 nonroutine events were identified in the 50 patients observed for 659 days of inpatient care. Of these, 256 (73%) were deemed process failures or a median of 4.5 process failures per patient. 85% of process failures were felt to be preventable and 51% directly led to patient harm or prolonged length of stay. Process failures occurred in all aspects of care, the most frequent being medication prescribing and administration, management of lines, tubes, and drains, and pain control interventions. Process failures accounted for 57% of all preventable adverse events. Although no patient died or suffered permanent disability, the frequency of potentially preventable process failures here points out potential vulnerabilities for serious adverse outcomes.


They further analyzed their data for factors contributing to the failures and found that communication failures and delays were the main causes, together accounting for over half of all process failures and almost three quarters of adverse events.


The authors suggest that interventions found to be successful in other venues, such as daily goals and team training programs, might be applied to the postoperative setting.


This was a good study. The observational methodology allowed them to identify process failures and events that might not have been found using other methodologies.


It’s important that the feedback from the observational audits be constructive in nature. It should be used to identify gaps in the care processes that can significantly impact patient outcomes. However, the observational methodology also from time to time may uncover risky behaviors that would not otherwise come to attention. Of interest, the NTSB report in our October 2, 2007 Patient Safety Tip of the Week “Taking Off from the Wrong Runway” mentioned that a LOSA Collaborative showed that flight crewmembers who intentionally deviated from standard operating procedures were three times more likely to commit other types of errors, mismanage more errors, and find themselves in more undesired aircraft situations compared with those flight crewmembers who did not intentionally deviate from procedures. We suspect the numbers in healthcare would be similar. So auditing as above might also identify risk for other situations.






Russ S, Arora S, Wharton R, et al. Measuring Safety and Efficiency in the Operating Room: Development and Validation of a Metric for Evaluating Task Execution in the Operating Room. Journal of the American College of Surgeons 2013; 216(3): 472-481



Symons NRA, Almoudaris AM, Nagpal K, et al. An Observational Study of the Frequency, Severity, and Etiology of Failures in Postoperative Care After Major Elective General Surgery. Annals of Surgery 2012; published ahead of print 20 September 2012

Annals of Surgery. 257(1):1-5, January 2013,_Severity,.1.aspx








Print PDF version










Tip of the Week Archive


What’s New in the Patient Safety World Archive