Among our numerous columns dealing with use of checklists in healthcare and other industries the WHO Surgical Safety Checklist has probably attracted the most attention. Our July 1, 2008 Patient Safety Tip of the Week “WHO’s New Surgical Safety Checklist” described the tool and provided the link to download the checklist tool and instructions how to use it. We also discussed checklist design and use in our September 23, 2008 Patient Safety Tip of the Week “Checklists and Wrong Site Surgery”.
In our January 20, 2009 Patient Safety Tip of the Week “The WHO Surgical Safety Checklist Delivers the Outcomes” we discussed the striking improvements in patient outcomes following implementation of the WHO Surgical Safety Checklist at hospitals in eight different countries. Haynes and colleagues (Haynes 2009) demonstrated that mortality at 30-days post-op decreased from 1.5% before introduction of the checklist to 0.8% after. Rate of any complication decreased from 11% to 7%. Both these outcomes were highly statistically significant. That’s a relative risk reduction of approximately 36% for mortality and major morbidity!
That striking improvement in outcomes occurred even without complete adherence to all items on the checklist. We discussed the debate as to whether the striking improvement was attributable to use of the checklist per se or to the change in “culture” that accompanied use of the checklist.
But in our April 2014 What's New in the Patient Safety World column “Checklists Don’t Always Lead to Improvement” we found that widespread adoption of a surgical checklist in over 100 hospitals in Ontario, Canada failed to demonstrate significant reductions in adjusted rates for mortality or complications (Urbach 2014). The rate of any complication decreased from 3.86% to 3.82% and mortality at 30-days post-op decreased from 0.71% to 0.65% in Canadian study, neither being statistically significant. There was also no significant changes in rates of hospital readmission and emergency department visits within 30 days after discharge. This result was surprising, especially since self-reported compliance with the checklist was over 90% at almost all participating hospitals.
Now a new study again demonstrates a striking improvement in mortality rates after implementation of a safe surgery checklist program (Haynes 2017). The South Carolina Hospital Association partnered with Ariadne Labs to establish a state-wide hospital collaborative to facilitate voluntary implementation of the surgical safety program. Hospitals generally followed a comprehensive 12-step implementation program that included active engagement of and buy-in by staff in roll out and adoption of the checklist. Their baseline period was January 2008 through December 2010 and then January 2011 through November 2013 was considered the post-implementation period.
Fourteen hospitals completed the program by December 2013. Risk-adjusted 30-day mortality among completers was 3.38% in 2010 and 2.84% in 2013, whereas mortality among other hospitals not participating in the project was 3.50% in 2010 and 3.71% in 2013. That represents a 22% difference between the groups.
In an interview in the Charleston, SC Post and Courier (Sausser 2017) Atul Gawande, a co-author of the study and well-known guru on checklists, was quoted “This is a big deal. There isn’t a drug or device that’s been discovered that can reduce mortality in surgery by that magnitude.”
In our April 2014 What's New in the Patient Safety World column “Checklists Don’t Always Lead to Improvement” we discussed several possible reasons that the Ontario study failed to show improvement but we also noted multiple other projects that did demonstrate improvement following implementation of checklist programs. Improvements of a magnitude similar to that of the original Haynes study were seen after implementation of the SURPASS checklist (de Vries 2010). That checklist is a very comprehensive checklist that deals with the entire surgical pathway, including pre- and post-operative care as well as events within the OR (see our November 30, 2010 Patient Safety Tip of the Week “SURPASS: The Mother of All Checklists”). After implementation of SURPASS the number of complications per 100 patients dropped from 27.3 to 16.7 and in-hospital mortality dropped from 1.5% to 0.8%. Note that outcomes at several comparable hospitals considered “control” hospitals did not change. And no other significant programs were introduced at the time, further suggesting that the improvements were due to implementation of the checklist. Moreover, complication rates were significantly lower in those patients for whom 80% or more of the checklist items were completed. But the authors are quick to note that the benefits of the checklist implementation are not just due to the checklist but also due to the development of a “culture of safety” that results from such implementation. Also of note was that these improvements occurred at hospitals which already had relatively high levels of quality of care.
Another study (van Klei 2012) demonstrated a 15% reduction in adjusted mortality rates after implementation of WHO’s Surgical Safety Checklist and showed outcomes were better in those with full checklist completion compared to those with partial completion or noncompliance.
Another group implemented both team training and a comprehensive surgical checklist and demonstrated significant reduction of 30-day morbidity (Bliss 2012). Overall adverse event rates decreased from 23.60% for historical control cases and 15.90% in cases with only team training, to 8.20% in cases with checklist use.
A systematic review of the impact of surgical checklists (Treadwell 2014) noted that 10 of 21 studies on implementation of surgical checklists included data on outcomes. Outcomes from those reporting were generally favorable, showing decreases in both inhospital mortality and complication rates.
The systematic review by Treadwell et al. (Treadwell 2014) cautions that the association between checklists and improved outcomes does not necessarily imply causation. First, they note that checklists are often implemented as part of a multifaceted strategy to improve care. They also note there may be reporting bias (i.e. perhaps only those with positive outcomes reported outcome data). And, third, it’s possible that not all surgical checklists are beneficial.
It’s important to keep in mind that none of these studies was a randomized controlled trial (RCT) and there are several practical barriers to ever doing such an RCT. They all have before/after observational designs and it is conceivable that factors other than just the checklist are important. Indeed, we have always strongly suspected that the change in culture is probably more important than the checklist per se. Developing checklists is not enough. You need to involve your staff in development of those checklists and educate all staff in their importance and implementation. The South Carolina project provides good guidance in how to implement such programs. You also need to audit the use of and adherence to the checklists you develop. The audit should be done for anything you develop a checklist for, not just a safe surgery checklist.
Checklists are some of the most valuable tools we have available in quality improvement and patient safety. They are simple and save time in the long run. Most take only minutes to complete. They are also the least expensive of all tools. Though they are simple and can be completed in minutes, the implementation and adoption process is much more complicated. The South Carolina project illustrates that well.
Some of our prior columns on checklists:
WHO Surgical Safety Checklist
Haynes AB, Weiser TG, Berry WR, et al. for the Safe Surgery Saves Lives Study Group. A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population. N Engl J Med. Online First January 14, 2009 (DOI: 10.1056/NEJMsa0810119), in Print January 29, 2009
Urbach DR, Govindarajan A, Saskin R, et al. Introduction of Surgical Safety Checklists in Ontario, Canada. N Engl J Med 2014; 370: 1029-1038
Haynes AB, Edmondson L, Lipsitz SR, et al. Mortality Trends After a Voluntary Checklist-based Surgical Safety Collaborative. Annals of Surgery 2017; Published Ahead of Print Post Author Corrections: April 6, 2017
Sausser L. New study shows surgery checklists in South Carolina saved patients' lives. Post and Courier (Charleston, SC) 2017; Apr 17, 2017
de Vries EN, Prins HA, Crolla RMPH, et al. for the SURPASS Collaborative Group. Effect of a Comprehensive Surgical Safety System on Patient Outcomes. N Engl J Med 2010; 363: 1928-1937
van Klei WA, Hoff RG van Aarnhem EEH et al. Effects of the Introduction of the WHO “Surgical Safety Checklist” on In-Hospital Mortality: A Cohort Study. Ann Surg 2012; 255: 44-49
Bliss LA, Ross-Richardson CB, Sanzar LJ, et al. Thirty-Day Outcomes Support Implementation of a Surgical Safety Checklist. J Am Coll Surg 2012; 215: 766-776
Treadwell JR, Lucas S, Tsou ay. Surgical checklists: a systematic review of impacts and implementation. BMJ Qual Saf 2014; 23: 299-31