View as “PDF version”

Patient Safety Tip of the Week

May 11, 2021

How Are Alerts in Ambulatory CPOE Doing?

When we did our first CPOE implementation back in 2007, we were flooded with suggestions for potential alerts that could be used for patient safety. We had many ideas for alerts ourselves. But we readily recognized the need to limit such alerts in order to avoid alert fatigue.

Perhaps the prime opportunity to use alerts to drive clinician behavior is in medication safety. Computers can work rapidly in the background to identify issues like allergies, drug-drug interactions, drug-disease contraindications, effects of renal or hepatic dysfunction, and other considerations that should be taken into account when prescribing medications. Most clinicians cannot take the time to consider all those factors when prescribing. So, computer-generated alerts can provide important information to the clinician at the time of order entry.

There are 2 types of alerts: “soft” alerts and “hard” alerts. Soft alerts are suggestions that the clinician can choose to implement or ignore. Hard alerts are ones that require the clinician to do something, such as accept the recommendation or explain why he/she is overriding the alert (or, in the extreme, would prevent the clinician’s action all together). Interruptive alerts are ones that require interruption of a clinician’s workflow to answer questions or input additional information. Hard alerts are examples of interruptive alerts. There are probably also some “soft” alerts that are interruptive (eg. the alert might make the clinician read a long message).

A recent review (Cerqueira 2021) assessed the effectiveness of interruptive medication-prescriber alerts in changing prescriber behavior and improving patient outcomes in ambulatory care settings via computerized provider order entry (CPOE) systems. The authors found that clinician behavior was influenced in the majority of studies, with most noting a positive change. They found that alerts decreased pharmaceutical costs, moved medications toward preferred medications tiers and steered treatments toward evidence-based choices. Importantly, they also decreased prescribing errors.

But they also found that clinician feedback was rarely solicited and, when it was, showed frustration with alerts creating a time delay. Notably, only one of the nine studies in their review reported feedback that was overall ‘positive’. Clinicians often commented that the alerts were inappropriate and intrusive.

Shi et al. (Shi 2021) recently studied barriers to using clinical decision support in ambulatory care. One of the 7 primary barriers they identified was the use of false and disruptive alarms. Another barrier was the requirement to redesign workflow.

Bombarding clinicians with alerts that are inappropriate and interruptive leads to the phenomenon of “alert fatigue” in which clinicians begin to ignore and override most alerts, even those that have the potential to avert unwanted outcomes.

There have been many studies linking clinician burnout to use of electronic medical records (EMR’s). While most relate to the overall time spent on the EMR (see our May 2021 What's New in the Patient Safety World column “More on Time Spent on the EMR”), frustration with clinical decision support (CDS) systems has been identified as a key factor in leading to clinician burnout. Jankovic and Chen (Jankovic 2020) reviewed articles dealing with aspects of CDS that contribute to burnout and identify key themes for improving the acceptability of CDS to clinicians, with the goal of decreasing said burnout.

In our June 2020 What's New in the Patient Safety World column “EMR and Medication Safety: Better But Not Yet There” we discussed results from the Leapfrog CPOE EHR evaluation tool shows some improvement over time but highlights the persistence of vulnerabilities and the wide variability of hospital CPOE EHR systems to identify medication errors and prevent adverse drug events (ADE’s). Classen et al. (Classen 2020) looked at results from over 2300 hospitals. The overall mean total score increased from 53.9% in 2009 to 65.6% in 2018. The mean hospital score for the “basic” CDS category increased from 69.8% in 2009 to 85.6% in 2018. The mean hospital score for the “advanced” CDS category increased from 29.6% in 2009 to 46.1% in 2018. Hospital EHR’s did better on some categories than others. For example, they did best on the drug-allergy category in each year, increasing from 92.9% in 2009 to 98.4% in 2018. The lowest performing category throughout the study was drug-diagnosis contraindications, where the mean score was only 20.4% in 2009 and 33.2% in 2018. The authors conclude “these systems have only modestly increased their safety performance during a 10-year period, leaving critical deficiencies in these systems to detect and prevent critical safety issues.”

So, why haven’t we been able to use CDS more effectively to improve medication safety and efficacy? It largely boils down to issues related to design and implementation of clinical alerts and reminders.

When the phenomenon of “alert fatigue” became apparent to us, we recognized the need to limit our alerts to those situations we considered most important for patient care. We did the following:

• Established a multidisciplinary committee to assess all suggested alerts and to monitor implemented alerts
• Established monitoring intervals for all implemented alerts (we suggest monitoring at one month, 3 months, and 6 months)
• Monitored the frequency that alerts triggered
• Monitored the alert acceptance and override rates
• Assessed whether the alerts achieved their intended impact
• Looked for any unintended consequences
• Interviewed clinicians to to assess their impression of both the utility and the degree of obtrusiveness of any alerts

Those actions were very much in line with the recommendations the whitepaper “Safe Practices to Reduce CPOE Alert Fatigue through Monitoring, Analysis, and Optimization” from ECRI's Partnership for Health IT Patient Safety (see our March 2021 What's New in the Patient Safety World column “ECRI Partnership Whitepaper on Alert Fatigue”). The ECRI whitepaper (ECRI 2021) actually goes into more detail about the metrics you should be following.

Regarding metrics, one unique metric not mentioned in the ECRI whitepaper but potentially very valuable was described by Einbinder and colleagues (Einbinder 2014): the “number needed to remind” (NNR). Analogous to the number needed to treat (NNT), this is the number of patients reached by a reminder to result in one recommended action being taken. They compared this to the “reminder performance” (RP), which is simply the measure of how often a recommended action is subsequently taken when the reminder is displayed.

Alagiakrishnan et al. (Alagiakrishnan 2019) used the NNR in primary care and geriatric clinics to assess clinical decision support for potentially inappropriate medications (PIM’s) from Beers criteria. Their CDS system used alerts known as Best Practice Advisories (BPA’s) to direct providers to a navigator where orders management, clinical information and educational materials were available. The BPA’s were used to advise clinicians of PIM’s among their patients' medication lists or new orders. The reminder performance (RP) across both clinics was 17.3%, which corresponds to an NNR of 5.8. The reminder performance was 37.1% in geriatric clinics vs. 13.4% in primary care clinics. The NNR in the primary care clinic was 7.4 and NNR in the geriatric clinic was 2.7.

They also developed a metric “Number Needed to Deprescribe” (NND) or the number of alert presentations specific to a medication and patient presented to a physician user before there was a deprescribing event. The reminder performance for deprescribing events was lower at 1.2%. The number needed to deprescribe (NND) was 82, with values for the primary care clinic of 80 and the geriatrics clinic of 96. There was considerable variation in all 3 parameters by the class of medication for which the BPA alerts fired.

The ECRI Partnership whitepaper reiterates the “5 Rights” model of clinical decision support (CDS) adopted from Osheroff et al. (Osheroff 2012):

1. The right information: evidence-based, suitable to guide action, pertinent to the circumstance
2. To the right person: considering all members of the care team, including clinicians, patients, and their caregivers
3. In the right CDS intervention format: such as an alert, order set, or reference information to answer a clinical question
4. Through the right channel: for example, a clinical information system such as an electronic medical record, personal health record, or a more general channel such as the internet or a mobile device
5. At the right time in workflow: for example, at time of decision, action, or need

One of the metrics in the ECRI whitepaper looks at whom the alert is targeted to. That raises a point we’ve often made in the past: you need to target the alert to the individual(s) most likely to get the desired action accomplished. That may not always be the clinician ordering something on CPOE or an ePrescribing system. For example, when an antidiarrheal medication is ordered on a patient receiving antibiotics, an alert to consider C. diff infection might be better targeted to an infection control worker than to the ordering clinician. Or alerts regarding some medication issues might be better targeted to clinical pharmacists.

The ECRI whitepaper also asks one very important question: “Is an alert the appropriate tool?”. Essentially, that is askng whether there is an alternative to accomplish the same goal. We’d like to emphasize that last point. In our March 3, 2009 Patient Safety Tip of the Week “Overriding Alerts…Like Surfin’ the Web” we noted that use of standardized order sets may avoid the need to generate some alerts (though standardized order sets can create some problems of the own, particularly when they contain outdated information that is no longer appropriate).

There is another phenomenon we’ve seen over and over. That is the disparity in effectiveness between “prospective” alerts and “look-back" alerts. When we’ve tried to use alerts to avoid using a “potentially inappropriate” medication (an example might be tricyclic antidepressants in an elderly patient), we found that clinicians almost never stopped that medication once they had already previously prescribed it. On the other hand, the alert was effective at preventing new prescriptions for that medication in the elderly. Awdishu et al. (Awdishu 2016), looking at the impact of alerts on prescribing in patients with renal disease, also found that prospective alerts had a greater impact than look-back alerts (55.6% vs 10.3%).

Marcilly et al. developed evidence-based usability design principles for medication alerting systems (Marcilly 2018). They note that alerts should:

1. Improve the system’s signal-to-noise ratio
2. Support collaborative work, advocate a team approach and make the alert system a ‘team player’
3. Fit with clinicians’ workflow and their mental model
4. Display relevant data within the alert
5. Make the system transparent for the user
6. Include actionable tools within the alert

The Marcilly paper notes that alert design should take into account parameters such as the

patient’s clinical context or the clinician’s specialty. That makes sense. An alert to a primary care physician reminding them to adjust a medication dose based upon the patient’s renal function might just create unnecessary “noise” for a nephrologist entering a similar order. Similarly, regarding context, some alerts about renal dosing may not be appropriate for a patient already on dialysis.

Alerts are also more likely to be complied with if they offer “actionable” tools. For example, an alert that offers an alternative action or choice is much more likely to be complied with than one that simply suggests the original action is discouraged.

Shah et al. (Shah 2021) looked at CDS alerts regarding renal dosing in hospitalized patients. They found that alerts were nearly always presented inappropriately and were all overridden during the 1-year period studied. This was distinctly different from data they had previously seen in a legacy system in which medication-related CDS alerts associated with renal insufficiency had been found to be the most clinically beneficial. They identified several potential reasons why the current medication-related CDS alerts associated with renal insufficiency were less effective than they had been in the legacy homegrown system:

• No automatic calculation of level of renal function (CrCl or estimated GFR)
• No automatic dose adjustment
• No recommendations for alternatives
• No consideration of patient specific parameters, such as dialysis

The Shah study tells us we cannot simply adopt all vendor-generated CDS alerts and reminders. Rather, we must apply the same rigorous evaluation and monitoring to those alerts and reminders that we used when we designed all our own alerts and reminders. We need to make sure they are evidence-based, are actionable, offer alternatives where appropriate, are prospective, fit with workflow, take clinical context and clinician specialty into consideration, and, most of all, are limited in volume to those that are most likely to impact patient safety and patient outcomes so we can avoid alert fatigue.

We can’t duplicate the extensive literature review done by the ECRI Partnership. We encourage you to go back to their whitepaper (ECRI 2021). Their “evidence tables” in the appendix includes a summary of the findings from 12 key studies. We hope you’ll use lessons learned in those plus the key elements in today’s column to make sure you are getting the most out of your clinical decision support tools without generating alert fatigue and contributing to clinician burnout.

See some of our other Patient Safety Tip of the Week columns dealing with unintended consequences of technology and other healthcare IT issues:

• June 19, 2007 “Unintended Consequences of Technological Solutions”
• May 20, 2008 “CPOE Unintended Consequences – Are Wrong Patient Errors More Common?”
• June 17, 2008 “Technology Workarounds Defeat Safety Intent”
• August 26, 2008 “Pattern Recognition and CPOE”
• September 9, 2008 “Less is More….and Do You Really Need that Decimal?”
• December 16, 2008 “Joint Commission Sentinel Event Alert on Hazards of Healthcare IT”
• February 2009 “Healthcare IT The Good and The Bad”
• March 3, 2009 “Overriding Alerts…Like Surfin’ the Web”
• October 2009 “A Cautious View on CPOE”
• November 24, 2009 “Another Rough Month for Healthcare IT
• April 20, 2010 “HIT’s Limited Impact on Quality To Date”
• July 27, 2010 “EMR’s Still Have a Long Way to Go”
• March 22, 2011 “An EMR Feature Detrimental to Teamwork and Patient Safety”
• January 24, 2012 “Patient Safety in Ambulatory Care”
• June 26, 2012 “Using Patient Photos to Reduce CPOE Errors”
• June 2012 “Leapfrog CPOE Simulation: Improvement But Still Shortfalls”
• July 17, 2012 “More on Wrong-Patient CPOE”
• January 2013 “More IT Unintended Consequences”
• April 23, 2013 “Plethora of Medication Safety Studies”
• April 30, 2013 “Photographic Identification to Prevent Errors”
• October 8, 2013 “EMR Problems in the ED”
• March 11, 2014 “We Miss the Graphic Flowchart!”
• October 2014 “Ebola Exposes Fundamental Flaw”
• January 2015 “Beneficial Effect of EMR on Patient Safety”
• March 2015 “CPOE Fails to Catch Prescribing Errors”
• March 31, 2015 “Clinical Decision Support for Pneumonia”
• August 2015 “Newborn Name Confusion”
• December 2015 “Opioid Alert Fatigue”
• January 12, 2016 “New Resources on Improving Safety of Healthcare IT”
• January 19, 2016 “Patient Identification in the Spotlight”
• February 9, 2016 “It was just a matter of time…”
• April 5, 2016 “Workarounds Overriding Safety”
• May 2016 “Name Confusion in the Pharmacy”
• May 3, 2016 “Clinical Decision Support Malfunction”
• May 24, 2016 “Texting Orders – Is It Really Safe?”
• August 23, 2016 “ISMP Canada: Automation Bias and Automation Complacency”
• November 22, 2016 “Leapfrog, Picklists, and Healthcare IT Vulnerabilities”
• January 2017 “Joint Commission Thinks Twice About Texting Orders”
• February 28, 2017 “The Copy and Paste ETTO”
• March 2017 “Yes! Another Voice for Medication e-Discontinuation!”
• April 2017 “How Much Time Do We Actually Spend on the EMR?”
• June 27, 2017 “Texting – We Told You So!”
• August 1, 2017 “Progress on Wrong Patient Orders”
• January 2018 “Can We Improve Barcoding?”
• January 16, 2018 “Just the Fax, Ma’am”
• January 30, 2018 “Texting Errors Revealed”
• June 19, 2018 “More EHR-Related Problems”
• September 2018 “More Clinical Decision Support Successes”
• December 11, 2018 “Another NMBA Accident”
• January 1, 2019 “More on Automated Dispensing Cabinet (ADC) Safety”
• February 5, 2019 “Flaws in Our Medication Safety Technologies”
• March 26, 2019 “Patient Misidentification”
• May 2019 “Too Much Time on the EMR”
• May 21, 2019 “Mixed Message on Number of Open EMR Records”
• July 23, 2019 “Order Sets Can Nudge the Right Way or the Wrong Way”
• September 10, 2019 “Joint Commission Naming Standard Leaves a Gap”
• September 24, 2019 “EHR-related Malpractice Claims”
• December 17, 2019 “Tale of Two Tylers”
• June 2020 “EMR and Medication Safety: Better But Not Yet There”
• June 16, 2020 “Tracking Technologies”
• July 2020 “Patient Requests for EHR Corrections”
• July 21, 2020 “Is This Patient Allergic to Penicillin?”
• September 2020 “More on Workarounds”
• November 17, 2020 “A Picture Is Worth a Thousand Words”
• March 2021 “ECRI Partnership Whitepaper on Alert Fatigue”

See some of our previous columns dealing with the Leapfrog CPOE EHR evaluaton tool:

• July 27, 2010 “EMR’s Still Have a Long Way to Go”
• June 2012 “Leapfrog CPOE Simulation: Improvement But Still Shortfalls”
• April 23, 2013 “Plethora of Medication Safety Studies”
• March 2015 “CPOE Fails to Catch Prescribing Errors”
• May 3, 2016 “Clinical Decision Support Malfunction”
• November 22, 2016 “Leapfrog, Picklists, and Healthcare IT Vulnerabilities”
• June 2020 “EMR and Medication Safety: Better But Not Yet There”

References:

Cerqueira O, Gill M, Swar B, et al. The effectiveness of interruptive prescribing alerts in ambulatory CPOE to change prescriber behaviour & improve safety. BMJ Quality & Safety 2021; Published Online First: 19 April 2021

https://qualitysafety.bmj.com/content/early/2021/03/14/bmjqs-2020-012283

Shi Y, Amill-Rosario A, Rudin RS, et al. Barriers to using clinical decision support in ambulatory care: Do clinics in health systems fare better? Journal of the American Medical Informatics Association 2021; ocab064 Published: 25 April 2021

https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocab064/6250921?redirectedFrom=fulltext

Jankovic I., Chen JH. Clinical Decision Support and Implications for the Clinician Burnout Crisis. Yearbook of medical informatics 2020; 29(1): 145-154

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7442505/

Classen DC, Holmgren AJ, Co Z, et al. National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018. JAMA Netw Open 2020; 3(5): e205547

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2766545?resultClick=3

ECRI Partnership for Health IT Patient Safety. Safe Practices to Reduce CPOE Alert Fatigue through Monitoring, Analysis, and Optimization. ECRI 2021

https://d84vr99712pyz.cloudfront.net/p/pdf/hit-partnership/partnership_whitepaper_alertfatigue_final.pdf

Einbinder J, Hebel E, Wright A, Panzenhagen M, Middleton B. The number needed to remind: a measure for assessing CDS effectiveness. AMIA Annu Symp Proc 2014; 14: 506-515

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4419960/

Alagiakrishnan K, Ballermann M, Rolfson D, et al. Utilization of computerized clinical decision support for potentially inappropriate medications. Clin Interv Aging 2019; 14: 753–762

https://www.dovepress.com/utilization-of-computerized-clinical-decision-support-for-potentially--peer-reviewed-fulltext-article-CIA

Osheroff JA, Teich JM, Levick D, et al. Improving outcomes with clinical decision support: an implementer’s guide. 2nd ed. Chicago (IL): Healthcare Information and Management Systems Society; 2012

https://www.amazon.com/Improving-Outcomes-Clinical-Decision-Support/dp/0984457739/ref=sr_1_1?dchild=1&keywords=Osheroff&qid=1614095482&s=books&sr=1-1

Awdishu L, Coates CR, Lyddane A, et al. The impact of real-time alerting on appropriate prescribing in kidney disease: a cluster randomized controlled trial. J Am Med Inform Assoc 2016; 23(3): 609-616

https://academic.oup.com/jamia/article/23/3/609/2909002

Marcilly R, Ammenwerth E, Roehrer E, et al. Evidence-Based usability design principles for medication alerting systems. BMC Med Inform Decis Mak 2018; 18: 69

https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-018-0615-9

(Marcilly 2018)

Shah SN, Amato MG, Garlo KG, et al. Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts. Journal of the American Medical Informatics Association 2021; ocaa222

https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocaa222/6124727?searchresult=1#227144466

Print “PDF version”

http://www.patientsafetysolutions.com/

Home

Tip of the Week Archive

What’s New in the Patient Safety World Archive