Print “PDF version”
What’s New in the
Patient Safety World
July 2021
EPIC Sepsis
Prediction Tool Falls Short
Early recognition and treatment of sepsis is essential to
reduce mortality from sepsis. Everyone agrees that it would be helpful to have
help in recognizing early signs of sepsis and the of data in the EMR
(electronic medical record) is a logical potential source for early warning
signs. We’ve discussed some early warning tools for sepsis in our Patient
Safety Tips of the Week for March 15, 2011 “Early
Warnings for Sepsis” and September
8, 2015 “TREWScore for Early Recognition of Sepsis” and our October 2015 What's New in the Patient Safety World column “Even Earlier Recognition of Severe Sepsis”.
But perhaps the most widely used tool for predicting sepsis is a proprietary tool that is part of the EPIC electronic health record, one of the most widely used EMR’s in the US.
Wong et al. (Wong 2021) recently published results of an external validation of the Epic Sepsis Model (ESM). Its performance was less than stellar. In the analysis of over 38,000 hospitalizations at the University of Michigan, they found the ESM had a sensitivity of 33%, specificity of 83%, positive predictive value of 12%, and negative predictive value of 95%. The ESM generated alerts on 18% of all patients but did not detect sepsis in 67% of patients who actually had sepsis.
The researchers used an ESM score threshold of 6, within the recommended range by EPIC. If the ESM were to generate an alert only once per patient when the score threshold first exceeded 6—a strategy to minimize alerts—then clinicians would still need to evaluate 15 patients to identify a single patient with eventual sepsis. And, if clinicians were willing to reevaluate patients each time the ESM score exceeded 6 to find patients developing sepsis in the next 4 hours, they would need to evaluate 109 patients to find a single patient with sepsis.
The authors conclude that “the increase and growth in deployment of proprietary models has led to an underbelly of confidential, non–peer-reviewed model performance documents that may not accurately reflect real-world model performance.”
In the accompanying editorial, Habib et al. (Habib 2021) emphasize not only the importance of external validation of such tools but also the need to avoid “black box” type proprietary tools and always use open-access models so external validation is possible. They also highlight the importance of having the appropriate staff to evaluate performance in each hospital’s own clinical setting.
Some of our other columns on sepsis:
· March 15, 2011 “Early Warnings for Sepsis”
· April 1, 2014 “Expensive Aspects of Sepsis Protocol Debunked”
· September 8, 2015 “TREWScore for Early Recognition of Sepsis”
· October 2015 “Even Earlier Recognition of Severe Sepsis”
· February 2, 2016 “Success
Against Sepsis”
· March 2016 “Finally…A
More Rationale Definition for Sepsis”
· February 2017 “Yes,
the New Sepsis Criteria Fit the Bill”
· June 6, 2017 “NYS
Mandate for Sepsis Protocol Works”
References:
Wong A, Otles E, Donnelly JP, et al. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Intern Med 2021; Published online June 21, 2021
https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307
Habib AR, Lin AL, Grant RW. The Epic Sepsis Model Falls Short—The Importance of External Validation. JAMA Intern Med 2021; Published online June 21, 2021
https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2781313
Print “PDF version”
http://www.patientsafetysolutions.com/
What’s New in
the Patient Safety World Archive