Patient Safety Tip of the Week

February 16, 2010 Spin/Hype…Knowing It When You See It

In many of our columns we have criticized published studies for presenting their results in a biased fashion. Understanding how to read articles published in peer-reviewed journals is more important than ever since even in the most respected journals articles are published with conclusions and discussions claiming outcomes not truly supported by the data.

An excellent review on the limitations of randomized controlled trials (RCT’s) was recently published in Journal of the American College of Cardiology (Kaul and Diamond 2010). This paper is very good at helping you understand some complicated statistical issues but really emphasizes three points we have often made in the past:

· Many articles report outcomes that are statistically significant but are of little clinical significance.

· Post-hoc subgroup analyses are prone to error and inappropriate interpretation and should be used only to generate ideas for futures studies. Otherwise, they may erroneously lead to adoption of practices that are not evidence-based.

· Use of composite outcomes is especially likely to give rise to inappropriate conclusions when the outcomes are driven by one component of that composite, especially when that component is not as clinically significant as other components.

They provide numerous examples from the cardiology literature demonstrating the above concepts. Whether or not you are interested in cardiology studies, the paper is worth reading just to help you better understand how to read a research article and identify biases that may render the conclusions erroneous.

When we teach medical students or residents or even our attending physician colleagues at Journal Club how to read and interpret the literature there are some articles that serve as good examples of such biases. One was a study on the use of dexmedetomidine in the ICU (see our February 10, 2009 Patient Safety Tip of the Week “Sedation in the ICU: The Dexmedetomidine Study”). That study was a good example of a study that was presented in a positive light even though the study failed to meet its primary outcome goals. It was also a good example of the failure to use clinically important outcomes as primary measures. And we discussed then the likelihood that dexmedetomidine would start to become adopted in ICU’s without a solid evidence base and that is exactly what were are beginning to see. Marketing trumps science most of the time!

By the way, we hope you also saw the article in the New York Times this past weekend demonstrating the same phemonenon with robotic surgery for prostate cancer (Kolata 2010).

And then there are some studies that are like “the gift that keeps on giving” – the study spawns multiple articles that all tend to “spin” the conclusions in a certain direction and create a “hype” about a new drug or new technology. A great example we like to use is a series of articles on the use of silver-coated endotracheal tubes for the prevention of ventilator-associated pneumonia.

In our September 2, 2008 Patient Safety Tip of the Week “Updates on VAP Prevention” we discussed an article on use of silver-coated endotracheal tubes to prevent VAP that appeared in the August 20, 2008 issue of JAMA (Kollef et al 2008). That article showed the silver-coated endotracheal tubes led to a relative risk reduction for VAP of 36% in patients who were intubated for at least 24 hurs. Certainly sounds impressive! However, closer scrutiny of the article raises numerous questions. The rationale for using silver-coated endotracheal tubes makes good sense from a biological standpoint. The diagnosis of VAP was based on quantitative culture of bronchoalveolar fluid if VAP was suspected on the basis of radiographic findings or other clinical signs including fever or hypothermia, leukocytosis or leukopenia, or purulent tracheal aspirate. However, when one looks at the data, the absolute risk reduction for VAP was only 2.7% and the NNT (number needed to treat) to prevent one case of VAP was 37. More importantly, the silver-coated endotracheal tube group showed no improvement in mortality, duration of intubation, or ICU or total LOS. Data on antibiotic use or costs was not provided.

Other issues with the study were lack of standardization of other interventions (in both the control and silver arms), single blinding (the investigators were not blinded), relatively low rates of VAP in both the silver and control groups, a disproportionate percentage of patients with COPD in the control group, a low percentage of patients able to provide informed consent, and industry funding for the study. Note also that the “supplemental” information referred to in the article (having to do with criteria for diagnosis of VAP) is not downloadable from the website address provided in the article. And a subsequent critical letter to the editor (Klompas 2008) even noted that much of the microbiological flora identified in the article may well have represented contaminants rather than truly pathological species.

The bottom line is that we were left with an intervention that requires a large NNT and fails to improve any of the really important clinical outcomes. So they broke Rule #1 “Make sure that results are not just statistically significant. Make them clinically important.”

But the spin does not stop there. A few months later another paper from the NASCENT trial (Shorr et al 2009) touts the cost effectiveness of the silver-coated endotracheal tubes. They use the data on percent of cases of VAP avoided with the silver-coated endotracheal tubes from the above NASCENT study but use the marginal cost savings for VAP cases avoided from the published literature. They fail to state that in the NASCENT study, in fact, the silver-coated endotracheal tube group showed no improvement in mortality, duration of intubation, or ICU or total LOS. Though it did not provide data on actual costs in the NASCENT study, it is hard to believe there were any cost savings given the lack of improvement in those clinically important outcomes. Yet the new paper leads you to believe that use of the silver-coated endotracheal tubes is a “slam dunk” and can save you as much as $16,000 per case of VAP prevented.

Not had enough yet? There’s more. A new paper (Afessa et al 2009) does a post-hoc cohort analysis on the NASCENT trial data and separates them out into cohorts that did develop VAP and those that did not. They then conclude that for those that did develop VAP the mortality rates were statistically significantly lower in the silver-coated group. They go on to discuss all the potential biological reasons why the silver-coated tubes might lower mortality in these patients with VAP. To be fair, the authors do include a statement that because this is a post-hoc analysis, the findings are exploratory and need to be confirmed in further studies. However, they give only 3 lines to the fact that in those patients who did not develop VAP (a much larger group) mortality was statistically significantly increased in the silver-coated group! So if post-hoc analysis is being allowed why wasn’t the title of this paper “Association Between a Silver-Coated Endotracheal Tube and Increased Mortality in Patients Who Don’t Develop Ventilator-Associated Pneumonia”? Of course, because this is a post-hoc subgroup analysis, neither conclusion should be made. The point is that post-hoc analyses can be used to drive practice patterns that are not truly evidence-based.

But the overarching theme is this: those who read the three papers without having their antennae up for spin and hype would think that a randomized controlled trial by respected investigators with results published in three prestigious peer-reviewed medical journals showed that silver-coated endotracheal tubes prevent VAP, save lots of money, and reduce mortality. When in fact that has not yet been demonstrated. And if the FDA approves these endotracheal tubes (which it may, depending on what route is chosen for review of a modification of devices that are already in use) you can bet many will rush out and purchase these new silver-coated endotracheal tubes.

Most bothersome is the fact that our leading medical journals allow papers to be written in such a fashion that “spins” results in ways that promote devices or treatments in ways well beyond what the science actually showed.

References:

Kaul S, Diamond GA. Trial and Error: How to Avoid Commonly Encountered Limitations of Published Clinical Trials. J Am Coll Cardiol 2010 55: 415-427

http://content.onlinejacc.org/cgi/content/abstract/55/5/415

Kolata G. Results Unproven, Robotic Surgery Wins Converts.

New York Times February 13, 2010

http://www.nytimes.com/2010/02/14/health/14robot.html?scp=1&sq=robot&st=cse

Kollef MH. Afessa B. Anzueto A. Veremakis C. Kerr KM. Margolis BD. Craven DE. Roberts PR. Arroliga AC. Hubmayr RD. Restrepo MI. Auger WR. Schinner R. NASCENT Investigation Group. Silver-coated endotracheal tubes and incidence of ventilator-associated pneumonia: the NASCENT randomized trial. JAMA 2008; 300(7):805-13

http://jama.ama-assn.org/cgi/content/abstract/300/7/805

http://jama.ama-assn.org/cgi/reprint/300/7/805

Chastre, Jean MD Preventing Ventilator-Associated Pneumonia: Could Silver-Coated Endotracheal Tubes Be the Answer?. JAMA 2008; 300(7):842-844

http://jama.ama-assn.org/cgi/content/extract/300/7/842

Klompas M. Silver-Coated Endotracheal Tubes and Patient Outcomes in Ventilator-Associated Pneumonia. JAMA. 2008; 300(22): 2605

http://jama.ama-assn.org/cgi/content/extract/300/22/2605

Kollef MH, Afessa B, Anzueto A. Silver-Coated Endotracheal Tubes and Patient Outcomes in Ventilator-Associated Pneumonia—Reply. JAMA 2008; 300(22): 2605-2606

http://jama.ama-assn.org/cgi/content/extract/300/22/2605-a

Afessa B, Shorr AF, Anzueto AR, Craven DE, Schinner R, Kollef MH.

Association Between a Silver-Coated Endotracheal Tube and Reduced Mortality in Patients With Ventilator-Associated Pneumonia. Chest 2009; prepublished online December 28, 2009

http://chestjournal.chestpubs.org/content/early/2009/12/24/chest.09-0391.abstract

Shorr AF, Zilberberg MD, Kollef M. Cost-effectiveness analysis of a silver-coated endotracheal tube to reduce the incidence of ventilator-associated pneumonia. Infect Control Hosp Epidemiol. 2009 Aug; 30(8):759-763

http://www.journals.uchicago.edu/doi/abs/10.1086/599005

Prior columns on VAP:

September 2, 2008 Patient Safety Tip of the Week “Updates on VAP Prevention”

November 11, 2008 Patient Safety Tip of the Week “Probiotics and VAP Prevention”

January 2009 What’s New in the Patient Safety World “Preventing Infections in the ICU”

http://www.patientsafetysolutions.com

Home

Patient Safety Tip of the Week Archive

What’s New in the Patient Safety World Archive