View as “PDF version

Patient Safety Tip of the Week

December 21, 2021     Spinning Misinformation

 

Ah…spin! It’s not just for politicians and cable news. Unfortunately, it has also become a part of medical research and medical publications. In our February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” we gave an example of a study that was “spun” in not just one, not just two, but three prestigious medical journals by several respected authors.

 

Randomized controlled trials (RCT’s) are the gold standard for determining whether a drug or procedure or other treatment is superior to placebo in a comparable population. But the literature is replete with examples where the primary outcome parameter of an RCT was not met, yet the published report on that RCT includes “spin” that suggests the study showed a positive result. And all too often, the busy clinician reads just the conclusion section of an abstract of a study published in a journal article and gets a positive impression that was not justified by the data in the study.

 

Khan et al. reported on the level and prevalence of spin in published cardiovascular randomized clinical trial reports with statistically nonsignificant primary outcomes (Khan 2019). They identified spin n 57% of abstracts and 67% of main texts of published articles. 11% had spin in the title, 38% had spin in the results section, and 54% had spin in the conclusions. Among the abstracts, spin was observed in 41% of results sections and 48% of conclusions sections. They conclude that, in reports of cardiovascular RCTs with statistically nonsignificant primary outcomes, investigators often manipulate the language of the report to detract from the neutral primary outcomes.

 

But sometimes it is not possible to perform an RCT with placebo or “standard care” as the comparator. For example, there may be a current treatment that is known to be successful. In such cases, a “noninferiority” trial is conducted, attempting to show that the new treatment is at least as good as the existing treatment.

 

A new study (Ito 2021) looked for the presence of “spin” in noninferiority oncology trials having results that were not statistically significant. Of 52 RCT’s published between 2010 and 2019, they identified spin in the abstract in 34 (65.4%) and spin in the main text in 38 (73.1%).

 

The researchers found that novel experimental treatments may be associated with a high spin prevalence. But, perhaps the most surprising finding of the Ito analysis was that trials with nonprofit funders only were more likely to have spin. We have always thought that spin is usually more common in studies that were funded by industry. But Ito et al. point out that it is publication bias rather than spin that is more common in industry-funded studies and that spin, per se, is actually more common in studies funded by non-profits.

 

In the accompanying editorial, Zhang and Guan (Zhang 2021) point out that spin was even more common in noninferiority trials analyzed by Ito et al. than in superiority oncology trials. They caution that “given the expanding influence of peer-reviewed publications on medical products’ regulatory approval, clinical use, and reimbursement, high levels of spin could mislead decision-makers under various situations and thus compromise patient health and health system efficiency. The findings of Ito et al. highlight that stakeholders should handle results for novel treatments that are not significant with caution.”

 

In our February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” we noted an excellent review on the limitations of randomized controlled trials (Kaul and Diamond 2010). This paper is very good at helping you understand some complicated statistical issues but really emphasizes three points we have often made in the past:

 

Probably the most common form of “spin” is when the primary outcome is not met but a post-hoc analysis finds a subgroup of patients that had a positive outcome. Such post-hoc analyses are meant to be hypothesis-generating and should lead to performance of another RCT in a population represented by that particular subgroup. The results of that post-hoc analysis should never be adopted into practice without undergoing their own RCT. Another common form of spin is focusing on positive benefit in a secondary endpoint when the primary endpoint was not met. Ito et al. found that the most prevalent spin strategy in their analysis of noninferiority trials was focusing on the secondary endpoint or subgroup analysis.

 

Another form of spin is extrapolating data (particularly cost data) from other studies to show a cost benefit from a currently studied drug or procedure when the data from the current RCT failed to show a cost benefit (see our February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” for an example).

 

Another big problem is when a treatment or procedure shows a positive result when studied in a very narrow population but the authors suggest that it is effective for a much broader population.

 

Particularly in oncology trials, use of surrogate markers can also lead to spin. An improvement in the surrogate marker may not translate to improvement in more clinically meaningful outcomes such as overall survival. But this is not limited to oncology trials. The recent controversial FDA approval of aducanumab for Alzheimer’s Disease was based upon improvement of a biomarker reflecting theoretical pathogenesis of the disorder but lacking evidence of clinical impact.

 

Ito et al. point out that, compared with superiority RCT’s, noninferiority RCT’s may have more factors that complicate interpretation, including noninferiority margin, assay sensitivity, and choice of analysis population.

 

Ito et al. also point out that safety is often a primary focus of noninferiority trials but that safety itself should not be the sole consideration. They caution that researchers should

consider the benefit of an experimental treatment based on the results of a confirmatory analysis of efficacy, not just on the safety of the treatment.

 

Boutron and Ravaud (Boutron 2018) did a literature review to determine the prevalence of some forms of “spin” in specific fields and the possible effects of some types of spin on readers’ interpretation and research dissemination. The article has a good discussion of the various types of spin. They also discuss why researchers would spin their reports and possible ways to avoid it. Among the reasons they posit is the pressure to publish, particularly in high-impact factor journals that favor publication of “positive” results.

 

We’ve always been suspicious of “noninferiority” trials. If a current treatment is effective, can you justify a new treatment that is quite likely to be marketed at a cost well above that of the existing treatment? Perhaps, the new treatment is “noninferior” to the existing treatment in terms of efficacy but has a more favorable side effect or safety profile. And, rarely, it may be looking to see if an existing low-cost drug might be “noninferior” to a current expensive treatment. But we always get our hackles up when we see “noninferiority” as an outcome.

 

“Spin” is everywhere these days. We can’t turn on the TV or radio without seeing it and it seems spin is the rule rather than the exception on many social media sites. Now we can’t even trust our medical journals to do a good job of rooting out spin. It’s important for our patients that we maintain a keen eye when interpreting results of any published studies.

 

 

References:

 

 

Khan MS, Lateef N, Siddiqi TJ, et al. Level and prevalence of spin in published cardiovascular randomized clinical trial reports with statistically nonsignificant primary outcomes: a systematic review. JAMA Netw Open 2019; 2(5): e192622

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2732330

 

 

Ito C, Hashimoto A, Uemura K, Oba K. Misleading Reporting (Spin) in Noninferiority Randomized Clinical Trials in Oncology With Statistically Not Significant Results: A Systematic Review. JAMA Netw Open 2021; 4(12): e2135765

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2786921

 

 

Zhang Y, Guan X. Misleading Reporting in Statistically Not Significant Oncology Trials—Joining Efforts Toward Unbiased Results Interpretation. JAMA Netw Open 2021; 4(12): e2138695

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2786922

 

 

Kaul S, Diamond GA. Trial and Error: How to Avoid Commonly Encountered Limitations of Published Clinical Trials. J Am Coll Cardiol 2010 55: 415-427

http://content.onlinejacc.org/cgi/content/abstract/55/5/415

 

 

Boutron I, Ravaud P. Misrepresentation and distortion of research in biomedical literature. Proc Natl Acad Sci 2018; 115(11): 2613-2619

https://www.pnas.org/content/115/11/2613

 

 

 

 

 

Print “PDF version

 

 

 

 

 


 

 

http://www.patientsafetysolutions.com/

 

Home

 

Tip of the Week Archive

 

What’s New in the Patient Safety World Archive