View as “PDF version”
Our February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” discussed in detail the problem of “spin”. Spin is the manipulation of messaging when the primary endpoint of a clinical trial is not met but results are presented in a positive light. For example, a paper may focus on analysis of a subgroup that fared better or on some secondary endpoint that was met. Post hoc subgroup analysis should only be hypothesis-generating and subject to a new clinical trial to test that hypothesis. Or they might focus on a result that is statistically significant but not clinically significant. Or they might use composite outcomes, which are especially likely to give rise to inappropriate conclusions when the outcomes are driven by one component of that composite, when that component is not as clinically significant as other components.
Unfortunately, reviewers and editors of our medical journals are not doing a very good job of preventing “spin” in published articles. In our February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” we gave multiple examples, including one study rife with spin that led to publication in not one, not two, but three “respected” medical journals.
Recently, two physicians who have served as editors-in-chief of several JAMA journals have spoken out about spin and the need for responsible reporting of trial results to the media and other stakeholders. Particularly in this age of disinformation, it is all too easy for the media to latch onto spin from a medical investigator and disseminate it widely, raising hopes of some unrealistically or even leading to unnecessary harm of others.
Howard Bauchner and Frederick Rivara (Bauchner 2022) point out that, especially during the COVID-19 pandemic era, misinformation and disinformation on social media platforms have heightened the need for scientists to responsibly report the results of their work. They discuss avoiding use of terms and words that should be avoided in communicating results. They say “Statements and adjectives that reflect extremes should be avoided. Few studies are the first of their kind, transformative, critically important, or provide definitive evidence that a treatment cures a disease.” Furthermore, they note it is the responsibility of the investigators to also review for spin any press releases put out by their institutions, funders, or publishing journal. They cite a study (Sumner 2014) that analyzed 462 press releases and their associated peer-reviewed manuscript from 20 leading UK universities and found that 40% of the press releases contained exaggerated advice and 33% causal claims. They also note that the presence of the lay press at large scientific meetings and the propensity for meeting organizers to issue press releases increases the need to consider the potential impact on the public.
They note that for RCT’s (randomized controlled trials), investigators should focus on the preplanned primary and secondary outcomes and acknowledge that any other outcomes are only hypothesis-generating. But they also note that observational studies, which have been increasingly published, don’t have preplanned primary outcomes.
How results are presented is an important consideration regarding spin. Data presented emphasizing relative differences often lead to exaggeration of the importance of the outcome. They make a case that all RCT’s and observational studies should report the number needed to treat (NNT) or number needed to harm (NNH). That better informs the public that not everyone receiving a particular treatment will have a successful outcome.
They also note that publishing the limitations of a study is important but that the media is less likely to be interested in those limitations.
Lastly, they issue a word of caution to those experts and influencers who are asked to comment on the results of a study that they themselves did not participate in. Those individuals “could decline to comment, or make it clear, on the basis of the information presented, that they have a specific opinion, noting that the manuscript has not yet been published. The media can then acknowledge that such comments are based on preliminary non-peer-reviewed results”.
A number of studies have shown that spin is evident in publications, even in some of our most respected journals. One study (Jellison 2019) reviewed results published between 2012 and 2017 in six top psychology and psychiatry journals. Of 116 trials in which the primary results had not been statistically significant, they found evidence of spin in the abstracts of more than half (56%) of the published trials. This included titles (2%), results sections (21%), and conclusion sections (49%). In 15%, spin was identified in both the results and conclusion sections of the abstract. Spin was more common in trials that compared a particular drug/behavioral approach with a placebo intervention or usual care. Interestingly, industry funding was not associated with a greater likelihood of spinning the findings.
That last point is of interest. We would certainly have predicted that industry funding of a study would have been a prime driver of spin. But it is clear that other factors may contribute to researchers spinning their results. These might include prestige, increasing the number of citations, promoting further funding, meeting requirements for tenure, etc. Khan et al. (Khan 2019) speculate on the reasons authors use positive spin. They note that incentives likely play a role. Publication in high-impact journals fosters career advancement and future grant funding. And, of course, studies with “positive” results are more likely to be published than those with negative or neutral results.
Another study (Khan 2019) analyzed 93 RCT’s published in 1 of 6 high-impact journals (New England Journal of Medicine, The Lancet, JAMA, European Heart Journal, Circulation, and Journal of the American College of Cardiology) with primary outcomes that were not statistically significant. Spin was identified in 57% of abstracts and 67% of main texts of published articles. 11% of reports had spin in the title, 38% in the results section, and 54% in the conclusions. Among abstracts, spin was observed in 41% of results sections and 48% of conclusions sections.
Closely related to “spin” is “hype”. Hype has been defined as hyperbolic and/or subjective language that may be used to glamorize, promote, or exaggerate aspects of research. Millar et al. (Millar 2022) recently looked for evidence of “hype” in successful NIH grant applications. They used a technique known as “keyword analysis”, in which the frequency of certain adjectives was used as a measure of “hype”. They found that, among 139 “hype” adjective forms, 130 hype adjectives increased in frequency in grant applications between 1985 and 2020. Only 9 hype adjectives decreased in frequency. Hype most often serves to promote the significance, novelty, scale, and rigor of a project; the utility of the expected outcomes; the qualities of the investigators and research environment; and the gravity of the problem; as well as conveying the personal attitudes of the applicants.
Bauchner and Rivara have once again given this problem the attention it deserves. Reviewers and editors of all our healthcare journals need to do a better job of identifying “spin” and “hype” and eliminating them from publication. And all those interacting with the media, whether they are authors or editorialists or others commenting on publications, need to be wary that the media plays a big role in dissemination of information that may not be meaningful.
Bauchner H, Rivara FP. The scientific communication ecosystem: the responsibility of investigators. The Lancet 2022; 400(10360): 1289-1290 Published:October 15, 2022
Sumner P, Vivian-Griffiths S, Boivin J, et al. The association between exaggeration in health-related science news and academic press releases: retrospective observational study. BMJ 2014; 349: g7015
Jellison S, Roberts W, Bowers A, et al. Evaluation of spin in abstracts of papers in psychiatry and psychology journals. BMJ Evid Based Med 2019: 178-181 Epub ahead of print August 5, 2019
Khan MS, Lateef N, Siddiqi TJ, et al. Level and Prevalence of Spin in Published Cardiovascular Randomized Clinical Trial Reports with Statistically Nonsignificant Primary Outcomes: A Systematic Review. JAMA Netw Open 2019; 2(5): e192622
Millar N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in abstracts of successful national institutes of health grant applications, 1985–2020. JAMA Netw Open 2022; 5: e2228676
Print “PDF version”