View as “PDF version”
Patient Safety Tip
of the Week
November
8, 2022
Spinning
It
Our
February 16, 2010 Patient Safety Tip of the Week “Spin/Hype…Knowing It When You See It” discussed in detail the problem of “spin”.
Spin is the manipulation of messaging when the primary endpoint of a clinical
trial is not met but results are presented in a positive light. For example, a
paper may focus on analysis of a subgroup that fared better or on some
secondary endpoint that was met. Post hoc subgroup analysis should only be
hypothesis-generating and subject to a new clinical trial to test that
hypothesis. Or they might focus on a result that is statistically significant
but not clinically significant. Or they might use composite outcomes, which are
especially likely to give rise to inappropriate conclusions when the outcomes
are driven by one component of that composite, when that component is not as
clinically significant as other components.
Unfortunately,
reviewers and editors of our medical journals are not doing a very good job of
preventing “spin” in published articles. In our February 16, 2010 Patient
Safety Tip of the Week “Spin/Hype…Knowing It When You See It” we gave multiple examples, including one
study rife with spin that led to publication in not one, not two, but three
“respected” medical journals.
Recently,
two physicians who have served as editors-in-chief of several JAMA journals
have spoken out about spin and the need for responsible reporting of trial
results to the media and other stakeholders. Particularly in this age of
disinformation, it is all too easy for the media to latch onto spin from a
medical investigator and disseminate it widely, raising hopes of some
unrealistically or even leading to unnecessary harm of others.
Howard
Bauchner and Frederick Rivara (Bauchner 2022) point out that, especially during the
COVID-19 pandemic era, misinformation and disinformation on social media
platforms have heightened the need for scientists to responsibly report the
results of their work. They discuss avoiding use of terms and words that should
be avoided in communicating results. They say “Statements and adjectives that
reflect extremes should be avoided. Few studies are the first of their kind,
transformative, critically important, or provide definitive evidence that a
treatment cures a disease.” Furthermore, they note it is the responsibility of
the investigators to also review for spin any press releases put out by their
institutions, funders, or publishing journal. They cite a study (Sumner 2014) that analyzed 462 press releases and their
associated peer-reviewed manuscript from 20 leading UK universities and found
that 40% of the press releases contained exaggerated advice and 33% causal
claims. They also note that the presence of the lay press at large scientific
meetings and the propensity for meeting organizers to issue press releases
increases the need to consider the potential impact on the public.
They
note that for RCT’s (randomized controlled trials), investigators should focus
on the preplanned primary and secondary outcomes and acknowledge that any other
outcomes are only hypothesis-generating. But they also note that observational
studies, which have been increasingly published, don’t have preplanned primary
outcomes.
How
results are presented is an important consideration regarding spin. Data presented
emphasizing relative differences often lead to exaggeration of the importance
of the outcome. They make a case that all RCT’s and observational studies
should report the number needed to treat (NNT) or number needed to harm (NNH). That
better informs the public that not everyone receiving a particular treatment
will have a successful outcome.
They
also note that publishing the limitations of a study is important but that the
media is less likely to be interested in those limitations.
Lastly,
they issue a word of caution to those experts and influencers who are asked to
comment on the results of a study that they themselves did not participate in.
Those individuals “could decline to comment, or make it clear, on the basis of the information presented, that they have a
specific opinion, noting that the manuscript has not yet been published. The
media can then acknowledge that such comments are based on preliminary
non-peer-reviewed results”.
A number of studies have shown that spin is evident in
publications, even in some of our most respected journals. One study (Jellison 2019) reviewed results published between 2012 and
2017 in six top psychology and psychiatry journals. Of 116 trials in which the
primary results had not been statistically significant, they found evidence of
spin in the abstracts of more than half (56%) of the published trials.
This included titles (2%), results sections (21%), and conclusion sections
(49%). In 15%, spin was identified in both the results and conclusion sections
of the abstract. Spin was more common in trials that compared a particular
drug/behavioral approach with a placebo intervention or usual care.
Interestingly, industry funding was not associated with a greater likelihood of
spinning the findings.
That
last point is of interest. We would certainly have predicted that industry
funding of a study would have been a prime driver of spin. But it is clear that other factors may contribute to researchers
spinning their results. These might include prestige, increasing the number of
citations, promoting further funding, meeting requirements for tenure, etc.
Khan et al. (Khan 2019) speculate on the reasons authors use
positive spin. They note that incentives likely play a role. Publication in
high-impact journals fosters career advancement and future grant funding. And,
of course, studies with “positive” results are more likely to be published than
those with negative or neutral results.
Another
study (Khan 2019) analyzed 93 RCT’s published in 1 of 6
high-impact journals (New England Journal of Medicine, The Lancet, JAMA,
European Heart Journal, Circulation, and Journal of the American College of
Cardiology) with primary outcomes that were not statistically significant. Spin
was identified in 57% of abstracts and 67% of main texts of published articles.
11% of reports had spin in the title, 38% in the results section, and 54% in
the conclusions. Among abstracts, spin was observed in 41% of results sections
and 48% of conclusions sections.
Closely
related to “spin” is “hype”. Hype has been defined as hyperbolic and/or
subjective language that may be used to glamorize, promote, or exaggerate
aspects of research. Millar et al. (Millar 2022) recently looked for evidence of “hype” in
successful NIH grant applications. They used a technique known as “keyword
analysis”, in which the frequency of certain adjectives was used as a measure
of “hype”. They found that, among 139 “hype” adjective forms, 130 hype
adjectives increased in frequency in grant applications between 1985 and 2020.
Only 9 hype adjectives decreased in frequency. Hype most often serves to
promote the significance, novelty, scale, and rigor of a project; the utility
of the expected outcomes; the qualities of the investigators and research
environment; and the gravity of the problem; as well as conveying the personal
attitudes of the applicants.
Bauchner
and Rivara have once again given this problem the attention it deserves.
Reviewers and editors of all our healthcare journals need to do a better job of
identifying “spin” and “hype” and eliminating them from publication. And all
those interacting with the media, whether they are authors or editorialists or others
commenting on publications, need to be wary that the media plays a big role in
dissemination of information that may not be meaningful.
References:
Bauchner
H, Rivara FP. The scientific communication ecosystem: the responsibility of
investigators. The Lancet 2022; 400(10360): 1289-1290 Published:October
15, 2022
https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(22)01898-0/fulltext
Sumner
P, Vivian-Griffiths S, Boivin J, et al. The association between exaggeration in
health-related science news and academic press releases: retrospective
observational study. BMJ 2014; 349: g7015
https://www.bmj.com/content/349/bmj.g7015
Jellison
S, Roberts W, Bowers A, et al. Evaluation of spin in abstracts of papers in
psychiatry and psychology journals. BMJ Evid Based Med 2019: 178-181 Epub ahead of
print August 5, 2019
https://ebm.bmj.com/content/25/5/178.long
Khan
MS, Lateef N, Siddiqi TJ, et al. Level and Prevalence of Spin in Published
Cardiovascular Randomized Clinical Trial Reports with Statistically
Nonsignificant Primary Outcomes: A Systematic Review. JAMA Netw Open 2019; 2(5):
e192622
https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2732330
Millar
N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in
abstracts of successful national institutes of health grant applications,
1985–2020. JAMA Netw Open 2022; 5: e2228676
https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2795635
Print
“PDF
version”
|
|
http://www.patientsafetysolutions.com/