That "breakthrough" study you read about? It might not be such a breakthrough

Follow-up studies get less attention

Many so-called "breakthrough" studies you may read out in the news don't hold up in further studies, according to new research published in PLOS One

Many so-called "breakthrough" studies you may read out in the news don't hold up in further studies, according to new research published in PLOS One.

For the study, researchers reviewed 4,723 primary studies, included in 306 meta-analyses, that examined risk factors related to psychiatric, neurological, and somatic diseases. Those studies and meta-analyses were classified as either relating to lifestyle factors—such as smoking—or non-lifestyle factors, such as genetic risk.

The researchers then reviewed 5,029 news articles included in the Dow Jones Factiva database (which excludes online-only publications, as well as television broadcasts and NPR stories). Of those articles, the researchers found 1,561 that covered a total of 156 primary studies—including 63 lifestyle studies—and five meta-analyses.

The researchers found that initial studies were about five times more likely than follow-up studies to be covered in newspapers, while meta-analyses garnered hardly any newspaper coverage. That's a problem, Brian Resnick writes for Vox, because the researchers also found that just 48.7% of the 156 studies covered by newspapers were later confirmed in a meta-analysis.

Overall, the researchers found that newspapers covered just:

  • 12.8% of initial lifestyle studies and 9.7% of subsequent related research; and
  • 13.1% of initial non-lifestyle studies and 1.2% of subsequent related research. 

The researchers also found that newspapers far preferred to cover studies that had positive findings, even though studies with negative findings are "equally valuable," Resnick writes. Overall, the researchers found that the newspapers they assessed "never covered initial studies reporting null findings and rarely reported subsequent null observations."

In addition, the researchers found that journalists generally preferred to cover studies relating to lifestyle choices, such as diet or exercise—particularly when they were published in prestigious journals. Studies on non-lifestyle topics, such as brain imaging or genetics, were less frequently covered.  

The researchers highlighted a 2003 study on depression, originally published in Science, as an example. While 50 newsarticles were produced about a 2003 study, not one newspaper in the Dow Jones Factiva database covered any of the 11 subsequent studies that tried and failed to reach the same conclusions as the original. And when scientists finally published a meta-analysis of those subsequent studies in JAMA in 2009, there were only four newspaper articles about it—and newspapers overall continued to cite the findings of the 2003 study.

Researchers and experts said the study shows how the media and scientists fall short in conveying the significance of medical research to the public. They also say the study suggests "that most journalists from the general press do not know or prefer not to deal with the high degree of uncertainty inherent in early biomedical studies."

The researchers recommended that journalists covering new studies contact the study authors to determine whether the study is an initial finding and, if so, specify in their coverage that the study findings are tentative and need to be validated by further research.

As Resnick explains, part of the issue is that "journalists have a need for digestible headlines that convey simple, accessible, and preferably novel lessons." But university press releases also contribute to the problem, Resnick argues, because they are more likely to highlight a "striking and dramatic single study" compared to a more definitive "meta-review." Such press releases can also "spread hype" about research, he writes.

Somewhat fittingly then, the study's lead author Estelle Dumas Mallet, a biologist at the University of Bordeaux in France, said in an email to Vox that the study was only preliminary—and limited to a comparatively small sector of science media coverage. "Our result only refers to a small sample of the scientific research," she wrote. "Also, we cannot extrapolate these results to other domains such as physics and chemistry" (Harris, NPR, 3/6; Resnick, Vox, 3/3). 

Which universities are leading the research pack? It depends on who you ask.

  • Manage Your Events
  • Saved webpages and searches
  • Manage your subscriptions
  • Update personal information
  • Invite a colleague