Good Studies Go to the Back of the Bus

It's a rare day when my daily newspaper doesn't include at least one medical or health related article. My subjective impression is that they frequently report on potential “breakthroughs,” but many of them are never heard of again, suggesting that the early results were not reproducible.

A new study by Senthil Selvaraj and two colleagues suggests that newspapers do not publish the best available studies. In medical research, the main criterion of a good study is whether participants were randomly assigned to receive either the treatment or some control procedure such as a placebo. In medical jargon, this is called an RCT study, which stands for randomized controlled trial. The major alternative is an observational study, in which the participants are contrasted with a comparison group that may differ from them in uncontrolled ways (a cross-sectional study), or are compared to themselves at an earlier time (a longitudinal study). Some observational studies are merely descriptive and lack a comparison group.

© mercatornet.com

The authors selected the first 15 articles that dealt with medical research using human subjects published after a predetermined date in each of the five largest circulation newspapers in the US. Referring back to the original research reports, they classified each study on several dimensions, the most important being whether it was an RCT or an observational study. For comparison, they selected the first 15 studies appearing in each of the five medical journals with the highest impact ratings. These impact ratings reflect how often studies appearing in these journals are cited by other researchers.

The main finding was that 75% of the newspaper articles were about observational studies and only 17% were about RCT studies. However, 47% of the journal articles were observational studies and 35% were RCT studies. A more precise rating of study quality using criteria developed by the US Preventive Services Task Force confirmed that the journal studies were of higher quality than the studies covered by the newspapers.

They also found that the observational studies that appeared in the journals were superior to the observational studies covered by the newspapers. For example, they had larger sample sizes and were more likely to be longitudinal rather than cross-sectional.

In one sense, these results are not a surprise. We could hardly have expected newspaper reporters to be as good a judge of study quality as the editors of prestigious medical journals. The authors, like many before them, call for more scientific literacy training for newspaper reporters, but it's hard to be optimistic that this will happen.

What criteria do the reporters use in selecting studies to write about? I was struck by the fact that observational studies resemble anecdotes more than RCT studies do. In addition, the newspapers chose observational studies with smaller sample sizes. These results could be driven by the base rate fallacy—the fact that the average person finds anecdotes more convincing than statistical analyses of much larger samples. In fact, the lead paragraph of these stories is often a description of some John or Jane Doe who received the treatment and got better. The results could mean either that reporters fall victim to the base rate fallacy, or that they think their readers are more interested in anecdotal evidence.

You may also be interested in reading:

Letters, . . . We Get Letters

Death By Anecdote, Part 1

Death By Anecdote, Part 1

You can lie with statistics, but a well-chosen anecdote is much more effective.

Here's something to look forward to, right along with your next colonoscopy.  The New York Times reports that the Republican Party plans to carry out a sustained, organized attack on the Affordable Care Act (ACA) for the next year, in the hope of gaining an advantage in the 2014 elections. The Republican campaign, described as a “multilayered sequenced assault,” is outlined in the House Republican Playbook, a 17-page strategy document prepared by their House leadership. It lists a series of talking points such as: “Because of Obamacare, I lost my insurance,” “Obamacare increases health care costs,” and “The exchanges may not be secure, putting personal information at risk.” House members are advised to collect anecdotes from constituents in support of these talking points through social media, letters and visits to their home distract. A new website, gop.gov/yourstory, centralizes the collection of these anecdotes. Republicans are instructed in the use of “messaging tools” for disseminating the stories, for example, a sample op-ed for submitting to local newspapers.

The idea is to flood the media with anecdotes in support of a particular talking point. If there is an effective counterresponse from Democrats, they will shift immediately to a different talking point. Topics waiting in the wings for possible use include insurance “rate shocks,” threats to being able to keep your doctor, and possible changes to Medicare Advantage policies.

The Republicans recognize that anecdotes can have a powerful influence on public opinion. When making inferences, people use judgmental heuristics, or mental shortcuts to make decisions quickly and easily. The use of heuristics is automatic and unconscious.  They usually lead to correct inferences, but they sometimes lead us astray.

One inference we often make is to estimate the size of a category or the frequency of an event. For example, how many Americans are being harmed by the ACA? The availability heuristic suggests that the size of a category is judged by the ease with which examples can be brought to mind. Examples are more easily retrieved from memory if they are concrete rather than abstract, if they are dramatic and interesting, or if they happened recently or nearby. Personal experiences are particularly salient. When people are asked to estimate the frequency of various causes of death, they overestimate homicides and auto accidents, but underestimate strokes and diabetes. Clearly, their estimates are influenced by media coverage.

The problem gets worse when you consider the base rate fallacy, which states that people are inattentive to population statistics, and their judgments are not sufficiently affected by them. In one study, participants were given vivid stories about misbehavior by prison guards or welfare recipients. Attitudes toward these groups were equally negatively affected regardless of whether they were told that the anecdotes were typical of the population, not typical of the population, or they were given no base rate information. In another study, college students' intentions to take courses were affected by single brief face-to-face comments from a stranger, but hardly at all by statistical summaries of the course evaluations of much larger numbers of students who had previously taken the course.

The availability heuristic and the base rate fallacy suggest that even if people are given accurate information suggesting than the story is unrepresentative, their false impression is unlikely to be corrected. There was heavy media coverage of the first wave of anecdotes from people who claimed that their insurance costs went up due to Obamacare.  When critics examined them more closely, many of these anecdotes were found to be misleading. Insurance companies cancelled policies and raised rates long before the ACA. The percentage of people whose policies were cancelled was small. Some of these people were able to get equal or better insurance through the exchanges without paying more. However, the corporate media can't be counted on to investigate anecdotes before airing them, and the debunking stories seldom receive anywhere near the attention given to the original report.

The Obama administration has apparently decided that the best defense is a good offense, so they are responding with anecdotes of their own—so-called Obamacare “success stories.” While this may be the best they can do under the circumstances, they are unlikely to get much cooperation from the corporate media in publicizing these stories. The media don't cover successful airplane landings--unless you land it in the Hudson River. Meanwhile, the administration and the media have largely overlooked another potential source of much more tragic stories: the 5.2 million people who are being denied health insurance entirely because they happen to live in states where Republican governors and legislatures have blocked Medicaid expansion. But to the corporate media, an upper middle class person losing a few dollars a month is much more newsworthy than a poor person losing his or her life.

But remember, the plural of anecdote is not data. To be continued.

You may also be interested in reading:

Death By Anecdote, Part 2

Policy By Anecdote

The Coverage Gap:  The Real Failure of Obamacare