Anecdotes versus data in public discourse

“The plural of anecdote is not data” is a phrase well-loved by scientific sceptics. Often attributed to Dr Ben Goldacre, but probably originating with Raymond WoIfinger, the phrase cautions us against the mistake of thinking that what you experience – or what you and your granny or friends experience – might not actually be representative of any significant trend, or give you valuable evidence regarding the causal efficacy or role of something you might regard as significant.

In short, we think we spot patterns where there might be no pattern at all, and we ascribe causality quite casually, based on things like temporal proximity (“I took this drug yesterday, and today I feel better”) and confirmation bias (if you are already sympathetic to a hypothesis, you’ll over-value confirming evidence, and discount contradictory evidence, even if you’re doing so in a biased manner).

So, in a purely scientific sense, we often make the mistake of over-valuing the significance of personal experience, because we’re taking that experience as evidence of something more general, whereas in fact, the experience might be explained in a way that relies on some other general principle, or on something quite specific and subjective.

But as individuals living subjective lives, we are of course invested in those subjective experiences, and less likely to be persuaded by a purely logical and scientific reminder that, in short, we are quite possibly making things up (at least in terms of what we think our experiences mean, never mind the fact that out memories are notoriously unreliable, as Elizabeth Loftus explains in a good TED Talk).

Nevertheless, if we want to persuade people to buy into some policy change, or to simply have productive conversations with them, it’s imperative that we take that subjectivity into account. Even if you have the long-term goal of minimising that subjectivity – and whether or not that’s necessarily the ideal course of action – the short-term challenge of communication requires taking it into account, because of how it frames perceptions and responses.

Eusebius McKaiser recently asked me to come onto his show to talk about the value of anecdotes in public discourse and for policy-generation, which led to these general thoughts, below, on the value and role of anecdotes across these different contexts.

As a starting point, I think my overall perspective is captured well by this: “In theory there is no difference between theory and practice. In practice there is.” (Typically, but falsely, attributed to Yogi Berra.)

In other words, even if we’d ideally see ourselves as a (relatively) trivial data-point amongst billions of others, there’s no way we’re going to do so (or that large numbers of us are going to do so), which means that real-world rhetoric needs to take the subjective importance of anecdotes into account, whether or not they are objectively significant.

This is because anecdotes and personal experience have a framing value. Our psychological modeling of available choices, and our choices themselves, are often contingent on stories or anecdotes – what we’ve heard from others, or experienced in the past ourselves, dictate our future reactions.

And, you can see why this sort of way of engaging with the world comes to us naturally. We haven’t lived with massive data-sets and scientific analysis of that data for most of our lives (in terms of human evolution), so using “stories” to understand the world was all we had, and in general, it worked. And if it didn’t work, nobody would likely point that out to you, because alternative models for explaining the phenomenon in question weren’t available yet.

That speaks to the social and political value of the anecdote, which persists today. However, that value often conflicts with the scientific value of an anecdote, where as much as experience might mean something to the agent who has the experience, it might tell us nothing about other agents, or experiences of that sort in general.

Lived experience can promote understanding of a social, cultural, or economic phenomenon, by helping us to see their impact, or to understand how things matter to people (and what matters to people). So if you want buy-in for a legislative amendment, for example, you have to acknowledge the emotive force of anecdotes, whether or not they are scientifically valid.

In this context, someone who responds like sceptics often do, saying something like “that’s atypical, and just an anecdote” is missing the point, or rather, making a different point. Both of them are good points, but from different perspectives, so we need to be clear on what the purpose of the discussion is, rather than simply set worldviews into battle.

If you are seeking empathy and understanding of a person’s experience, it’s not “just” an anecdote. But if you are seeking a general truth, then it is, because anecdotes can’t typically give us reliable data about the prevalence of a phenomenon, nor its causes or impact.

Why? For various reasons, including:
  • Sample size. Your experience might not indicate a pattern, rather than an exception.
  • Confirmation bias and the availability heuristic. People tend to heavily weigh their judgments toward more recent information, making new opinions biased toward that latest news.
  • The fallibility of human memory (as mentioned above).
  • The presence of multiple possible causal factors, and the fundamental attribution error. The person who cut you off in traffic might have been in a hurry, rather than being rude or hateful. For example, remember the photo of Zuma “snubbing” Obama? We have no idea what happened before or after that shot was taken, yet if you want to confirm your negative impression of Zuma, you assume the worst.
  • Reporting bias. We tend to hear self-selected reports, mostly of people who have had adverse experiences, for example of bad meals at a restaurant, rather than the complimentary ones.
More specifically related to science, anecdotes are unreliable because of factors like:
  • Personal experience uses vague outcome measures, rather than the objective ones we aim for in science. What counts as “success” or “improvement” in personal experience isn’t the same as it would be in a clinical trial, where these things are precisely defined and objectively assessed.
  • Anecdotes don’t allow or rely on controlled observation. Many unknown variables could potentially have affected the perceived outcome, meaning that we can’t make any reliable assumptions about which variable (for example a homeopathic “medicine”) was responsible for any apparent improvement.
  • Regression to the mean. As the joke goes, “if you don’t treat a cold it will last for seven days, and if you treat it it will last for a week.”
  • Placebo effects.
  • As above, reporting bias. Cancer survivor groups who have been taking some naturopathic remedy don’t contain the people who took that same “remedy”, and died.

Case reports (as anecdotes are referred to in science) can of course be valuable, but they are merely a sign or a clue – preliminary indications of where we might want to start looking when we design our trial, which will hopefully give us the robust evidence that the anecdotes suggested might be there.

General issues:

Policy can’t be tailored to individuals (consider insurance, and the logistical nightmare of a policy for just you and your habits) – it has to take aggregate data into account. This is different to our social lives and conversations, where you are dealing with interpersonal and emotional content, rather than an abstracted human or group of humans.

As individuals faced with this agent-neutral, impersonal policy landscape, I’d say that some things are too important to take personally, and that we should try to set aside our subjectivity. Examples of this include:

  • “Vaccine-damaged” children. Your (inaccurate) attribution of causality to the vaccine ends up killing children (and not only children), by decreasing herd immunity. You need to accept the scientific consensus; not cling dogmatically to your anecdotal experience.
  • Blood deferrals for gay men. If it’s true that this is a risk group for contaminated blood, even gay men should support this, whether or not they practice safe sex, because one day they might need a transfusion themselves.  (The issue here should instead be consistency, and whether there shouldn’t also be blood deferrals for hetero people who have anal sex.)
Summary and conclusion

Reflecting on one’s own experiences of can create a self-perpetuating conservatism in the wider discussion around the prevalence and impact of the phenomenon in question. If we’re not open enough to considering our experiences atypical, we’d therefore often not accept evidence that should directly dictate the policy view.

However, a dogmatic “anecdotes are not data” position ignores the fact that there’s a difference between scientifically robust data, and psychologically robust data. Even if the latter is scientifically flawed, empathy demands that we aren’t callous with respect to personal experience.

The podcast of the radio discussion (which didn’t touch on many of the above points, but is nevertheless interesting, is available via Radio702.)


Also published on Medium.