Lessons in bad science – Tim Noakes and the SAMJ

The SAMJ has published a “study” by Prof Tim Noakes, in which 127 self-reported anecdotes are presented as evidence of the need for a trial.

As I’ve said before, Professor Tim Noakes might well be right in his advocacy for a low carb, high fat diet. Whether he is right or wrong is however – to some extent, at least – a separate issue from whether he’s providing an example of sound scientific reasoning in reaching the conclusions he does.

It’s understandable that people outside of science, academia, or any overtly evidence-based work might confuse personal experience, anecdotes and badly drawn samples for good evidence. However, the principles of scientific reasoning (even as taught to first-year students, as I’ve done for years), remind us of these and other common errors. Once you know about confirmation bias, you shouldn’t flaunt the fact that you deploy it, but rather regret that you (like everyone) are prone to quite typical errors.

As a practitioner of evidence-based something-or-other, you’d know and understand counter-intuitive things, for example that a conclusion can sometimes be the correct one, regardless of how poor the argument for that conclusion is. And, that a good argument can sometimes result in a false conclusion.

Most important, perhaps, is that you’d know of the various resources we have for making it more likely that we reach the correct conclusion, and you’d know of the sorts of errors to avoid because they tend to lead us to false conclusions. And given that you care most of all for getting closer to the truth of the matter, you’d be aware of your responsibilities to set an example to others of sound reasoning about science – especially if you are perceived to be an authority figure.

When a scientist like Noakes seems willing to ignore most of these principles in favour of promoting a conclusion – through whatever support comes along – he doesn’t sound passionate, committed, maverick and so forth. He sounds like a quack (and sometimes, even a conspiracy theorist, at least when citing some of the reasons why his “obvious” (to him) truths are somehow not adopted by everyone). To Noakes, all research that disagrees with him is “industry-funded” or otherwise tainted, but never treated as counter-evidence.

Among those that Noakes has quoted in support of either his view or his approach to science are the evolution-denier and proponent of scientific racism that is Louis Agassiz, and the anti-vaccine Weston Price Foundation. He’ll tweet links to pieces arguing that carbohydrates might have played a role in the Newtown shootings. And when challenged on whether his approach is sound, he says things like this:

Worst of all, he doesn’t seem to really understand what the stakes are, even as he speaks of a decades-old diet lie that is apparently killing us all. He’s embraced the false dichotomy that we have a choice between the “orthodox” dietary advice (that is harming us) and his model (that will make us thin, cure diabetes and so forth). On his model, choosing option B (his model) has no risks, even though we have no long-term data regarding the possible harmful consequences of a high-fat diet.

This doesn’t mean people won’t be persuaded by Noakes. They will be and are being persuaded in their droves, thanks partly to positive short-term results on the diet, and also to some very engaged PR on Twitter, radio, magazines and television. But they’re also being persuaded because – like most people – they are prone to overstating the value of anecdotes in supporting a conclusion.

The fact that they are persuaded is not, however, further evidence for your conclusion.

Yet, according to Noakes, perhaps it is. And sadly, South Africa’s flagship medical journal, the SAMJ, has published a Noakes paper that is a textbook example of bad science in this and other ways. Somehow, this paper got through peer-review, but I can pretty much guarantee you that if we did a Sokal-type experiment, submitting this under fictitious names, it would never get published. It’s solely on the basis of the authority of the “name” of an acclaimed scientist that it was published as a study, rather than on the letters pages or somewhere less notable.

Here’s the 140-character summary of why this shouldn’t have been published as a study, or made it through peer review:

Basically, the “study” is confirmation bias run amok. If I were a UFO conspiracy theorist, and led a society dedicated to discussing our abduction experiences, the letters I might receive from prospective members might appear to me to be further evidence that UFOs exist. But I’d be wrong. They’re not, because my letter-writing audience are self-selected (we can predict, in advance, that they are likely to share my conclusion) and unreliable (they are offering anecdotes, and we have no idea how reliable their testimony is).

People who try the “Noakes diet” to no effect will not write letters to him saying so. And as for those who do write, we have no knowledge (or control) over any other factors relevant to their weight-loss experiences. It’s plausible that someone trying a new diet will, for example, also be motivated to exercise more – but we can’t control for that.

In response to these and other criticisms on Twitter, Noakes asked his critics why we were focusing on the anecdotes, rather than the controlled trials that agreed with the anecdotes. But this again misses the point – one wouldn’t need the anecdotes to make the case if the data did so already. To include the anecdotes means that you see them as adding some value, and you shouldn’t do so.

It’s not everyone else’s job to defeat your anecdotal evidence (that should never even be presented). It’s your job to gather non-anecdotal evidence. And if you don’t have that as yet, it’s not someone else’s responsibility to get it for you, as in comments like

(This, because of another elementary principle of scientific reasoning, namely the burden of proof.)

To wrap this up: I agree that a trial is needed. But there’s a different issue at stake also, and this is that Noakes is encouraging incompetent and pseudoscientific thinking on matters of science, and encouraging a cultish adherence to a model that hasn’t yet been scrutinised in full, and for which we have no long-term data. This is irresponsible, and an abrogation of his responsibility as a scientist and an educator.

But these are no doubt all irrelevances to Noakes, even perhaps evidence of bias or conspiracy. After all, at the end of the day, the ultimate evidence is remarkably elegant: the thinnest person in the argument wins, and obese people can’t be right.

P.S. Prof Noakes has left a substantial comment below. Unfortunately, it does his cause absolutely no good, as I explain in this follow-up post.

14 Pingbacks/Trackbacks