Problems with evidence-based medicine aren’t a license to make stuff up

In the paper that brought the idea of evidence-based medicine (EBM) to prominence, Professor David Sackett et. al. wrote that

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research. By individual clinical expertise we mean the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice. … By best available external clinical evidence we mean clinically relevant research, often from the basic sciences of medicine, but especially from patient centred clinical research into the accuracy and precision of diagnostic tests (including the clinical examination), the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens.

In short, EBM is about integrating the best available research knowledge with the expertise of clinicians, who might – and often do – spot something that a model or a manual might not recognise as significant. As much of a naturalist as you might be – and I’m a very committed one – raw data exists in a context, and the context might often be a significant clue in telling you what data are relevant, and how they should be interpreted.

downloadSo I share some of Dr. Malcolm Kendrick’s concerns, when he writes that “EBM is now almost completely broken as a tool to help treat patients” thanks to the “evidence” being susceptible to corruption by vested interests, and pharmaceutical companies in particular. If you fund enough research, and suppress results that you don’t like, it’s certainly possible to end up with all the “evidence” pointing in a favourable direction. Favourable for you, that is, but not for the patient.

But the fact that something is funded by a pharmaceutical company doesn’t guarantee bias. There’s a difference between being cognisant of potential biases, and writing something off in advance, just because of whence it came. That sort of pre-emptive dismissal is a logical error called “the genetic fallacy“, and you can tell when it’s being committed if someone stops paying attention to the evidence at all, or starts claiming that they don’t need to even bother doing so. Like this, perhaps (from Kendrick’s post):

Some years ago I stated that I no longer believe in many research papers that I read. All I tend to do is look at the authors, look at the conflicts of interest, look at the companies who sponsored the study, and I know exactly what the research is going to say – before I have even read the paper.

I have also virtually given up on references. What is the point, when you can find a reference to support any point of view that you want to promote? Frankly, I do not know where the truth resides any more. I wish to use evidence, and the results of clinical studies, but I always fear that I am standing on quicksand when I do so.

We are at a crisis point. Medical research today (in areas where there is money to be made) is almost beyond redemption. If I had my way I would close down pubmed, burn all the journals, and start again, building up a solid database of facts that we can actually rely on – free from commercial bias. But this is never, ever, going to happen.

It’s rather alarming to see the person responsible for writing The Great Cholesterol Con – and for encouraging most of us to stop taking statins to lower cholesterol – professing “I do not know where the truth resides any more”. (If he really means that, a career in anthropology rather than medicine might suit him better, I’d suggest, in that he’s already learnt a key mantra of the field.)

Another peculiar thing you’ll find on his website represents quite a cunning stratagem. You see, he’s talked himself into a bit of a bind with this “don’t trust The Man” stuff – if he ever wants to sell you something like a drug, how could he offer you “evidence” in support of it’s efficacy, assuming that he or someone else with a vested interest was involved in that research?

Easy – by redefining what conflict of interest means. For him only, mind you – not for others, where it means that you can’t trust them, and don’t even have to read them to know that you can’t trust them. In his “Disclosure of Interest” page, he notes “I have become the medical director for a company making a heart health supplement called ProKardia”, for whom he does paid consultancy work. And here’s the cunning bit:

If I do write about ProKardia or any of the ingredients in ProKardia, in a positive light, you need to know that I have a financial interest. I did not use the word conflict of interest in this statement, as I do not believe I have a conflict. I have become involved in developing, and using, a product that I entirely believe to be a good thing.

Got that? He has a financial interest alone, but no conflict. Authors of papers with connections to something he doesn’t trust, or research sponsored by pharmaceutical companies, always represent a conflict – and you don’t even have to read the paper in question to know this. (Which, amusingly, means that even if they were to try to insert the same Humpty Dumpty clause into their papers, Kendrick would never read that either.)

I say “Humpty Dumpty clause”, because the book Alice in Wonderland contains this fabulous line: “‘When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.'”

It’s the same causal relationship to what words – and evidence – means that allows for claims that sugar is addictive (as addictive as cocaine, according to some), or that Harvard have endorsed low-carb high-fat diets. They haven’t – they’ve just said that carbs can sometimes be worse than fats, particularly the “bad”, saturated fats – yet the LCHF proponents cherry-pick this as support, even though the same group are explicit that the type of fat matters, and that animal fats remain something to be cautious about.

Or, it’s this casual relationship that allows for claims that Sweden has officially adopted the LCHF diet, even though the report – now finally available in English (the summary, at least), though that didn’t stop English speakers like Tim Noakes from relying on blog sources as authoritative on what it said – actually says (my emphasis):

in the short term (six months), advice on strict or moderate low carbohydrate diets is a more effective means of achieving weight loss than advice on low fat diets. In the long term, there are no differences in the effect on weight loss between advice on strict and moderate low carbohydrate diets, low fat diets, high protein diets.

I realise that this is just one element of the debate, which includes diabetes, heart disease and so forth. But these are two examples of scientists cherry-picking data that happens to support something they want to argue for – or that can be made to appear to support it, so long as you whip up enough hysteria, or persuade them that there’s some conspiracy afoot.

As Mark Wallace points out in piece for The Guardian on “addictive” sugar, “if the issue is people failing to act responsibly, then it won’t be rectified by treating them like children”. To that, we can add “treating them like fools”.

And when the same people doing the cherry-picking are giving themselves licence to simply ignore research they don’t like, and to dismiss conflict of interest issues (in their case, but not for you) with a regal wave, they stop being scientists, and start becoming shills.

2 Pingbacks/Trackbacks

  • Leon Retief

    I would not go so far as to say that EBM is broken but it certainly has a couple of flat tires and an ominous knock in the engine – I can understand that some people may become cynical. You are probably aware of a paper by an author whose name escapes my mind now – Greek-sounding surname – which stated that most published medical papers are useless. As a clinician who has been involved in sometimes acute, emergency decisions for many years I know all too well how difficult it is to use the iiterature to arrive at he correct course of action.

    • Sure, there are big problems (see Ben Goldacre’s Bad Pharma for a catalogue of them). Of course, EBM isn’t only about the literature, as per the definition I posted. But most of all, I concur that there are flaws – my post was drawing attention to one sort of serious over-reaction to those flaws.

      • Leon Retief

        I have read Goldacre’s book. The thing is, for people like me who work at the coalface of medicine, EBM is all about literature because that (as well as congresses and so on) is where we get our information. I agree though that overreactions like that are plain silly.

  • George

    I would contend that EBM need not always be based on interpretation and exaggeration of weak effects. For example, evidence that some new oral combinations of HCV antivirals are more effective and significantly safer than the current SOC regime. Even at the point of approval, where things stand now, the difference is obvious. I am sure there are many other examples like this in the areas of antibiotic medicine (in the wider sense) and emergency medicine, where the evidence is not trivial or contradictory.
    When weak evidence can only be used to worry people into doing things that may well do more harm than good, or that have significant costs – basically gambling on a long shot – then it would perhaps be better not to bother us with the news. Where it can inform clinicians who understand the nuances of individual trials, before these are crushed into meta-analyses that obscure their differences, and are able to extract the nuggets that may be hidden therein, well and good.
    The fundamental position of Noakes and Kendrick is a default one: we have been fed a farrago of exaggeration and nonsense to promote the message that we need to stop doing something we once took for granted, that is, including animal fats in the diet as a health measure. If there is insufficient evidence, or indeed only weak evidence (take your pick) to justify a change to the dietary advice formerly given in this regard (beginning with Sir John Boyd Orr, Sir Edward Mallenby and so on), then the effort to undermine animal-based nutrition needs to stop.
    Promoting LCHF, or ProKardia, as answers to health problems does require a high standard of evidence. Not exactly lacking in the case of LCHF with regard to diabetes, but some people are better than others at reading and presenting evidence, which is why lawyer is a profession not a hobby. But defending an existing practice against already weak or contradictory evidence, or redeeming it once intervention based on that evidence has consistently failed, should be an easier task than it is.

  • Pingback: On causes, Noakes, and making prudent claims related to diet()

  • Pingback: On causes, LCHF, and making prudent claims related to diet - Towards a Free Society | Towards a Free Society()