Damned lies, journalism and private healthcare

What is it about journalists and numbers? Mathematical incompetence among journalists is, as the American media scholar Steve Maier once put it, ‘legendary”. Actually, Maier was wrong. Far from being legendary, the numerical cluelessness of journalists has been empirically established in a number of studies, including one of my own, soon to be published in the journal Communicatio. My survey of numerical competence among journalists at a major South African daily newspaper found that one in ten stories containing a mathematical element got it wrong. Studies in the US and Canada revealed broadly similar error rates.

Often, numerical errors in news reports are minor and, apart from embarrassing the journalist and publication, inconsequential. But sometimes they matter, as in a May 15 report in The Times about doctors’ responses to the government’s plans to check medical costs increases.

Doctors in private practice, the report warns, will “flee the country in droves” if the government attempts to cap the rates they charge patients. It then cites a survey of doctors, commissioned by the South African Medical Association, which “found that most doctors and specialists in private practice would emigrate if the National Health Amendment Bill were enacted”. But the survey found no such thing; in fact, its methodology precludes any generalised finding of that sort. A basic understanding of statistics and sampling procedures would make this clear.

According to The Times, 60% of doctors who responded to the SAMA survey said they would “consider” leaving the country if the Bill became law. Sounds shocking, but here’s the catch: the survey elicited 2 658 responses. About 1595 of those (60%), said they would consider emigrating. SAMA, a quick phone call established, has about 17 000 members. So in actual fact, only about 9% of SAMA members said they would consider emigrating. And SAMA represents about 70% of the country’s doctors, so the actual percentage of the country’s doctors who said they would consider emigrating is even smaller. Mass exodus? Leaving in droves? I think not.

What’s more, this survey is a classic example of what statisticians call a “self-selecting sample”. You send out an e-mail questionnaire and ask people to respond voluntarily. Those who feel strongly about the issue are more likely to respond than those who don’t. In this case, doctors who mistrust the government’s intensions and those who were thinking about emigrating would be more likely to respond than those who don’t and weren’t. The result is that the sample is skewed, and the results of the survey cannot be scientifically generalised to the whole population of doctors.

(Incidentally: it is also a leap from saying 60% of respondents “said they would consider emigrating” if the proposed legislation were enacted, as the survey found, to “most doctors … would emigrate”, as The Times put it. Another common journalistic fault: exaggeration.)

So the journalist got it wrong. Why does it matter? Well — and this illustrates yet another common problem with journalism — the story has been uncritically picked up by other media, and thus the error is already entering the discourse around the issue of medical costs. It is being cited as fact to bolster one side of an ideological debate, and therefore influencing the national discussion around a very important issue. If the government persists with its meddling in private healthcare, most doctors will leave the country! Except, there is as yet no evicence to support this suggestion. Author Darrell Huff referred to this sort of thing as “lying with statistics“.

News media have a critically important role in informing national debates such as these. But sometimes, misinforming is worse than not informing. Journalists should ask more questions when evaluating statistical data, and media owners should start investing more in numeracy training for their reporters.