If you really want to confuse people about where the country is headed, say it with numbers.

There is a certain magic about numbers and the spells they weave in many heads. Go to most conferences, for example. Even the most brilliant presentation in words may provoke only glassy stares — a PowerPoint slide with numbers will have everyone scribbling notes and eating out of your hand. Most of us, it seems, believe that those who produce numbers must know more than those who deal only in words.

Which may explain why, if you want to draw attention to yourself by attracting media coverage, all you need do is come up with a claim about our society, its politics and economy, and wrap it in numbers. And why, if you do, you will never have to explain or justify how you came up with the numbers. Producing surveys is the social and political equivalent of being a second-hand car seller with a guarantee that no one will ever ask you where the rattletrap has been.

Numbers can be as misleading as words. If a survey or opinion poll uses a faulty method, it will lead us astray, not tell us something we need to know. And, when numbers mislead us about important social trends, or about who people would like as their next president, the costs can be severe. All of which raises the obvious question — why do our media (and many of us) not show the same healthy suspicion about surveys and their numbers as we should about used cars?

This week the South African Institute of Race Relations purported to show that poverty has doubled in the 10 years since 1996. This was widely reported in local and international media

It is a startling finding — while we have a large and pressing poverty problem, very serious and competent economists have been researching this question for years and none have come up with these results. If a single researcher at an institute that does not specialise in this branch of research comes up with totally new findings, we should at least be told how he arrived at them.

So far, however, I have not heard or seen any media attempt to discover from the institute and its researcher how they arrived at the conclusion. I also searched the web — and the institute’s website — in vain for something on how they calculated their figures. By Wednesday afternoon, one of our senior economists — who feels the figures may be way out — had also failed to find the study or the method it used.

So the finding is being widely reported, but we have no way so far of judging its accuracy.

This is hardly the only case of this sort — often we hear sweeping claims about the state of our society, but are not told how those who make them arrived at them.

Right now, the field in which number crunchers are attracting most attention, and in which they can get away with just about anything, is politics in general and the ANC presidential race in particular.

Market-research companies have been trying to make money out of political surveys for a while. One of the top firms produced one before the last general election announcing solemnly that we had a political apathy problem because 13% of the country said they did not plan to vote. If you haven’t yet noticed the absurdity, consider that, according to this survey, 87% of adults planned to vote — I know of no democracy anywhere that comes close to that level of voter turnout.

The survey’s estimate of how many people would vote turned out to be way off the mark. And the claim that a country in which almost nine in 10 people planned to vote was one in which people were apathetic vividly showed up the ignorance of the pollsters.

The presidential race has prompted more of the same — what better way to attract clients and promote yourself than to tell people who is winning? And so both commercial and some academic researchers have conducted polls that purport to tell us who the next ANC president will be.

The most obvious problem with this is that the people do not elect the ANC president — ANC delegates do. And so a public opinion survey cannot possibly tell us who will be chosen in Polokwane.

Another is that many of the polls are clearly not to be taken seriously, but are reported by media as though they were serious contributions to our knowledge. Some examples:

  • First, two surveys last year within a week claimed that Jacob Zuma was supported by, respectively, 27% and 54% of the people. It was impossible for both to be accurate — both were reported as if they were.
  • Second, one of these surveys asked people what they wanted the next ANC president to do. More than 70% said they wanted a president who promoted socialism — more than 60% wanted one who promoted capitalism. Both findings could not be right.
  • Third, if that does not convince you, more than 40% said they wanted Zuma to be president because he came from the same ethnic group as them. But, according to the last census, Zulu speakers (like Zuma) make up only about one-quarter of the population. So at least 15% of the interviewees wanted Zuma because he belongs to their ethnic group, even though he doesn’t. Clearly, this finding is nonsense and should have been treated as such.
  • Fourth, only a few days ago, a survey announced that young people in Soweto support Zuma in large numbers. This time, someone did bother to ask about method — and discovered that the survey had not asked people how they felt about Thabo Mbeki! As a guide to whom people support for president, it is, therefore, entirely worthless. And yet it received much publicity — and was often quoted as a sign of a political trend.

    • In all these cases, our media coverage and our public debate is taking seriously “information” that is either clearly wrong or open to serious question unless the surveyors can tell us what method they used and convince us that they came up with an accurate finding.

    And in all cases, why do our media not bother to ask surveyors some basic questions about how they reached their findings and then publish the answer? Even better, why don’t they also call some specialists and ask them what they think of the research method? If they did, we could all decide for ourselves which of the figures we believe and which we can dismiss.

    Until they do, those of us who want to know what is happing in our country need to take care — don’t believe any survey you see or hear unless the people who conducted it tell you how they did it and convince you their findings are accurate.

    If we want to be better informed, conducting a survey should not mean never having to say you are sorry.

    Author

    • Steven Friedman is a research associate at Idasa and visiting professor of politics at Rhodes University. He is a newspaper columnist and a media commentator on South African politics. His academic speciality is the study of democracy. He wrote Building Tomorrow Today, a study of the trade-union movement, and edited two studies of the South African transition.

      View all posts

    READ NEXT

    Steven Friedman

    Steven Friedman is a research associate at Idasa and visiting professor of politics at Rhodes University. He is a newspaper columnist and a media commentator on South African politics. His academic speciality...

    Leave a comment