Intended for the lay reader, Damned Lies and Statistics does an excellent job of illustrating the pitfalls of failing to exercise critical thinking skills. While somewhat chilling in his value‐neutral presentation of such emotional subjects as child abuse, violent crime, and homelessness, Joel Best makes a compelling case for the importance of separating values and emotions from the seemingly straightforward tasks of enumeration and analysis. Alas, he spends far more time describing the problems than offering practical solutions.
Best implores us not to accept statistics mindlessly, but points out that we are products – and apparently perpetrators – of an innumerate society. Innumeracy, a concept akin to illiteracy, results in uncritical acceptance of most big numbers (“A million, a billion, a trillion, what's the difference?”). Our society, Best asserts, has fetishized statistics to the extent that we believe them to be magical numbers, daring not to question them. Consequently, we tend to equate numbers with facts. To be fair, this is not entirely the fault of a gullible populace. The media also bear some responsibility for this sad state of affairs.
Best points out that, while some statistics are “born bad,” others are “mangled” by innumerate reporters. The cycle begins when innumerate social activists unwittingly feed bad statistics to the innumerate media, who compound the problem by offering “mutated” – a completely different concept from mangled, by the way – statistics to a naÏve and innumerate public. Reporters often just write whatever people tell them, because it's easier than doing independent research when they have deadlines to meet. To further complicate the problem, the media frequently report dueling statistics without bothering to provide any contextual information about their sources.
We generally rely on statistics to identify – and suggest the relative priority of – social problems such as global warming, AIDS, child abuse, etc. However, in stark contrast to government's obsession with economic statistics, it does not collect or publish much about social indicators, which means that much available data is produced by interest groups. So it is extremely important to question the numbers we hear bandied about. Best quite logically ventures that activists have a propensity to overestimate their numbers in order to make their causes seem more urgent. Not surprisingly, those with an interest in preserving the status quo usually manage to counter with smaller numbers, perhaps by defining the problem more narrowly.
In addition to identifying activists among the naÏve producers of bad statistics, Best also nails the cynics. Cynical researchers – knowing that they “can prove anything with statistics” – can easily design surveys to produce the results they want. For example, with contentious issues such as abortion and gun control, activists commonly frame questions to elicit whatever response will “prove” their point. Particularly when measuring diverse attitudes, it would be refreshing to hear the debate focus more on the gray continuum rather than the black and white extremes. Also, when discussing difficult social issues, in addition to how the questions themselves are framed, interpretation is another big piece of the puzzle. Obviously, interpretation is more than just a statistical problem but this book does not offer any detailed solutions other than to preach awareness.
Best asks us to remember that, between the polarities of naÏve acceptance and cynical rejection of statistics, lies the opportunity for thoughtful, critical analysis. While statistical presentations inevitably oversimplify the complex realities of social problems, thoughtful critics trouble themselves to ask questions in order to assess the limitations and usefulness (if any) of the numbers at hand. (In a telling example, Best describes the almost hysterical response to a statistic seeming to suggest that white males were disappearing from the US work force.) Just as we question the validity of unnamed sources in news reports, we should also question the validity of unexplained statistics.
Best advises us to be aware of the limitations of “organizational practices.” Statistics are not politically neutral, but reflect the character of the organization that creates them. Statistics are not facts; rather they represent a series of choices about how to define, count, compile, and present data. These data frequently derive from – and reflect – political conflicts, which can lead to a high stakes effort at manipulation. For example, data derived from the US Census have serious financial implications for the States, in addition to their use in apportioning a finite number of Congressional seats.
In an enlightening chapter, Best discusses the dynamic he calls “Stat Wars”. In a phenomenon that has virtually nothing to do with the numbers themselves, some people feel compelled to defend their guestimates by challenging the motives of anyone who questions them. The best known example from the past decade, which Best discusses at length, was the rancorous debate about attendance at the ambitiously named “Million Man March” on October 17, 1995 in Washington, DC. The author wisely admonishes that, rather than assuming that those who disagree with us have nefarious motives, we might begin by asking how the competing numbers were calculated.
In a list evocative of the seven deadly sins, the book explains several causes of bad statistics:
- 1.
Guessing!
- 2.
Unclear definitions!
- 3.
Unsound data collection methods!
- 4.
Convenience sampling!
- 5.
Transformations!
- 6.
Compound errors!
- 7.
Inappropriate comparisons!
And while the “Dark Figure” may sound like a character from the Harry Potter series, the term actually refers to the unknown number of uncounted instances of crime, homelessness, or other social problem. Amusing, yes, but such mysterious terms don't make statistics any more accessible to the innumerate public. The book warns ominously that mutant statistics take on a life of their own. (cue horror movie soundtrack). It would almost be funny if it weren't so unfortunately true.
To combat the evil forces of bad statistics, Best offers a few sensible questions to help us identify the good ones: “Who created this statistic? Why was it created? How was it created?”. It is our responsibility as consumers to learn to recognize good statistics. First, they are based on more than guesses. Beyond that, they employ clear definitions, reasonable measures, and good samples. How are we to know when this ideal state has been achieved? Unfortunately for lay readers (i.e. most people reading the daily newspaper), asking the right questions will not always bring them closer to satisfactory answers.
