The Evidence Enigma: Correctional Boot Camps and Other Failures in Evidence-Based Policy Making

Evidence-based policy making is very much “in vogue”. The government in the UK has invested in a network of six “what works” centres with the specific endorsement of the most senior civil servant in the UK to ground policy making in evidence. In the USA, the Obama administration has also made serious moves to better connect evidence and policy, for example the recently passed Patient Protection and Affordable Care Act which committed $1.5 billion to evidence-based home visiting.

That said, there remains a major disconnection between what we know about what works and what happens day-in day-out in services. The fact that none of the programmes with the strongest evidence have been taken to scale is proof enough. The holy grail of widespread diffusion of an evidence-based programme remains elusive. Researchers understandably ask the question “Why don’t policy makers act on the evidence?”.

This book makes a helpful contribution to this debate by turning that question on its head, instead asking “How come policies without an evidence base go to scale?”. The particular policy under the microscope is the use of correctional boot camps as an alternative to prison in the USA.

When first introduced, such camps were intended to act as shock treatments for first-time and juvenile offenders to deter them from future crime, similar to the “Scared Straight” programme, and fitted with the “tough-on-crime” rhetoric of the 1980s and 1990s^[1]. Their diffusion was rapid. In 1983 only two states had boot camps, but in just over a decade this had increased to 40. They were intended to reduce recidivism, prison overcrowding and costs, and failed on all fronts. Evidence of their ineffectiveness is pretty conclusive: a Campbell Collaboration systematic review found no significant differences between graduates of boot camps and control groups. Yet despite overwhelming evidence of their ineffectiveness, boot camps spread like wild fire.

Bergin helpfully reminds us that boot camps are not some type of policy anomaly and cites several similar examples: DARE (a widespread drugs education programme), the death penalty, electronic monitoring of offenders, abstinence education – for all of these the evidence is mixed but on balance it suggests they are ineffective with the potential to be harmful. Yet all are widely used.

So why do such policies spread? Bergin took a qualitative and quantitative approach to studying the adoption of the boot camps and, perhaps unsurprisingly, struggles to make firm conclusions about factors that reliably predicted the take-up of this policy. However, some useful insights emerge. A conservative political climate and states with high proportions of young Hispanics and African Americans were more likely to adopt punitive criminal justice policies, of which boot camps were one example. It would appear that states that are early adopters generally of policies were also early adopters of boot camps. A somewhat counterintuitive finding was that some factors that one would assume would have an influence, for example the impact of the media, did not appear to contribute to the adoption of boot camps.

As well as asking a smart question about the adoption of an ineffective policy, Bergin looked at the pattern of the abolition of boot camps. Boot camps reached their peak in the mid-1990s, since when their use has been in decline. It would be nice to think this desistance was because of the evidence, but this does not appear to be the case. The media arguably played a significant role in the contraction of boot camps. The deaths of a number of young people as a result of overly punitive treatment in the camps were widely reported in the media. It is impossible to know whether the media reporting led to the closure of the camps or if the camps were closing and the media reported it, but one might surmise that the negative attention encouraged states to abolish their boot camps.

Bergin helpfully demonstrates that the factors influencing diffusion are not the same as those influencing desistance; one is not simply the inverse of the other, as the example of the media highlights. She also shows clearly that, despite the findings being mixed, myriad political, economic, and social factors alongside the influence of the media unfortunately trump evidence every time.

Bergin should be congratulated for coming up with a simple and smart question that challenges assumptions and looks at a neglected area – the hallmarks of a good PhD. The book is based on her PhD thesis and as such is extremely well referenced, cogently argued and strong on methodological detail. Any doctoral student would do well to read it to realise that long hours spent alone in the garret can lead to book publication. For readers less interested in method, the summary of the evidence, the case studies of two states and the analysis of theoretical perspectives are highly engaging.

Bergin highlights how little crossover there is between policy diffusion research and diffusion of innovation research, and identifies it as an area ripe for further study. Rogers’ (1962) seminal work on the diffusion of innovation over 50 years ago remains relevant today and should be heeded by those of us who bemoan the lack of diffusion of evidence-based interventions. Four elements influence the spread of a new idea: the innovation itself, the communication channels, time, and a social system. Diffusion is the process by which an innovation is communicated over time among the members of a social system. According to the theory, members of the social system rarely evaluate an innovation from a scientific standpoint: rather, they rely on subjective perceptions. Designing an evidence-based practice or programme with the principles of diffusion in mind would likely radically change what programmes look like, how they are communicated and their ultimate reach. Rather than designing a programme, rigorously testing it and then working out how to diffuse it, we should start with diffusion in mind and then build in rigorous evaluation to the process. If we do that, we might not be able to stop ineffective practices getting into policy but we might be able to increase the spread of the effective ones.