Neither monsters nor saints [1]

Article Type: Editorial From: Journal of Children's Services, Volume 8, Issue 3.

The backlash is in full flow. New and exciting for many, evidence-based programmes (EBPs) have become the scourge of others in children's services in the UK. In our view it is rather like protesting against Cornwall's nationalist movement – it is not as if there is much to resist. Most EBPs here still resemble “pop-up” shops – here today but gone tomorrow, barely surviving on the fringes of the high street of major public service systems. They are hardly taking over!

Leaving this aside, we recognise that there are enough pronouncements and, to a lesser extent, policy developments, to signal the likely increased use of such interventions. The birth of the Early Intervention Foundation, the expansion of Family Nurse Partnership, the creation of the Alliance for Useful Evidence, the views of the country's leading civil servant, Sir Jeremy Heywood, on evidence – they all point, it seems, in one direction: if not quite “the future”, EBPs are gaining traction.

In this edition Harriet Churchill describes these developments, among others, in her overview of children's services policy and practice in the past decade or so, and particularly under the Coalition Government. Of course, it is not a trend unique to the UK: as Hans Grietens demonstrates, also in this edition, other European countries are also investing more in EBPs –though again, not with universal enthusiasm: states with social-democratic or Catholic welfare regimes tend to be instinctively sceptical about interventions developed in the USA, where most such interventions originate.

It seems timely, then, to reflect on some of the main critiques of EBPs and, if possible, to separate myths and straw-man arguments from concerns that deserve proper attention. We generally support the greater use of EBPs in children's services because they have considerable potential to improve child well-being. But we also think they have limitations and are certainly no panacea. To use an anthropomorphic metaphor, they are neither monsters nor saints. So, what are some of the main critiques?

For most proponents of EBPs, the designation “evidence based” tends to require evidence from a particular evaluation method,namely the randomised controlled trial (RCT). Critics argue that while this rather narrow method might be suitable in medicine for testing pills, it is rather less suitable for testing complex social interventions addressing complex problems in complex conditions.

A related scientific critique is that the RCTs used to claim that a given programme is “evidence based” are often fatally flawed – badly designed, badly executed, and badly reported. The resulting combination of bias and obfuscation means that such programmes rest less on foundations of rock than on sand: the results simply cannot be trusted. A particular concern is the tendency often to “cherry-pick” positive findings.

From a more ideological perspective, it is argued that EBPs embody a neoliberal approach to children's services that emphasises the role of the market. They are part of a wider set of policy reforms designed to reform a feckless underclass and generate private profit. Such programmes, it is claimed, tend to be intrusive, time-limited, didactic, risk-obsessed and moralising. They favour individualised solutions to societal problems, overlooking the role that inequality plays in creating and perpetuating disadvantage.

Next, some professionals and their advocates maintain that EBPs “de-professionalise” practitioners because the requirement to implement such programmes with fidelity – interpreted as stickling rigidly to manuals and guidelines – denies professionals opportunities to tailor what they do in response to the needs and wants of service users. Put another way, EBPs present a“cookie-cutter” or one-size-fits-all approach to services – providers become robots and users become passive recipients in what should be a much more human and reflexive enterprise.

Still the brickbats come. Another commonly voiced worry is that EBPs stifle innovation: big well-resourced programmes will flood the market and squeeze out promising “home-grown” initiatives – many originating in the voluntary and community sector.

Then there is the view that EBPs do not translate well across cultures. Specifically, a programme tested and found effective in one country may not – perhaps even cannot – work in another country. What works in Seattle would not work in Stoke, Sweden or Singapore: the cultures and the needs to be addressed are different.

Lastly, most EBPs, it is suggested, were developed and tested in rarefied conditions and do not translate into real-world children's services systems. Their developers were heavily involved in delivery, resources were plentiful, recipients were screened within an inch of their lives to ensure homogeneity of needs and the setting was essentially a university laboratory. A million miles from, say, your average social services department.

Rules of engagement

We do not have the space here to appraise these critiques, indeed some responses are set out elsewhere (Axford and Morpeth, 2013). For now we want to tentatively suggest some rules of engagement in what will be an ongoing debate. We do so in the hope of generating more light than heat and ensuring that improving children's well-being remains a (if not the) paramount goal.

First, we should seek to represent evidence and opinions accurately and fairly. It is tempting to over-egg our case and misrepresent those who take a different view. A good example is the use of neuroscience in this area. In our view it has its place but some advocates and critics alike of EBPs have exaggerated its importance to the case for evidence-based prevention and early intervention.

Second, we need a broad framework, if not detailed criteria, to assist with judging whether an intervention can fairly be said to be effective. Personally, we have no confidence in a claim supported by anecdotes alone that an intervention “works”. Equally, we have no confidence in an RCT that analyses programme completers only, or has high or uneven drop-out, or fails to report how randomisation was done.

Third, we need to look at the evidence before jumping to conclusions based on ideology or professional experience. Take the issue of the “transportability” of EBPs. For us this is essentially an empirical question. Anyone can speculate why programmes may or may not work in different places, and why, but we want to know what happens when such hypotheses are tested. Does a given programme, or class of programmes, work across cultures? If so, why, and if not, why not?

Fourth, we should accept that there are nuances to the debate and both seek and recognise positions that are more sophisticated than “for or against” or “either/or”. It is possible, for example, to be broadly in favour of EBPs but also in favour of innovation– not least because all EBPs are, ultimately, products of innovation. It is possible to find the fertile ground between fidelity and versatility that preserves elements core to the intervention's success while permitting the adaptations necessary for its wider take-up.

We hope that the Journal of Children's Services can be a home for a constructive debate on these and similar issues and welcome submissions along these lines.

Nick Axford and Michael Little

Note

1. The title is borrowed from the book Between the Monster and the Saint: Reflections on the Human Condition, by Richard Holloway (2008).

Reference

Axford, N. and Morpeth, L. (2013), “Evidence-based programs in children's services: a critical appraisal”, Children and Youth Services Review, Vol. 35 No. 1, pp. 268-77