The challenge of evaluating care integration programmes
Care integration evaluation is characterised by methodological pluralism and diversity. Moreover, researchers of care integration have chosen different research philosophical paradigms, i.e. basic belief systems or world views to guide their investigations. In order to measure integration from the perspective of patients or healthcare providers, interpretivism has played an important role (e.g. Bautista et al., 2016). However, positivism has dominated the field since the early days as policy makers were mostly interested whether integrated care is (cost-)effective. With most reviews and meta-analyses summarising the insights derived from quantitative data being mostly inconsistent (e.g. Damery et al., 2016; Valentijn et al., 2018), a shift towards realism is taken place in the last few years. In doing so, researchers argue that integration needs to be regarded as a tailor-made approach with evaluation paying attention to what works for whom in what circumstances, and how (e.g. Busetto et al., 2016; Turner et al., 2018).
Notwithstanding the value of individual research philosophical paradigms, their independent application limits the generalisability of evaluation findings and contributes to the rather low level of shared learning across the sector. Moreover, the methodological quality of evaluation in the field is concerning. This applies to studies with empirical data (e.g. Looman et al., 2018), reviews (e.g. Briggs et al., 2018) and studies on the development and validation of instruments to measure integrated care (e.g. Bautista et al., 2016).
Against this background, the Journal of Integrated Care invited authors to submit their work paying attention to the (many) challenges in evaluating care integration programmes. We were also very much interested in learning about novel methodologies to overcome such challenges.
In their case study evaluation, Hinde report on an extensive analysis of the Vanguard programme, exploring whether the implemented integrated response services resulted in any meaningful change in secondary healthcare activity. Data access, poorly specified aggregate secondary care data, and a poorly specified intervention were put forward as the main challenges confronted with. The methodological challenges of one of the Vanguard sites is also discussed by Grimwood (2019), who points our attention to what (modified) realist evaluation has to offer when dealing with complexity, strategy and rhetoric. By applying a multi-method evaluation protocol, Grooten et al. (2019) are looking into the implementation of a scaling-up strategy for integrated care initiatives. Pragmatism may perhaps be the best way to describe their approach as they plan to use realist evaluation and implementation science.
With the increasing interest in implementation science in this field, the viewpoint paper by Sadler et al. (2019) discusses three potential contributions from implementation science that can help clinicians and researchers to design and evaluate more effective integrated care programmes for older people with frailty. Another relative novel method is discussed by Gray and Shaw (2019). They explore summative, process and developmental approaches to evaluating complex interventions and suggest three guiding principles in doing so. A final contribution to this special issue comes from Kaehne (2019) who explores how Big Data may help us in the evaluation of care integration programmes and whether the evaluation and research community in this field is ready to take on what Big Data has to offer.
The challenge of shared learning in the field of care integration and its evaluation goes beyond a special issue or a section on evaluation in the Handbook Integrated Care (Amelung et al., 2017). It requires a culture where people share what they know, where people receive constant and constructive feedback, and where the expertise of team members is actively promoted. When care integration programmes and their evaluations take place in separate worlds with different dynamics and languages, it is no surprise that shared learning is not happening. In that sense, the Vanguard Programme has much to offer, including briefing series, blogs, podcasts, vanguard videos, resources, reports and site visits (NHS Confederation).
Researchers need to step out of their comfort zone as shared learning is not what most of them do on a regular base. Moreover, keeping dogmatically to existing study designs and outcome measures for publication purposes is not what evaluation of care integration programmes needs. If integration is indeed about the transformation of service delivery into new models of care, why then apply methods that are most suitable for the evaluation of existing services?
The same holds for editors of scientific journals. We need to embrace the challenges that come with change too, for example, by sharing what we know, by providing authors with constructive feedback, and by referring authors to each other, in short, by actively creating a community of practice. This special issue is an example of such an effort as both the Journal of Integrate Care and the International Journal of Care Coordination want to foster shared learning about how the evaluation of care integration programmes can further be improved to support informed decision making by all those involved.
