Using the policy capacity framework to evaluate New Zealand’s primary health care policy implementation Open Access

Policy researchers and practitioners have long explored the question of what capacities, skills, and resources are required for successful policy implementation. The Policy Capacity Framework (PCF), outlined by Wu et al. (2015), which distinguishes between political, operational, and analytical capacities at multiple levels, has made an important contribution to scholarship in this area. This parsimonious and flexible framework provides a means of diagnosing the strengths and weaknesses of policy capacity in specific settings (Tenbensel and Silwal, 2023). Howlett and Ramesh (2016) used the PCF to suggest that different policy domains may require varying emphases on these policy capacity elements. This paper extends this line of inquiry by asking whether different profiles of policy capacities are required to address different issues within a policy domain. The empirical focus is on primary health care (PHC) policy in Aotearoa | New Zealand.

Scholarly interest in the concept of policy capacity is strongly connected to efforts to understand the success and failure of policy implementation (Brenton et al., 2023). A crucial feature of the PCF has been its emphasis on situating policy capacity within a governance context, in which policy is the product of interaction between state and non-state actors.

The PCF identifies political capacity (obtaining and maintaining political support and commitment); operational capacity (aligning resources with policy actions so that they can be implemented in practice); and analytical capacity (ensuring policy actions are technically sound and can contribute to policy goals). This framework can help make sense of implementation success and failure in specific policy contexts, because it offers a way to diagnose the policy capacity elements, and combinations between them, that may enhance and/or inhibit policy success.

The PCF is part of a broader strand of policy capacity research (Hughes et al., 2015; Bali and Ramesh, 2023) which advocates a broader focus than is usually taken by those with a focus on policy implementation success. In the extant policy studies literature, the analysis of implementation success is tied to specific, identifiable decisions made by governments. This concreteness dates to Pressman and Wildavsky’s (1973) launch of policy implementation as a domain of policy research. Similarly, in McConnell’s framework for analysing policy success, implementation success is defined in terms of meeting governmental objectives, fidelity to policy design, and benefits for specified target groups (McConnell, 2010).

The PCF’s governance perspective, however, requires an approach to implementation success that is more expansive in two ways. Firstly, it raises the question of ‘success for whom’, given that different policy actors, including implementers, often lack shared understandings of what success means (Sabatier, 1986; Matland, 1995). In addition, important and innovative policy-shaping initiatives can also have non-governmental origins. Secondly, it is important to broaden the object of evaluations of implementation success/failure beyond ‘policies with names’. Hill and Hupe advocated framing implementation research as ‘the comparison of the expected with the achieved’ (2002, p. 8). This means that research into policy success should incorporate a broader view of policy processes, beyond identifiable decisions and programs to one that includes understanding success in the context of expectations, both of government and non-government actors.

Health policy is one policy domain where strong path dependence and institutional structures lock-in key features of health systems such as financing mechanisms (Tuohy, 1999; Bali and Ramesh, 2023). Therefore, a focus that is restricted to named governmental initiatives would miss the parameters shaping policy and governance that are beyond the reach of decision-makers.

This article develops and applies a broader conceptualisation of policy success which focuses on progress towards ‘valued policy objectives’ (VPOs) that are important to policy actors within policy subsystems, and are prominent in policy discourse. The first key component of this approach is locating it in policy subsystems organised around policy domains and subdomains – our area of interest is primary health care (PHC) policy. As a way of understanding policy success, this requires a focus on the worlds of policy insiders, rather than broader constituencies and the general public.

Each policy domain and community of insiders develops a language of policy aspirations which are more specific than general values (such as efficiency), and strongly shaped by jurisdictional health policy histories. VPOs characterise aspirations that are medium to long-term, and language shared by some (although not necessarily all) policy actors and stakeholders. They are part of the discourse within policy subsystems and are recognised, though not necessarily subscribed to, by most participants within. Any VPO may have divergent interpretations within a policy subsystem. Two examples of VPOs in PHC policy are “better integration with secondary care”, and “reduction of access barriers for PHC services”.

VPOs serve as a suitable anchor for applying the PCF to understand policy implementation success. Our proposition is that different elements, or combinations of elements, of policy capacity may be more or less important for different VPOs. Political policy capacity may be vital for some VPOs, whereas. analytical capacity may be more important for progress towards other VPOs.

The idea that specific elements of policy capacity are more important in some contexts than others has been explored in relation to broad policy domains (Howlett and Ramesh, 2016). This article explores the possibility that different VPOs (or perhaps different types of VPOs) have different policy capacity profiles.

Aotearoa | New Zealand has a predominantly publicly funded health system, with government revenue (taxes and levies) as the source of approximately 80 percent of health spending (Gauld, 2020). While hospital services are delivered by public entities, most PHC services are delivered by non-government providers.

Government contributes around 77 percent of spending on primary care services (Health and Disability System Review, 2019). Most patients are required to contribute co-payments, which vary considerably across practices and between categories of patients (Jeffreys et al., 2020). Between 2001 and 2022, the Ministry of Health led policy development and held limited contracting responsibilities. Non-government actors included professional bodies such as the Royal New Zealand College of General Practitioners and the Nursing Council.

Below the national level, hospital services provision, and the contractual funding of community and PHC services were the responsibility of twenty District Health Boards (DHBs) (Cumming, 2016).

Most New Zealanders access PHC services through traditional general practices, which are predominantly for-profit. Other PHC providers include non-profit, community-based organisations, including organisations set up to serve Māori and Pacific communities (Southwick et al., 2012; Gifford, 2018), although most Māori and Pacific patients are enrolled with ‘mainstream’ (traditional or corporate) providers (Sheridan et al., 2023).

Primary Health Organisations (PHOs) are the meso-level provider organisations which serve as the contracting agent for ‘first-contact’ (mostly general practice) services. Many PHOs also develop service initiatives at scale for their member practices (Tenbensel et al., 2020). At the district level, DHBs and PHOs were the central players in PHC policy implementation and development until mid-2022.

Governments have attempted to take a more active purchasing role in PHC since the1990s, and a longstanding structural tension has been about the role and reach of the state, and the legitimacy of its expectations of PHC providers (Gauld, 2008; Croxson et al., 2009). Aotearoa | New Zealand was one of the first countries to adopt a population health focus in health policy, attempting to steer the system towards better, and more equitable, health outcomes (Tenbensel et al., 2008).

The relationship between indigenous (Māori) and non-indigenous peoples has been a defining consideration of New Zealand health policy, based on state recognition of the foundational status and importance of Te Tiriti o Waitangi (The Treaty of Waitangi). The persistence of inequities of health outcomes between Māori and non-Māori has consequently been a central focus PHC policy specifically, where inequities of health outcomes and access to services are well-documented (Waitangi Tribunal, 2019; Jeffreys et al., 2020). Another persistent PHC policy concern has been the considerable cost-related barriers of access to PHC services, despite various policy measures to reduce or mitigate these barriers. Policy responses in the 2010s focused on subsidising PHC practices for visits from patients in targeted eligibility categories, rather than more complex but potentially more powerful approaches such as adjusting capitation funding formulae.

Governments have also sought to address the structural separation between PHC and specialist and hospital services, partly for reasons of resource allocation and efficiency, but also in response to advocacy from PHC providers. Since 2000, policy has focused on collaboration between PHC and community providers, alongside greater community participation in service design (Tenbensel et al., 2021).

A focus on supporting PHC quality improvement emerged in the mid-2000s, and since 2010, concerns over the future shape of the PHC workforce have become prominent policy concerns. PHC providers and their representatives have also expressed increasing levels of concern around the adequacy of funding and its distribution, particularly to providers that serve high-needs patients and populations, and the sustainability of PHC services.

This background covers the spectrum of PHC policy issues and associated VPOs. At one end, there are ‘system-driven’ VPOs focused on improving population health, reducing inequities of access and outcomes, and accountability for PHC spending. At the other are ‘sector-driven’ issues such as quality improvement and sustainability of PHC practices and services. Based on this overview of the Aotearoa | New Zealand PHC policy context, we adapt Wu et al.’s levels of policy capacity from systemic, organisational, and individual, to focus instead on macro (national level), meso (geographic districts), and micro (individual PHC practices).

Given this wide-ranging policy agenda for PHC in Aotearoa | New Zealand, the interest is in which features of policy capacity are crucial elements in progress towards addressing these issues across this terrain. The authors explore the relationship between policy capacities and VPOs through the following set of questions.

• 1.

  What reasons are given for progress towards VPOs in PHC, and which policy capacity elements are relevant to these reasons?

• 2.

  What reasons are given for lack of progress towards VPOs in PHC, and which policy capacity elements are relevant to these reasons?

• 3.

  What are the areas of overlap between policy capacity interpretations of progress and lack of progress?

• 4.

  Which PCF elements are less visible in accounts of progress /lack of progress?

• 5.

  Do different VPOs (or types of VPOs) have different profiles of crucial policy capacity elements, and if so, why?

By answering these questions, the paper provides useful diagnoses for PHC policy practitioners (funders, decision-makers, implementers) to enable them to identify and prioritise resources and capabilities that are needed to make progress towards VPOs in the future.

The methodology is informed by a pragmatist orientation (Sanderson, 2009; Zittoun, 2014), in which the motivation for research is to contribute to reflection on past approaches and practices. This in turn can assist in making practical contributions to strengthening and improving PHC policy. By identifying the policy capacity resources and capabilities that are present, and highlighting areas in need of attention, we seek to build and enhance policy capacity in PHC.

To answer the research questions, the authors opted for an approach which elicited the perceptions of key players in the policy subsystem about where and why progress towards VPOs was made, and where and why it was not. Our research team began in 2019 by identifying and naming a suitable range of PHC VPOs. We sought VPOs that reflected PHC priorities, connected to each other without extensive overlap.

The two research leads in the team (TT and LM) developed a list of ten VPOs drawing from PHC policy documents, primary care media articles, and our collective knowledge and experience of PHC policy discourse since 2000. The list was modified and adapted in research team discussions.

The full list of VPOs is in Table 1, arranged along a spectrum from ‘system-driven’ (top) to ‘sector-driven’ (bottom).

We conducted 55 interviews with PHC key informants between late 2019 and early 2021. Interviewees included public officials and PHC interest group leaders at the national level, and board members, managers, and clinicians from government (DHB) and non-government (PHO) organisations at the local level. Sixteen interviewees were from national-level organisations, 16 from DHBs and 23 from PHOs. As national-level respondents included both government and non-government actors, the overall proportion of state and sector participants was roughly even.

The interviews did not include most key Māori key informants, as they were interviewed as part of a related, but separate process. Many participants had occupied a range of roles (both government and non-government, national and local) over their careers. Interviews were either conducted in person or by video link during the COVID-19 pandemic.

Interviews were structured and covered a range of topics. In one part of the interview, respondents were given an array of ten VPOs in the form of laminated cards (in-person interviews) or PowerPoint slide (video link interviews). They were asked to identify which two VPOs had seen most progress since 2010, and which two VPOs had seen least progress. They were then asked to articulate the reasons for their choices. Ten VPOs were chosen to limit the cognitive load on interviewees when identifying VPOs with most and least progress. We asked respondents to indicate if important objectives were missing. Few respondents identified additional objectives, and most were happy with the range provided.

Data from answers to the follow-up questions were collated in a qualitative research software database. We extracted the relevant data for each VPO, read the extracts multiple times and constructed hand-written notes (1-2 pages) for each VPO. We then coded this data using six relevant categories from the PCF (political, operational, and analytical capacity at macro and meso-levels). After exploring our data, we opted not to code for the micro-level as we had not recruited respondents from that level of the subsystem.

From the spreadsheet, profiles of policy capacity elements relating to progress and lack of progress towards VPOs were compiled. The spreadsheet was subdivided by VPO, and further subdivided by type of interviewee (PHO, DHB, National, plus a summary row). If any respondents’ reasoning fit under our PCF codes, we listed these reasons in summary phrases. These summary phrases were collated within the table rows, and we then assessed their profile based on the PCF definitions.

Figure 1 shows the range of reasons given for progress and lack of progress, categorised in terms of policy capacity elements, towards all objectives, arranged along the spectrum from system-driven to sector-driven. Cells shaded in green indicate where multiple participants gave policy capacity-related reasoning for progress towards VPOs.

The key finding here is that respondents offered more policy capacity related reasons for progress towards sector-driven VPOs than for system-driven VPOs.

Overall, meso-level policy capacity reasons were more prominent than reasons that referenced the resources and capabilities of macro-level agents. The key policy capacity component at the macro level was political capacity. At the meso-level, both political and operational resources and capabilities were commonly cited as reasons why progress had been made. At both levels, there were few references to the role of analytical capabilities and resources.

At the macro level, respondents pointed to leadership and commitment by both government agencies, and non-government entities. In reasoning why progress had been made towards quality improvement, many respondents referred to initiatives of professional colleges.

Similarly, for progress towards for multi-professional primary care, many respondents referred to the Ministry of Health’s Access and Choice programme, which supported primary care practices to provide new mental health services delivered in primary care settings.

At the meso-level, it was primarily political capacities of PHOs and formal alliances between PHOs and DHBs, that were cited as reasons for progress.

Reasons for progress citing operational capacity resources and capabilities were also important at the meso-level across most VPOs.

Reasons relating to analytical capacity only seemed relevant to understanding progress towards two VPOs – quality improvement, and integration with secondary care.

Reducing inequities between Māori and non-Māori, and accountability for primary care spending were two VPOs in which few respondents articulated reasons for progress. For reducing inequities, this was simply because respondents overwhelmingly judged that progress had not been made.

While some respondents did identify policy mechanisms that aimed to strengthen accountability for primary care spending, most regarded these as trivial or not helpful in achieving what they regarded as genuine accountability.

Turning our attention now to reasons for lack of progress understood in terms of PCF elements, cells shaded in brown in Figure 2 indicate where multiple participants gave policy capacity-related reasoning for lack of progress towards VPOs. The emphasis here was on lack of progress towards system-driven objectives, indicated by the shaded cells are located predominantly in the middle and left of the spectrum.

Macro-level capacity was often cited for lack of progress on system-driven VPOs, while meso-level political and operational capacity were relevant for a wider range of VPOs. Issues of analytical capacity at the meso-level were also cited as reasons for lack of progress towards system-driven VPOs.

Lack of political capacity at the macro level was regarded as a key reason for least progress on reducing inequities and reducing access barriers. At first glance, this seems to refer to the changing priorities of central government.

However, most reasons that invoked political capacity reasons at the macro level focused on the role of government agencies. Weaknesses in the operational skills and analytic capabilities of national-level agencies was frequently cited as a reason for lack of progress over the 2010s, often by those who had worked in these organisations.

Moving to the middle of the spectrum, many respondents identified meso-level political capacity (reasons referring to PHO and/or DHB leadership and political commitment) as reasons for lack of progress.

For these VPOs, many respondents drew attention to difficulties experienced by PHO and DHB leadership in putting political commitment into operational practice, and to the lack of experimentation and innovation at local levels.

There were fewer examples where respondents pointed to gaps in analytical capacity at the meso-level.

Notably, we found very little reasoning for the lack of progress that highlighted analytical capacity at the macro level.

Figure 3 highlights where there is congruence between policy capacity-related reasons for progress and lack of progress. This figure is created by superimposing Figure 2 (lack of progress) on to Figure 1 (progress). Cells shaded black indicate capacities identified as contributing to progress and lack of progress. The patterns of PCF-based interpretations for progress and lack of progress overlap mainly in the middle of the spectrum, predominantly at the meso-level.

For three VPOs in the middle of the spectrum – integration with secondary care, community participation, and collaboration with non-health – meso-level political capacity seems particularly crucial. This is what distinguishes geographic districts where respondents thought progress had been made from those who thought it had not. Local perceptions of progress varied, reflecting differing assessments of local policy capacity.

One VPO that showed a unique pattern is reducing access barriers to PHC. Here, both political and operational capacity at both macro and meso-levels were identified as contributing to progress, or the lack of it. This is a policy area in which macro policy settings, such as changes to patient eligibility for subsidised PHC, and funding formulas, play a crucial role but respondents also clearly identified the importance of local level political commitment and operational skills in addressing this problem.

The analysis has allowed us to build a nuanced and informative picture of how the policy subsystem perceives policy capacity strengths and weaknesses in PHC. The key messages are summarised in three points.

Firstly, two broad policy capacity profiles are identified – one which is relevant for tougher, system-driven PHC policy problems, and one which is more applicable to sector-driven VPOs. Sector-led valued objectives, such as quality improvement and moves towards multi-professional health care were regarded by respondents as areas of greater progress. Here the energy for change came from political capacity and commitment from meso-level organisations, buttressed by meso-level operational capacity and macro-level political capacity. The latter took the form of infrastructural support from state agencies and professional colleges.

By contrast, implementation of system-driven VPOs was far more problematic, and required a different profile of policy capacity. Key informants identified very few policy capacity resources and capabilities that had effectively supported progress towards objectives such as reducing inequities, and enhancing accountability for primary health care spending, and their reasoning suggests that operational capacity at the macro level is the area that requires most attention to progress these VPOs.

This reflects that system-driven primary health care VPOs are more ambitious and challenging than others. Some respondents provided their own explanations of why VPOs towards the sector-driven end saw more progress than others.

At the macro level, it would be easy to conclude that the lack of national political commitment was the main reason that little progress was made towards reducing inequities and reducing access barriers because these were not high-profile government commitments during the 2010s. Political commitment to system-driven VPOs waxes and wanes according to electoral dynamics and ministerial appetites.

However, this reasoning for the lack of progress was overshadowed by a stronger focus on the lack of operational capacity at the macro level. Throughout the 2010s there was residual commitment to such objectives throughout the primary care system. Generally, our respondents regarded these goals (particularly reducing inequities) as vitally important. As 75 percent of respondents chose reducing inequities as the one of the two most important policy objectives for the 2020s, this far outstripped any other VPO. Even when the stars align and such VPOs are politically sanctioned, our analysis shows that weaknesses in macro-operational capacity can mean that such policy commitments bear little fruit.

We also note the absence of operational capacity at the macro level around reducing access barriers. The few policy measures adopted and implemented in the 2010s were the consequence of party election promises, and the absence of other internally generated policy proposals from central government agencies is another indicator of weakness in macro analytical and operational capacity.

The pivotal role of operational capacity is also clear in relation to the VPO of accountability for primary health care spending. This is an area in which there is significant operational activity, in the form of targets, contract management and other NPM routines and techniques. Our research shows that such practices are regarded by primary care policy community as trivial at best, and harmful at worst, while doing little to enhance meaningful accountability.

To address policy issues that are of most importance to PHC providers, the meso-level really matters. Our respondents identified many innovations that contributed to progress that were the product of networked policy design at both macro and meso-levels. This finding is consistent with a wider range of PHC research in New Zealand and internationally in policy implementation of initiatives requiring inter-organisational collaboration (Gauld, 2017; Aunger et al., 2021; Cumming et al., 2021; Tenbensel et al., 2021; Sharma et al., 2024). In these respects, our analysis is consistent with Howlett and Ramesh’s (2016) argument that meso-level (organisation) operational capacity is crucial for policy areas where network governance is necessary, such as PHC policy.

Secondly, a puzzling finding from our research is that analytical capacity did not feature strongly as providing explanations either for progress or for lack of progress. In fact, we found little if any reference to analytical capacity at the macro level as contributing to progress or lack of progress for any VPO.

This is startling and may reflect the invisibility of macro-level analytical capacity in the 2010s. It is hard to imagine how progress towards state-led goals such as improved accountability, reduction of inequities and reduction of access barriers could be achieved without some significant macro-level analytical capacity. External reviews have concluded that the required data on utilisation and service provision needed to underpin policy is not available in the sector (Downs, 2017; Health and Disability System Review, 2019, 2020).

Since our interviews, this situation may have changed. The 20 DHBs were merged into a single organisation, Health New Zealand, in July 2022. Since 2020, more useful analyses of gaps in PHC service utilisation, provision and funding have been provided by researchers and external consultancies (Love et al., 2022; Sheridan et al., 2023).

Finally, applying the PCF to New Zealand PHC policy leads us to re-emphasise the crucial importance of operational capacity, in its wider sense of not only implementing government policy directives, but also of being a creative force in developing new ideas, options and practices at macro and meso levels.

Operational capacity does not necessarily require prior political commitments to be made (Tenbensel and Silwal, 2023). Local experimentation by health sector leaders can also lead to enhanced policy capacity, in the form of commitment (Denis et al., 2023). Our extended understanding of implementation success is consistent with governance literature, which also involves incorporating bottom-up implementation perspectives (Hughes et al., 2015).

This interpretation of the PCF has some parallels to Kingdon’s multiple streams framework (Kingdon, 1995). A healthy policy stream requires experts within policy subsystems (both in government agencies and non-govt orgs) to be active in formulating policy options that can be picked up at the propitious time. Some policy proposals, including substantive changes to funding mechanisms, require governmental political commitment, but this still leaves room for initiatives to be developed and tested by actors throughout the system.

Our analysis shows how the meso-level capacity may be crucial to the implementation of many important VPOs in PHC, strengthening calls for increased policy attention to the creation and/or strengthening of meso-level organisations.

Basing judgements of policy success solely on interviews with policy insiders has a potential downside, as such perceptions may be distorted, may collectively miss some features, or be characterised by ‘groupthink’. Our analysis also sought to identify which policy capacity elements were less commonly identified (particularly analytical capacity). This could reflect blind spots characteristic of the PHC policy community, and/or may indicate genuine deficiencies in PHC policy processes.

The limited number of Māori key informants may mean that some reasons for lack of progress towards reducing inequities are less prominent. The roughly concurrent Waitangi Tribunal report’s findings clearly identify weaknesses in political and operational capacity at macro and meso levels, and of macro analytical capacity (Waitangi Tribunal, 2019).

By only interviewing senior managers and decision-makers, it was likely their worldview would highlight operational capacities. However, our respondents were those best placed to identify operational strengths and weaknesses, which are less visible outside the policy subsystem.

Similarly, our finding about the crucial importance of meso-level policy capacity could reflect that 70 percent of interviewees were drawn from meso-level organisations. However, we did not detect a pattern in which macro-level interviewees attributed progress to macro-level and lack of progress to the meso level. A large proportion of our respondents had careers in which they crossed between macro and micro levels, and between government and non-government roles, meaning that they had a view of PHC policy shaped by multiple system roles.

The key finding is that the type of policy capacities that need strengthening vary according to the type of valued policy objective. Meso-level policy capacities are crucial for addressing sector-driven policy issues such as quality improvement and sustainability of practices. Macro policy capacities provide support in terms of political commitments, but most of the heavy lifting is done at the meso level.

The paper also suggests that a different approach to strengthening policy capacity is needed for the more challenging system-driven VPOs in more challenging times. In New Zealand, considerable attention has focused on the political capacities in tackling larger, systemic PHC issues. We suggest that those who seek to make progress on tougher system-driven issues such as reducing inequities and access barriers need to focus attention on the strengthening of analytical and operational capacity at the macro level.

The approach to analysing implementation success, redefined in terms of reasons for progress towards valued policy objectives, has shown the utility of Wu et al.’s PCF in helping to pinpoint strengths and weaknesses within New Zealand’s PHC policy subsystem. More generally, we suggest that it can help policymakers and researchers identify which policy capacities matter for different types of policy problems and policy objectives in any policy domain.

This study was conducted in accordance with the ethical standards of Victoria University of Wellington and was reviewed and approved by the Victoria University of Wellington Human Ethics Committee, Approval No: 27171, date of approval: 05/03/2019.

All participants provided informed consent prior to their participation in the study. Participation was voluntary, and respondents were informed of their right to withdraw at any time without consequence. All data were collected and stored confidentially and used solely for research purposes.