Public health strategies and activities are intrinsically complex. According to the literature, this “wickedness” depends on the different interests and expectations of the stakeholders and the community, the fragmented governance of the related services and the challenges in measuring and assessing public health outcomes. Existent performance measures and management systems for public health are not designed to cope with wickedness since they are mainly focused on inputs and outputs, neglecting broader outcomes because of their long-term impact and the poor accountability of results. This research aims to tackle this shortfall by adopting a dynamic performance management (DPM) approach.
This research explores the case of the vaccination campaign of a Regional Health System. Through the analysis of an illustrative case study, the research discusses both opportunities and limits of the proposed approach.
This research highlights that DPM supports performance management (PM) in wicked contexts, thanks to the adoption of a system-wide perspective and the possibility of using simulation to experiment with alternative strategies and benchmarking performance results with simulated trends.
This article tackles a gap related to the management of wicked problems both from a theory and a practical perspective. In particular, this research suggests the adoption of DPM as an approach that may support policymakers in tackling social pluralism, institutional complexity and scientific uncertainty all at once.
Introduction
Our society and its current challenges are growing in complexity, so modern literature defines these last ones as “wicked” (Rittel and Webber, 1973; Head and Alford, 2015). Wicked does not mean “evil” but refers to the impossibility of sharing an understanding of societal problems and their definition. According to Head (2008) and Head and Alford (2015), “wickedness” mainly depends on three key factors: social pluralism – i.e. the presence of multiple interests and values of stakeholders; institutional complexity – i.e. the horizontal fragmentation and multilevel governance characterizing public administrations; and scientific uncertainty – i.e. the existence of gaps in reliable knowledge.
The emergence and persistence of wicked problems pose important challenges to performance management (PM) in the public sector. PM is a methodological approach whose main purpose is to guide organizations toward their objectives according to effectiveness, efficiency and viability principles (Ouchi, 1979; Ferreira and Otley, 2009).
However, traditional PM systems are considered unable to cope with wicked problems and their characteristics due to a static approach that does not account for feedback-loops and non-linearities; and the inter-organizational fragmentation characterizing the public sector that leads to poor accountability of results (Drury, 2014; Noto and Bianchi, 2015; Bianchi, 2016). To manage performance in wicked contexts, key solutions proposed by the literature refer to the implementation of collaborative arrangements and co-production (Cristofoli et al., 2017; Loeffler and Bovaird, 2019; Bianchi, 2021). These solutions have been designed and tested to address the source of complexity deriving from institutional fragmentation and social pluralism. However, to the best of the authors' knowledge, only some scholars proposed solutions that also tackle scientific uncertainty.
Based on this gap, this work aims to illustrate how dynamic performance management (DPM) can be used to cope with wicked problems in managing performance by tackling the key limits characterizing traditional PM schemes.
DPM is a scientific approach combining system dynamics (SD) with PM theory (Bianchi, 2016). SD is a modeling technique that allows analysts to frame and simulate complex systems and experiment with the models to design strategies for management and change (Forrester, 1958).
The adoption of SD to tackle wicked problems finds its foundation in the methodological opportunity to explore with simulation and engage with stakeholders (Vennix, 1999; Sterman, 2000), thus providing more robust decision support to inter-institutional settings. As such, SD allows analysts to adopt a holistic perspective to frame social pluralism, institutional complexity and scientific uncertainty as the key factors characterizing wicked contexts. As a result, it supports collaborative PM systems fostering a shared understanding of their operating principles and processes.
This article focuses on public health and health prevention to explore and analyze the contribution of SD in the case of wicked issues. Public health is considered fundamental in tackling current social challenges such as the COVID-19 pandemic. In particular, the process of planning and implementing vaccination campaigns in western Countries represents one of the most challenging wicked problems in the current era.
The research begins with a literature review on wicked problems and PM in the public sector. A second theoretical background explores the literature on PM in public health. Building on this literature review, the article proposes and illustrates the DPM approach (Bianchi, 2016, 2021) as a systemic method for PM and governance in wicked contexts. Then, the approach is tested on a case study exploring the vaccination campaign in Lazio – an Italian regional health system that reported a successful experience with vaccination operations – to explore how this health system could benefit from the adoption of DPM. The case is developed by collecting empirical data, active engagement and discussing results in focus groups with the Lazio regional managers. In the last section of the article, the case study results enable us to discuss the main advantages and limitations of the proposed approach also in comparison with the gaps found in the literature. Eventually, the article outlines critical implications for theory and practice and concludes with future research perspectives.
Theoretical background
Wicked problems and performance management
In the public administration context, the last years have been characterized by a growing and renewed interest in exploring the so-called “wicked” problems and the possible countermeasures public policy research proposes to deal with their negative implications. “Wicked” problems were defined back in 1973 as issues that are hard to define and manage due to their complexity (Rittel and Webber, 1973).
Wicked problems are counterposed by the literature to tame ones (Roberts, 2000; Weber and Khademian, 2008; Head and Alford, 2015). Differently from these lasts – for which both the definition of the problem and the likely solution are clear to decision-makers – wicked problems have no definitive formulation and may have multiple solutions which are necessarily open to further interrogation and adaptation (Rittel and Webber, 1973; Head and Alford, 2015). Moreover, robust evaluation frameworks to assess the success of initiatives and solutions implemented are not suitable to support public managers in addressing them (Rittel and Webber, 1973; Head and Alford, 2015).
In spite of these challenges, in the last two decades, we witnessed an emergence of research focusing on how to deal with these problems through the public administration and management lenses (Roberts, 2000; Weber and Khademian, 2008; Ferlie et al., 2011; Head and Alford, 2015; Cristofoli et al., 2017; Bianchi et al., 2017; Massey, 2022). As previously mentioned, Head (2008) and Head and Alford (2015) framed the sources of wickedness in three factors, i.e. social pluralism, institutional complexity and scientific uncertainty. Public managers and leaders are thus called to cope with these factors to guide their organizations in dealing with wicked problems. Decision-makers operating in wicked contexts are indeed exposed to multiple trade-offs between divergent and sometimes conflicting results (Belle and Cantarelli, 2022).
An emerging topic on wicked problems literature is promoted by a group of scholars who focused their research on how to manage performance in wicked contexts (Blackman et al., 2006; Drury, 2014; Noto and Bianchi, 2015; Cepiku, 2017; Bianchi et al., 2017; Herrera et al., 2019; Costumato, 2021; Noto et al., 2022). PM is defined as the activity that guides an organization, or a social system, toward its objectives and targets (Ouchi, 1979; Lebas, 1995; Ferreira and Otley, 2009). This first definition clarifies how challenging may be the design and implementation of PM systems in contexts whose problems and issues are not even definable and, consequently, desired outcomes are ambiguous or unclear (Rittel and Webber, 1973; Head and Alford, 2015).
To deal with wickedness and lead organizations and networks towards desirable performance, extant literature strongly suggested the adoption of collaborative (Roberts, 2000; Jackson and Stainsby, 2000; Christensen and Lægreid, 2007; Ferlie et al., 2011; Lagreid and Rykkja, 2015; Cristofoli et al., 2017; Noto et al., 2022) and co-production practices (Bianchi et al., 2017; Loeffler and Bovaird, 2019; Steen and Brandsen, 2020). The idea beyond these studies is that collaboration and co-production may foster a shared understanding of problems. Indeed, such countermeasures find their rationale in the institutional fragmentation characterizing the current governance structure of many public sectors in Western countries and in need to involve the public at large to comprehend and address the multiple needs and interests carried out by the community (Christensen and Lægreid, 2007; Turrini et al., 2010). Collaboration and co-production may foster a shared understanding of problems and align stakeholders' expectations toward preferred solutions and outcomes (Cristofoli et al., 2017; Bianchi et al., 2017). As such, the suggested solutions fit well with two of the sources of wickedness previously mentioned – i.e. social pluralism and institutional fragmentation. However, these kinds of solutions do not necessarily directly tackle the third cited source of wickedness – i.e. scientific uncertainty.
Uncertainty is a phenomenon that is part of existence and public administration and management are disciplines that necessarily involve uncertainty (Weber, 1999; Cairney et al., 2016). There are different sources and types of uncertainty. Regarding scientific uncertainty, we refer to gaps in reliable knowledge (Reckhow, 1994; Head and Alford, 2015).
Among the main approaches used to cope with uncertainty, and in particular scientific uncertainty, research in multiple disciplines widely relies on policy models and simulation techniques (see, among others, Meadows et al., 1972; Reckhow, 1994; Papadopoulos and Yeung, 2001; Welter and Kim, 2018). Simulation models could provide practical support to explore and experiment with complex social systems and thus comprehend their functioning (Forrester, 1958; Sterman, 2000, 2002, 2014; Kim et al., 2013; Cosenz, 2018; Noto and Cosenz, 2021; Noto, 2022). Moreover, simulation allows one to experiment with alternative scenarios and test the sensitivity of a system regarding specific variables or parameters. Such an approach is of great help when scientific knowledge about variables and value parameters is unclear or incomplete (e.g. the contagiousness of a virus variant). In the words of Sterman (2002, pp. 524), “simulation is essential for effective systems thinking, even when the purpose is insight, even when we are faced with a “mess” rather than a well-structured problem”.
Assessing and managing the performance of public health services
Since the implementation of New Public Management reforms, PM has been extensively applied in public administrations and, more generally, in the public sector (Hood, 1991; Bouckaert and Halligan, 2007; Rajala et al., 2018). The public health sector has not escaped this cultural wave, implementing PM systems and mechanisms (Aidemark, 2001; Lega et al., 2013; Nuti et al., 2018; Noto et al., 2021).
In the realm of public services, health service has been an interesting field of implementing PM practices due to its high input and output measurability. The importance of measuring health service end-results emerged already in the seminal article of Donabedian (1966), which highlighted the need to assess the quality of care provided through the assessment of structures, processes and outcomes. The Donabedian performance assessment framework has become one of healthcare's most diffused performance measurement frameworks. It has successfully been adopted in the hospital setting, both in the case of inpatient and outpatient services (Berwick and Fox, 2016; Lenzi et al., 2020). Donabedian's framework also stands at the basis of the performance measurement schemes of public health (Deber and Schwartz, 2016; Riley et al., 2012; Scutchfield et al., 2004; Handler et al., 2001) that, in the USA, had led to the development of the Turning Point PM Conceptual framework in 2003 and the release of a new version in 2013. The five components of the Turning Point are (Landrum and Baker, 2004; DeAngelo et al., 2014): performance standards (set goals, targets and relevant indicators to improve public health performance), performance measures (refinement and application of performance indicators and measures), reporting of progress (analysis of data and feedback to stakeholders), quality improvement (use of data for decisions to improve policies and programs, manage change and achieve quality outcomes) and visible leadership. Following this stream of thought, other research works tried to develop Balanced Scorecards for Public Health (Robinson et al., 2003; Weir et al., 2009). In the studies mentioned above, PM systems were not designed to provide routine data to monitor ongoing public health strategies but only to give a static view of public health performance (Schwartz and Deber, 2016). As argued by Schwartz and Pais (2009), these frameworks are poorly designed to assess the inherent complexity of the public health sector characterized by policy fragmentation, feedback loop and non-linearity.
Therefore, PM still encounters complex challenges when implemented in public health and health prevention services. Moreover, public health performance measurement systems rarely link broad outcomes to measures of processes and output controlled by public health organizations (Schwartz and Deber, 2016). This is mainly due to the specific mission of public health and, consequently, the characteristics that distinguish public health services from others in the healthcare sector. The mission of public health is indeed multifaceted. It depends on the different healthcare governance systems and infrastructures; however, there are some common traits across the public health systems worldwide. Public health has the aim to improve the health of populations through (Ingram et al., 2012; Weir et al., 2009): the prevention and monitoring of diseases; the development of policies to inform, educate and empower people and community about health issues; the enforcement and application of regulations that protect health and ensure safety.
Due to the complexity of its mission, multiple features make public health PM challenging.
First, many factors that influence population outcomes are beyond the direct control of public health organizations, such as economic factors, lifestyle, gender and genetics (Weir et al., 2009; Deber and Schwartz, 2016). This is different in the hospital setting, where the outcome mainly depends on organizational and individual performance.
Second, public health services are usually delivered by a network of organizations (e.g. health authorities, police forces, laboratories and other public organizations) that collaborate (Schwartz and Pais, 2009; Noto et al., 2022).
Third, it is difficult to directly measure the impact of public health actions on population health outcomes due to the time lag between public health interventions and their effects. Outcomes of prevention activities – such as education, inspections and vaccinations – can only be appreciated in the long run. As such, it is not possible to assess their performance in time to put corrective actions in place (Hunter, 1990; Ingram et al., 2012; Cinquini et al., 2014). Another temporal lag is related to the long latency of some diseases that may obscure the impact of performance on measures that come out later, like mortality rates (Ingram et al., 2012).
Last, public health initiatives and outcomes affect other domains such as economic development.
These characteristics describe a wicked context and pose essential challenges to PM in public health. Performance measures and PM systems for health prevention services usually focus on inputs and outputs, neglecting broader outcomes due to their long-term impact and the poor accountability of results. Namely, due to governance fragmentation, it is impossible to identify a single institution responsible for the outcomes obtained.
The traditional performance measurement and management frameworks (from Donabedian's to Turning Point's systems) present some gaps related to the wicked nature of public health. However, the basic elements of their conceptualization are still valuable for elaborating new models for managing scientific uncertainty alongside social pluralism and institutional complexity. Therefore, the instrumental view of DPM conceptualized by Bianchi (2016) is appropriate to repropose the basic elements of the PM frameworks for public health. This view identifies and connects strategic resources (e.g. human resources, equipment, pharmaceutical products), intermediate results or processes outputs (e.g. number of inspections delivered, number of people vaccinated, etc.) and end-results or outcomes (e.g. epidemic outbreaks, disease morbidity, etc.).
In addition, the DPM supports decision-makers by allowing them to experiment and simulate alternative strategies with the SD model. This feature enables the switch from feed-back to feed-forward control of results. Namely, decision-makers can monitor the progress toward the achievement of desired outcomes by periodically comparing results with simulated behaviors and estimating projections for subsequent periods to eventually anticipate adjustments to the objectives or to the pattern of actions.
Method
This article proposes the DPM as a method to tackle PM in wicked issues. This scientific approach combines traditional PM theory with SD modeling (Bianchi, 2016).
SD is a methodological approach for modeling and simulating complex physical and social systems and experimenting with the models to design strategies for management and change (Forrester, 1958). It provides a systemic perspective and a set of conceptual tools that enable one to frame complex, non-linear and multi-loop feedback systems (Forrester, 1958; Meadows, 1980; Sterman, 2000).
To represent complex and dynamic systems, SD models adopt a graphical syntax in which flow (rate) and stock (level) variables interact with each other through a set of causal relationships mathematically modeled as differential equations that can be simulated (Sterman, 2000; Größler et al., 2008). This means that a variable influences another variable (1) positively (i.e. an increase of the one corresponds to an increase of the other and vice versa), (2) negatively (i.e. an increase of the one corresponds to a decrease of the other and vice versa) and (3) according to a non-linear relation between them. If such relations originate closed circuits, i.e. the feedback loops could be reinforcing (R), i.e. they produce exponential trends of the system over time; or balancing (B), i.e. they limit such an effect by tending to a steady-state.
According to an instrumental view of performance (Bianchi, 2016), performance can be framed into strategic resources, performance drivers (or intermediate results) and end-results. Strategic resources are those stock variables whose activation and use allow the implementation of specific actions. Performance drivers are those indicators, usually computed as ratios, which display the progress of an organization or a system toward achieving desired results. End-results represent the system's goals whose achievement allows to foster back the strategic resources – i.e. generating accumulation and/or depletion processes. Figure 1 illustrates how performance drivers are built and connected to end-results and strategic resources according to a causality-based perspective.
Due to the above considerations, this research argues that a DPM approach is suitable to address PM in wicked contexts – such as public health – since it combines (1) a PM framework, (2) a system-wide perspective in which the various interests and perspective of the various stakeholder, as well as the multilevel governance, are represented and (3) it allows to cope with scientific uncertainty through the use of simulation.
Consistently with the purpose of this research, DPM is applied to an illustrative case study (Scapens, 2004). A case study strategy is particularly valuable for addressing theory-building research and demonstrating that the existing research does not properly address the investigated propositions (Eisenhardt and Graebner, 2007). The case study focuses on the adoption of the COVID-19 vaccine strategy and its implementation in the best performer Italian regional system identified through the data on vaccine delivery (National Health Institute data [1]), i.e. Lazio. The case has been selected to explore and discuss the potential advantages and risks associated with the adoption of DPM to foster performance in the health prevention sector and, more generally, when operating in wicked contexts.
In vaccination activities, we can find the three sources of wickedness. First, vaccination campaigns embrace multiple interests and values (social pluralism), e.g. this pandemic created trade-offs between public health and the competitiveness of economic activities due to lockdowns and restrictions. Second, health prevention services are provided through the collaboration of multiple public and private organizations (institutional complexity), i.e. national and regional governments, local health authorities, hospitals, private health providers, general practitioners, national armies, pharmaceutical companies, etc. Last, health prevention results are usually obtained in the long term, and it is not possible to univocally attribute these results to activities executed in the past (scientific uncertainty). Thus, it appears almost impossible to assess in advance the effectiveness of specific measures (i.e. vaccination strategy) as it depends on the other actors' behavior (virus – and its variants – included).
The case study was based on combined data collection tools obtained from a variety of sources: government open data about vaccines and epidemiological trends, official documents, institutional reports and data from the health authorities and other public entities (Lazio region, National Health Institute, Ministry of Health [2]) and three focus groups with key informants (the General Director of the Lazio Health System and his staff). In particular, government open data about vaccines (e.g. delivered doses) were used to develop the vaccines production and distribution section of the model. The epidemiological dynamics of the model (referring to variables such as population, deaths, healed, etc.) originate from open data on epidemiological trends, official documents and institutional reports. The Laboratorio Mes and Agenas (2021) report was employed to build up the communication section of the model. Two focus groups with key informants were held in October 2021. Each lasted about one hour and served to understand the regional health authority's organizational structure and the related model section (regarding hubs and human resources). A third two-hours-focus group was run in November 2021 to explain and discuss with key informants the research results, the strengths and weaknesses of the simulation model compared to the PM systems currently used to monitor the healthcare system and the results of vaccination strategies and – eventually – further application of this methodology in public health.
The DPM simulation model was developed and calibrated based on the data referred to the first six months of the vaccination campaign (1st January 2021–30th June 2021). The software used was Stella Architect®.
The model validation was performed according to the SD literature's requirements (Barlas, 1996; Sterman, 2000; Homer, 2012). An SD simulation model is indeed validated both in its structure and behavior. Structure validation tests assess the validity of the model structure by direct comparison with knowledge about real system structure (Barlas, 1996). In this regard, the model has been compared with knowledge about the system analyzed as emerging by the data, documents and reports used to develop the case study. The model has also been discussed in its sections during the focus groups with the key actors of the Lazio health system. Behavior validity is assessed by comparing model output with empirical data (Barlas, 1996; Homer, 2012). The validation process verifies the ability of the model to replicate the actual behavior of the variables considered during the reference period, performing extreme condition tests (Barlas, 1996) and partial model testing (Homer, 2012). Once the simulation model was developed, validated and tested to determine whether it realistically behaves, inputs were modified to conduct scenarios and sensitivity analyses of how short- and long-term results would change in response to alternative strategies and actions (Kunc and O'Brien, 2017; Torres et al., 2017; Noto and Cosenz, 2021).
Results
A high complexity characterizes the analysis of the vaccination campaign due to its numerous physical and social factors. A causal loop diagram (CLD) was designed to highlight the relationships between the key variables that emerged during the case study's development. CLD is a qualitative representation of a complex system often used in system thinking applications (Sahin et al., 2020). Figure 2 displays the vaccination campaign CLD.
As portrayed in Figure 2, the system analyzed is characterized by four key balancing loops (which depict dynamics tending to counteract any disturbance and move the system toward an equilibrium point) and five key reinforcing loops (dynamics driving toward exponential growth).
Loops description
| Loop | Description |
|---|---|
| R1 | R1 loop explains the dynamic according to which the more vulnerable population, the more people get infected. Once healed, these people get antibodies and can be considered immune for the following period |
| R2 | This loop shows that the more people get vaccinated, the more population can be considered non-vulnerable. The vaccine effect will last for about 6 months, and then the vaccinated will be considered vulnerable again |
| R3 | It shows the dynamic according to which the more vaccines are available, the more people get vaccinated. On the other hand, vaccination decreases the stock of vaccines available |
| R4 | According to this loop, the more people get vaccinated, the fewer restrictions the health authorities impose on the population. These restrictions reduce contact between people and, thus, infections |
| B1 | This loop portrays the dynamics according to which the more people get infected, the less the stock of vulnerable people is |
| B2 | According to this loop, the more people get vaccinated, the less the stock of vulnerable people is |
| B3 | This loop shows that the more people need vaccines, the more hubs, and human resources are activated for the vaccination campaign. This positively influences the number of vaccines inoculated, decreasing the stock of vulnerable people |
| B4 & B5 | This loop portrays the dynamics according to which the more people get infected (or hospitalized), the more restrictions to reduce contact will be introduced |
While CLD is effective in representing the structure of a complex system, understanding the combined effect of the loops and their dominance requires building a proper simulation model, i.e. the so-called stock-and-flow diagram (SFD). This also allows adopting a DPM perspective by framing the abovementioned variables in terms of strategic resources, end-results and performance drivers (or indicators) (Bianchi, 2016).
The whole model structure comprehends four key sections: epidemiological (related to the infection diffusion model); production (related to the production of the vaccines); organizational (related to the creation and management of vaccination hubs and the recruitment of staff); and communication (related to the communication strategy and its effectiveness).
The graph in Figure 3 shows a synthetic version of the SFD framed according to the DPM logic – i.e. strategic resources, performance drivers and end-results. The complete list of variables used is reported in Appendix.
Key variables specification
| Name | Type | Description |
|---|---|---|
| Infected | Strategic resource | This represents the stock of people infected by COVID-19. Data to initialize the variable (and compare behavior) gained from the Ministry of Health website |
| IC hosp | Strategic resource | The number of people infected and hospitalized in Intensive Care (IC) units. Data to initialize the variable (and compare behavior) gained from the Ministry of Health website |
| Vulnerable | Strategic resource | The stock of healthy people, non-vaccinated, who can be infected. Variable initialized as 0, since the vaccination campaign started at the beginning of the period considered. Data to compare behavior obtained from government open data |
| Vaccinated | Strategic resource | The stock of people vaccinated. Variable initialized as 0, since the vaccination campaign started at the beginning of the period considered. Data to compare behavior obtained from government open data |
| Hub & HR | Strategic resource | This variable represents the health system's strategic resources consisting of the number of vaccination hubs and human resources recruited or assigned to the vaccination campaign. In the full running model hubs and staff are split into two stock variables. Data gained from the Lazio health system |
| Comm. Level | Strategic resource | This is a stock variable representing the communication level of the vaccination campaign (initialized through Laboratorio MeS and Agenas (2021) data) |
| Intensive care | Performance Driver | This variable is computed as the ratio between the stock of people hospitalized in intensive care departments and the intensive care beds available. A higher value of this indicator brings to the adoption of restriction policies to limit contact between people with a consequent effect on the economic activity and productivity of the region |
| Vaccination | Performance driver | This driver compares the number of people fully vaccinated with the number of vulnerable people. Together with the production one, this driver is related to the number of hubs activated to vaccinate the population |
| Production | Performance driver | This variable focuses on the ability of the pharmaceutical providers to respond to the vaccine demand computed as the population of vulnerable people. It is the ratio between the vaccines available and the vulnerable population and represents the production capacity constraint of the vaccination campaign |
| Accessibility | Performance driver | It measures the adequacy of the organizational structure to perform a successful vaccination campaign. It directly impacts the accessibility of vaccines for the population in terms of availability and capillarity of the distribution network |
| Communication | Performance driver | This driver measures the communication's effectiveness that incentivizes people to accept vaccination by comparing the actual communication level with the desired one |
| Infection | End result | This variable measures the infection rate and mainly depends on the vaccination campaign results and the restriction policies |
| Chg in hub & HR | End result | This result portrays the change in the strategic organizational resources (hubs and human resources) and depends on the magnitude of the pandemic |
| Immunization | End result | This rate represents the flow of vaccines delivered to the vulnerable population, which is the direct result of the vaccination campaign |
The results of the model are portrayed in Figures 4 and 5. Figure 4 shows the behavior of the key model variables, i.e. number of infected, number of death, number of doses delivered and the flows of vaccine administration (first dose, second dose and Janssen).
Figure 5 shows the result of the five performance indicators. The behaviors of the performance drivers allow one (e.g. policymakers, leaders, managers) to monitor the progress toward the success of the vaccination campaign.
Experimentation
The critical feature that allows DPM to deal with scientific uncertainty is the possibility of experimenting with the model and simulating scenarios based on alternative policies or model sensitiveness concerning specific parameters.
In the case analyzed, four alternative scenarios have been tested through simulation (see Figure 6) based on three key policy levers: communication effectiveness, vaccination production and restrictions.
Run 1 represents the simulation as is, i.e. the behavior validated through empirical data.
Run 2 shows the system behavior in case of low communication effectiveness on the vaccine campaign (from the original value of 0.7 estimated through Laboratorio Mes and Agenas (2021) data to 0.2), other conditions being equal. In this case, we may notice a worse performance of the vaccination campaign in terms of both outputs (doses delivered) and outcomes (infected and deaths).
Run 3 portrays the extreme scenario in which vaccines are not available. The only constraint to the virus is represented by the restriction policies leading to related impacts on the economy and the social well-being of the population. In this case and the previous one, it is possible to notice a worse performance in terms of infected people and deaths compared to the original scenario.
Run 4 shows cases where vaccines are unavailable, and no restriction policies are applied. Such a scenario shows an uncontrolled diffusion of the virus, especially in the first months, dramatically impacting the number of deaths.
Last, Run 5 portrays a scenario in which, other conditions being equal, restriction policies are strengthened. In this case, health outcomes show improved behavior at the expense of potential damage to the economy and the social well-being of the population.
The ones portrayed in Figure 6 are some of the possible simulations we can obtain through the DPM model developed to represent the vaccination campaign of the Lazio Region. Such experimentation allows us to appreciate the typical trade-offs implicit in health prevention activities and understand the system's functioning to design policies and monitor their performance through the set of performance drivers previously commented on.
Discussion
Wickedness is today recognized by both literature and practice as a key trait of current policy problems (Rittel and Webber, 1973; Head and Alford, 2015). As such, facing wickedness has become pivotal for every public manager (Massey, 2022).
Since the implementation of NPM, PM has been the most used approach to pursue efficiency and effectiveness in using resources and creating public value (Moore, 1995; Bryson et al., 2014). PM schemes have been widely applied in multiple settings of the health sector, from hospitals to public health and prevention (Vainieri et al., 2020).
Managing performance in wicked contexts is controversial. PAs and their managers are called to achieve targets to address problems that are not even definable – or, at least, there is no shared understanding of their traits. As such, public managers are called to deal with multiple trade-offs deriving (Head, 2008; Head and Alford, 2015) (1) from the social pluralism characterizing them, (2) from the presence of several institutions having a stake and an influence on their management and (3) from gaps on reliable knowledge about their implication.
Previous literature suggested collaboration and co-production for addressing those trade-offs deriving from social pluralism and institutional fragmentation (Cristofoli et al., 2017; Loeffler and Bovaird, 2019; Bianchi, 2021) thus trying to cope with the shared understanding of policy problems. However, a research gap still exists in dealing with scientific uncertainty.
According to Cairney et al. (2016), public administrations should rely on evidence to cope with uncertainty. However, some of the issues public organizations are dealing with are new, and there is no past (or similar) evidence that managers and leaders can rely on. As such, simulation may be a powerful tool to deal with knowledge gaps and learn how complex systems work (Kim et al., 2013; Sterman, 2014; Noto, 2022).
Combining simulation with PM schemes and system-wide perspective (e.g. a perspective taking into account the role, influence and interests of the stakeholder of a system) is proposed in this article as a possible solution to address wickedness. More in detail, the article suggests adopting the DPM approach (Bianchi, 2016). Based on the illustrative case developed, it is possible to discuss the potential benefits and pitfalls of the proposed approach.
Through the use of SD, DPM may bring in the perspectives and mental models of different stakeholders participating in the system analyzed (Vennix, 1999). As in our case study, the model “challenges the clouds” including and “endogenizing” in the model all the elements having an influence on the system and being influenced by it (Sterman, 2002; Noto and Bianchi, 2015; Nabavi et al., 2017).
DPM “speaks the same language” of healthcare and public health PM schemes. The instrumental perspective integrating strategic resources, performance drivers, and end-result can easily be overlain with the Donabedian (1966) and the Turning Point Performance Management Collaborative (2003) frameworks. Nevertheless, these traditional PM schemes suffer from several weaknesses, which the DPM may overcome.
First, they were developed to emphasize the role of PM systems in implementing quality improvement projects. At the same time, the DPM is flexible and can be adapted to several different health policy strategies.
Second, traditional public health PM schemes adopt typically backward-looking approaches (i.e. showing the effectiveness of past decisions) and are composed mainly of lagging (outcome) indicators. Therefore, they need to be accompanied by the development of intermediate results measures (i.e. performance drivers) for creating sustained performance improvement (Riley et al., 2010). The DPM, instead, is a comprehensive method based on a forward-looking approach suggesting expected trajectories of future performances and supporting strategy execution. The DPM model is designed to identify, monitor and simulate the behavior of those performance drivers representing intermediate results over time. These performance drivers are defined as causally linked to the system's end-results - or outcomes (Bianchi, 2016); due to this, they represent the key levers policymakers should act on to manage the system's performance. Decision-makers are indeed enabled to benchmark effective drivers' behavior with the simulated one and to put in place corrective actions in case of significant gaps. This is not possible when focusing exclusively on the long-term outcomes characterizing public health strategies (Hunter, 1990; Cinquini et al., 2014). Neither it is possible to effectively monitor intermediate results (e.g. vaccine doses delivered, number of inspections performed, etc.) without a target to benchmark with. Simulation may provide these benchmarks as intermediate results values are causally related to simulated expected outcomes. Decision-makers of the Lazio health system could have used this tool to evaluate policy trade-offs and monitor policy implementation processes by comparing results and simulated trends.
Third, the Turning Point Model, as well as the other approaches considered in the theoretical background, is mainly used for guiding the single public health department and, consequently, it cannot work appropriately for addressing public health issues or programs on which the actions of several different institutions may converge. On the contrary, DPM simulates complex systems on which public and private institutions, along with the preferences of the different population segments, can influence the achievement of results. As such, DPM embraces the suggestion provided by collaborative governance literature which asks for the engagement of multiple institutions and players having a stake in the public health issue (Weber and Khademian, 2008; Cristofoli et al., 2017).
Last, as highlighted by Schwartz and Pais (2009), the traditional assessment methods provide little insight into synergies, feedback loops and non-linearity in the complex interweaving of public health interventions. The result is that “those wishing to develop and refine better strategies therefore have insufficient knowledge as to which interventions to deploy in what sequence and in what combinations under varying contexts” (Schwartz and Pais, 2009, p. 20).
To overcome this latter weakness, DPM recognizes the feedback loops and non-linearity of interventions, simulates alternative scenarios (e.g. lockdown vs. vaccination) and tests parameters' sensitiveness (e.g. communication effectiveness). This feature allows decision-makers to experiment with the model and explore the trade-offs characterizing wicked contexts.
Conclusions
This article aims to fill a research gap related to PM in a wicked context such as public health. Based on this theoretical gap, this research suggests the adoption of DPM as an approach that may support policymakers in tackling social pluralism, institutional complexity and scientific uncertainty all at once. The result of this research highlights that this support comes from the following features: (1) system-wide perspective, (2) PM framework and (3) simulation.
The limits of the approach are mainly related to the inner characteristics of SD methodology. This last aims to comprehend the overall dynamic behavior of a system rather than performing a precise quantitative prediction (Meadows, 1980). Moreover, while SD adopts a system-wide perspective, it could be interesting to understand how single agents interact in determining the system performance (thus combining a system-based perspective with an agent one).
Other limitations of this study are related to the illustrative case developed. We chose the Italian Regional Health System case study, which best performs in the vaccination campaign in terms of vaccines delivered. However, other case studies – such as worst performers – could have brought interesting insights to the discussion.
As the limitations represent an opportunity to make suggestions for future research, we believe that future studies may develop multiple case studies to compare the strategies adopted to address the vaccination campaign. Additionally, comparing or combining the suggested approach with other simulation techniques, such as agent-based modelling or discrete event simulation, could be insightful.
The authors gratefully acknowledge Massimo Annicchiarico, general director of the Lazio regional health department, and the key informants who participated in the research activities. The author would also like to thank the editors and the anonymous reviewers.
Notes
https://www.epicentro.iss.it/vaccini/archivio (accessed on October 2021)
https://www.salute.gov.it/portale/nuovocoronavirus/dettaglioContenutiNuovoCoronavirus.jsp?lingua=italiano&id=5351&area=nuovoCoronavirus&menu=vu (accessed on February 2022), https://www.epicentro.iss.it/vaccini/archivio (accessed on February 2022), https://githu.com/pcm-dpc/COVID-19/tree/master/dati-regioni (accessed on February 2022)
References
Appendix
Stocks
Communication_level(t) = Communication_level(t–dt) + (Chg_in_comm) * dt.
Deaths(t) = Deaths(t–dt) + (Death_rate) * dt.
Delivered_doses(t) = Delivered_doses(t–dt) + (doses) * dt.
First_dose_vaccinated(t) = First_dose_vaccinated(t–dt) + (Immunization_1_dose–Total_immunization - “1_vaccinated_infection”) * dt {NON-NEGATIVE}
Healed(t) = Healed(t–dt) + (Healing–Loosing_immunity) * dt {NON-NEGATIVE}
Hubs_-_lines(t) = “Hubs_-_lines” (t–dt) + (Chg_in_hubs) * dt.
Infected(t) = Infected(t–dt) + (Infection–Death_rate–Healing) * dt {NON-NEGATIVE}
Janssen_available(t) = Janssen_available(t–dt) + (Jansenn_production–Janssen_consumption) * dt.
Moderna_available(t) = Moderna_available(t–dt) + (Moderna_production–Moderna_consumption) * dt.
Pfizer_available(t) = Pfizer_available(t–dt) + (Pfizer_production–Pfizer_demand) * dt.
Staff(t) = Staff(t–dt) + (Recruitment) * dt.
Vaccinated(t) = Vaccinated(t–dt) + (Immunization_Janssen + Total_immunization + Vaccinated_healed–Vaccinated_infection–Loosing_protection) * dt {NON-NEGATIVE}
Vaccinated_deaths(t) = Vaccinated_deaths(t–dt) + (Vacc_death_rate) * dt.
Vaccinated_infected(t) = Vaccinated_infected(t–dt) + (Vaccinated_infection + “1_vaccinated_infection” - Vaccinated_healed–Vacc_death_rate) * dt {NON-NEGATIVE}
Vaccines_available(t) = Vaccines_available(t–dt) + (Vaccines_incoming–Vaccination) * dt.
Vaxevria_available(t) = Vaxevria_available(t–dt) + (Vaxevria_production–Vaxevria_consumption) * dt.
Vulnerable_population(t) = Vulnerable_population(t–dt) + (Loosing_immunity + Loosing_protection – Immunization_1_dose–Immunization_Janssen–Infection) * dt {NON-NEGATIVE}
Flows
1_vaccinated_infection = Contagious_contacts*Contagiousness* Probability_of_infection_compared_to_non_vaccinated_1.
Chg_in_comm = (Communication_lever-Communication_level)/Time_to_chg_comm
Chg_in_hubs = (Planned_hubs-“Hubs_-_lines”)/Time_to_build_hubs
Death_rate = Deadliness*Infected
Doses = Immunization_Janssen + Total_immunization + Immunization_1_dose.
Healing = Healing_rate*Infected.
Immunization_1_dose = IF Vaccines_available*Converter > Overall_productivity THEN MAX((Overall_productivity-“1_vaccinated_infection”)*Community_acceptance; 0) ELSE MAX(Vaccines_available*Converter*Community_acceptance; 0).
Immunization_Janssen = IF Janssen_available*Converter > Overall_productivity THEN MAX((Overall_productivity-“1_vaccinated_infection“-Immunization_1_dose)*Community_acceptance; 0) ELSE Janssen_available*Converter.
Infection = Contagious_contacts*Contagiousness.
Jansenn_production = LOOKUP(Janssen_capacity; TIME)*Vaccination_policy
Janssen_consumption = Immunization_Janssen
Loosing_immunity = Healed*Loosing_immunity_rate.
Loosing_protection = Loosing_immunity_rate*Vaccinated.
Moderna_consumption = MAX(Moderna_available/Delivery_time; 0).
Moderna_production = LOOKUP(Moderna_capacity; TIME)*Vaccination_policy
Pfizer_demand = MAX(Pfizer_available/Delivery_time; 0).
Pfizer_production = LOOKUP(Pfizer_capacity; TIME)*Vaccination_policy
Recruitment = (Desired_staff-Staff)/Time_to_recruit
Total_immunization = IF Vaccines_available*Converter > Overall_productivity THEN MIN(Overall_productivity; First_dose_vaccinated/Time_to_get_second_dose) ELSE Vaccines_available*Converter.
Vacc_death_rate = Vaccinated_deadliness*Vaccinated_infected
Vaccinated_healed = Vaccinated_infected*Healing_rate
Vaccinated_infection = Contagious_contacts*Contagiousness*Probability_of_infection_compared_to_non_vaccinated
Vaccination = (Immunization_1_dose + Total_immunization)*Vaccine_per_person
Vaccines_incoming = Moderna_consumption + Pfizer_demand + Vaxevria_consumption.
Vaxevria_consumption = MAX(Vaxevria_available/Delivery_time; 0).
Vaxevria_production = LOOKUP(Vaxevria_capacity; TIME)*Vaccination_policy
Converters and constants
Accessibility_PD = (“Hubs_-_lines”/Planned_hubs)*Human_resources_PD
Communication_lever
Communication_PD = Communication_level/Desired_communication_level
Community_acceptance = People_accepting_vaccination/Vulnerable_population
Contacts = Infected*Sociability.
Contagious_contacts = Contacts*Potential_concentration
Contagiousness.
Converter.
Deadliness.
Delivery_time
Desired_communication_level
Desired_staff = “Hubs_-_lines”*“People_per_hub-line”.
Factors = Communication_PD*Accessibility_PD
Fully_vaccinated = First_dose_vaccinated + Vaccinated + Vaccinated_infected + Vaccinated_deaths
Healing_rate
Hosp_rate
Hospetalizations = Infected*Hosp_rate
Human_resources_PD = Staff/Desired_staff
IC_capacity.
IC_rate.
Intensive_care = Infected*IC_rate.
Intensive_care_PD = Intensive_care/IC_capacity.
Janssen_capacity = GRAPH.
Loosing_immunity_rate.
Moderna_capacity = GRAPH.
Overall_productivity = Productivity*“Hubs_-_lines”.
People_accepting_vaccination = Vulnerable_population*Factors.
People_per_hub-line.
Pfizer_capacity = GRAPH.
Planned_hubs = Planned_hubs_1*Production_constrain
Planned_hubs_1 = GRAPH(Vaccination_PD).
Potential_concentration = Vulnerable_population/Potential_infection
Potential_infection = Infected + Vulnerable_population + Healed + First_dose_vaccinated + Vaccinated + Vaccinated_infected
Probability_of_infection_compared_to_non_vaccinated
Probability_of_infection_compared_to_non_vaccinated_1.
Production_constrain = GRAPH(Production_PD).
Production_PD = (Vaccines_available + Janssen_available)/Vulnerable_population
Productivity.
Restriction_policies = GRAPH(Intensive_care_PD).
Sociability = Restriction_policies
Time_to_build_hubs
Time_to_chg_comm
Time_to_get_second_dose
Time_to_recruit
Vaccinated_deadliness
Vaccination_PD = Vaccinated/Vulnerable_population
Vaccination_policy
Vaccine_per_person
Vaxevria_capacity = GRAPH.
