Effective decision-making for the provision and maintenance of infrastructure systems requires strategic performance indicators aligned with a clear vision of the societal benefits that infrastructure will be expected to enable and systemic awareness of interdependencies between infrastructure sectors. This paper proposes a conceptual outcome-oriented approach to strategic infrastructure performance indicators as a systemic alternative to current approaches which predominantly focus on performance within individual sectors. The conceptual approach proposed aligns performance measures with stated priorities and future aspirations rather than past performance; provides a transparent framework for decision-making; can be applied at a range of scales; and creates an evidence base against which indicator design can be justified and reviewed. Additionally, the conceptual outcome-oriented approach proposed is adaptable for application in other areas of infrastructure decision-making, in particular the development of methodologies to assess what infrastructure will be needed in the future.
Introduction
Infrastructure is important for the simple reason that societies rely on infrastructure networks to provide public services, enhance quality of life, generate profits and aid economic growth (Boin and McConnell, 2007). However, infrastructure is a means to end, not an end in itself; therefore, shaping the infrastructure of tomorrow requires a clear understanding of the link between it and the desired societal outcomes that it is expected to support. It follows that it is important to acknowledge a number of features of infrastructure when framing any infrastructure research problem, project or objective.
The purpose of infrastructure is to enable desired outcomes, including economic growth (Beca and Covec, 2013; HM Treasury and Infrastructure UK, 2014; Infrastructure and Projects Authority, 2016; OECD, 2007; President’s Commission on Critical Infrastructure Protection, 1997).
Infrastructure is a complex interdependent system of systems (Beckford, 2010; Carhart, 2016; Council for Science and Technology, 2009; Johansson and Hassel, 2010; Oughton and Tyler, 2013; Rinaldi et al., 2001; Tolone, 2009).
The context (social, political, economic, financial, legal, environmental, regulatory, local, global, spatial and temporal) in which any infrastructure is embedded is of profound importance (Adhitya et al., 2016; Beckford, 2010; Carhart et al., 2014; Dolan et al., 2016; Hollnagel, 2014; Omega Centre, 2012; Rittel and Webber, 1973).
The objective of the study presented here is to develop an approach to strategic infrastructure performance indicators (PIs) that makes it possible to regularly and meaningfully evaluate (a) the extent to which infrastructure enables the desired outcomes expected of it and therefore its ‘fitness for purpose’; (b) infrastructure performance at the whole-system level, and (c) whether changes to infrastructure strategy are needed to ensure that infrastructure performance remains fit for purpose in the face of future challenges (i.e. any change to the context in which infrastructure is embedded).
The need for this study is justified on the basis of a critique of current UK practice (‘A critique of infrastructure PIs in the UK National Infrastructure Plan’ section), in-depth literature review and analysis of international best practice (‘Lessons from performance management literature and international best practice’ section) and engagement with infrastructure practitioners (‘Stakeholder workshop to elicit expert perspectives’ section). This identified a significant gap in the literature and current practice regarding the development and use of strategic PIs in infrastructure systems that fulfil objectives a–c.
Synthesis of ‘A critique of infrastructure PIs in the UK National Infrastructure Plan’ to ‘Stakeholder workshop to elicit expert perspectives’ sections led to the articulation of a set of guideline criteria for strategic PIs (‘Guideline criteria’ section) and design principles (‘Design principles for outcome-oriented PIs’ section) that ensure that the process of developing the indicators is consistent with the guideline criteria and fulfils objectives a–c. The most important emerging issue from this was the need for these indicators to be aligned directly with an agreed, clearly stated and regularly reviewed set of the desired outcomes that society expects infrastructure to enable. These findings directly inform a high-level conceptual process for developing outcome-oriented PIs for infrastructure that is presented in ‘Process for outcome-oriented PI development’ section.
The conceptual approach outlined here is intended to be applicable internationally, in any national or regional context where strategic evaluation of infrastructure performance is required.
The final element of this conceptual approach emphasises adherence to a simple process for a number of reasons. Firstly, the use of a robust, transparent, auditable and replicable process can help to ensure that the logical basis of the PIs is widely understood by all affected stakeholders; secondly, it encourages a collaborative, evidence-based approach; and, thirdly, it aligns the justification for all decisions with the desired outcomes that infrastructure is expected to help enable.
Terminology
To aid development of the conceptual approach, a number of terms are first proposed and defined in Table 1.
Terminology for infrastructure performance indicators
| Term | Explanation/definition |
|---|---|
| Desired outcome(s) | A forward-looking statement (or set of statements) of what it is that infrastructure is expected to enable. Note: Desired outcomes provide infrastructure with purpose; infrastructure performance cannot be meaningfully measured without them. |
| Outcome-oriented PI | A strategic PI directly aligned with one or more desired outcomes that infrastructure is expected to enable. |
| Outcome dimension | A specific component of a desired outcome. Note: This term is necessary, as it is postulated that desired outcomes are typically broad concepts comprising multiple dimensions and that they can be better understood and measured if decomposed into a set of simpler elements or outcome dimensions. |
| Outcome subdimensions | A specific component of an outcome dimension. Note: This term is necessary because in some instances outcome dimensions may also need to be decomposed further before they can be measured or understood. |
Additionally, for purposes of clarity, it is important to emphasise the distinction between two complementary but distinct groups of PIs, both of which are used to inform infrastructure decision-making.
Technical PIs, which have the purpose of measuring and providing insight into real-time performance and are used by infrastructure operators to inform operational decision-making
Strategic PIs, which have the purpose of providing an overview of performance at the sector or system level and are used to support long-term evaluation of infrastructure performance against high-level strategic objectives.
It is this second group on which this paper focuses and in relation to which the term ‘outcome-oriented PI’ is proposed.
Paper structure
The research presented here was developed using the conceptual framework illustrated in Figure 1; the paper is structured using headings from Figure 1. In the next section, the paper presents a critical review of the UK’s current approach to infrastructure strategic PIs. International best practice and relevant literature are considered in ‘Lessons from performance management literature and international best practice’ section, followed by the findings from a workshop consultation with infrastructure experts in ‘Stakeholder workshop to elicit expert perspectives’ section. ‘A conceptual approach to outcome-oriented infrastructure PIs’ section presents and discusses the guideline criteria, design principles and strategic PI development process. ‘Discussion’ and ‘Conclusions and recommendations for practical application’ sections present the conclusions and recommendations, respectively. Applications of the conceptual approach from the second phase of this study, and the associated conclusions from doing so, are presented in a subsequent paper published in this issue (Carhart et al., 2016).
A critique of infrastructure PIs in the UK National Infrastructure Plan
In the UK, the National Infrastructure Plan (NIP) sets out the challenges facing infrastructure and the government’s strategy for meeting the infrastructure needs of the UK economy. The first NIP was published in 2010 (HM Treasury and Infrastructure UK, 2010), and since 2011 Infrastructure UK, now the Infrastructure and Projects Authority, has published an annual set of PIs for national infrastructure systems (HM Treasury and Infrastructure UK, 2011). Recognising the importance of the World Economic Forum and the World Bank’s global infrastructure rankings, the 2011 NIP (HM Treasury and Infrastructure UK, 2011: p. 15) stated that its principles were ‘to maintain the overall performance of the UK’s infrastructure over time’ and ‘to address the UK’s weaknesses and catch up with the best performers in the world’.
Global infrastructure rankings are based on subjective assessments informed by opinion surveys of perception, while the 2011 NIP developed a set of PIs based on objective data. The link between the two is not clear. In the 2012 NIP, it was stressed that the chosen dimensions are not intended to provide targets (HM Treasury and Infrastructure UK, 2012). The 2013 NIP reinforced the intended purpose of the indictors: (a) to monitor whether infrastructure performance levels are, as a minimum, maintained over time; (b) to monitor infrastructure performance levels in relation to other countries; and (c) to monitor the impacts of investment (HM Treasury and Infrastructure UK, 2013). While the 2014 NIP did not explicitly mention the purpose of the indicators, it implied that they are also related to delivering value for money (HM Treasury and Infrastructure UK, 2014).
At the national scale, the NIP’s quantitative metrics monitor performance trends of key infrastructure: major roads, passenger rail, airports, container ports, communications, electricity network, gas network, flood risk prevention and waste disposal. As such, they can make only a limited contribution to the future development of infrastructure. Information is obtained from a number of different sources, including regulatory authorities (e.g. Office of Gas and Electricity Markets, Office of Communications), the Office of National Statistics, the Environment Agency, the Department of Energy and Climate Change, National Grid and the Organisation for Economic Co-operation and Development. For each sector, and where applicable, the indicators attempt to report on performance in six areas: (a) capacity, access and availability; (b) asset or capacity utilisation; (c) service quality and reliability; (d) asset condition; (e) safety; and (f) efficiency. A total of 65 PIs are calculated across the sectors. Values are normalised against those for the year 2005, which are given a value of 100. A value greater than 100 indicates an increase in the measure compared with that for 2005, whereas a value below 100 indicates a decrease (HM Treasury and Infrastructure UK, 2014). This comparison to a baseline performance in 2005 makes the indicators inherently backward looking. Indicators for the six areas of interest are combined within each sector to give an overall index for a particular sector’s performance.
The underlying data from which indicators are derived pose a number of concerns: the data are not recorded and gathered by the Infrastructure and Projects Authority, nor are they collected specifically for the purpose of contributing to infrastructure PIs. Therefore, there is a risk that organisations collecting these data will cease to do so, or change the way in which the data are collected and reported. This could affect the ability to report consistent indicators to accompany the NIP in line with its publication cycle. Furthermore, although there is some commonality around the themes across sectors (e.g. service reliability, capacity), comparison of performance between different systems is difficult. Current indicators remain firmly embedded in their infrastructure ‘silos’ and fail to help address performance assessment at the system-of-system level.
In addition to data-related challenges, the usefulness and appropriateness of current indicators are further limited for three reasons. Firstly, the current indicators do not have a clearly articulated purpose or purposes. Despite the statements in the early NIPs, it is not evident how the PIs are intended to inform stakeholder action or decision-making, or, indeed, who the target audiences are. In theory, PIs have the potential to identify issues with the state of the current systems and direct resources; measure progress towards achieving a goal; attract investment; demonstrate success against specific criteria; or enforce penalties. They can inform real-time day-to-day decision-making, or guide long-term strategy. Data collected for one purpose are not necessarily appropriate for other purposes, and the same is true of PIs. Understanding what performance information infrastructure stakeholders require for the decisions they make, why they need that information and how they use it are all key to moving towards an improved set of PIs. Secondly, the current PIs are, at best, loosely coupled with the values and requirements of society. For the vast majority of the PIs, there is no clear link between the data and the achievement of the desired outcomes of those relying on the infrastructure systems. This may stem from collective uncertainty over these societal needs and how they relate to the infrastructure systems which facilitate their fulfilment. Understanding societal, environmental and economic needs and wants is also important for developing an improved set of PIs. In his review of long-term infrastructure planning in the UK, Armitt (2013) concluded that the NIP does not represent an evidence-based strategic vision for Britain. Thirdly, the indicators do not take a systemic approach which would be needed if characteristics such as resilience and interdependencies that emerge more frequently at the system’s level, rather than at the sectoral level, are to be managed, mitigated and exploited effectively.
Lessons from performance management literature and international best practice
Literature on infrastructure performance management emphasises the importance of understanding the purpose, goals and plans for the systems whose performance is being measured prior to designing any PIs (Behn, 2003; Hatry, 1999; Lebas, 1995). In particular, Lebas (1995) identified five general reasons for measuring performance – namely, to understand (a) where one has been; (b) where one is now; (c) where one wants to go; (d) how one is going to get there; and (e) how one will know when he or she gets there. The performance management literature makes it clear that to provide meaningful information, PIs must be aligned with the purpose of the system that they are measuring. Furthermore, the true value of PIs is the role that they can play to establish a continual cycle of performance improvement based on both understanding past and current performance (a and b) and having a clear vision for the level of future performance expected (c).
The US National Research Council Committee on Measuring and Improving Infrastructure Performance (1996: p. 5) observed that ‘[i]nfrastructure is a means to other ends, and the effectiveness, efficiency, and reliability of its contribution to these other ends must ultimately be the measures of infrastructure performance’. This acknowledgement that the primary function of infrastructure is to enable other outcomes is also supported by countries, including New Zealand (Beca and Covec, 2013), and individual sectors – for example, aviation (Airports Council International, 2012). New Zealand’s National Infrastructure Unit is developing a framework for infrastructure PIs that focuses on welfare obtained from infrastructure, as well as the infrastructure itself. A comprehensive report produced as part of that work (Beca and Covec, 2013) makes a number of important points on strategic PI design. These include (a) the proposal that the flow of welfare created by the infrastructure stock can be used to measure infrastructure performance in terms of the degree to which infrastructure enables other activities; (b) infrastructure PIs must always be interpreted with regard to trade-offs, because it is not always possible to increase one indicator without reducing one or more others; (c) strategic PIs are those that facilitate decisions that support ‘better use of existing infrastructure’ and ‘better allocation of new investment’; and (d) the specific characteristics of infrastructure should play a role in dictating the type of infrastructure PI chosen for that system.
The Airports Council International (2012) guide highlights the importance of knowing the purpose of measuring system performance before designing PIs for any system and advocates a stepwise approach to PI development in infrastructure. The first step is to identify key performance areas (KPAs), to provide purpose to PI development for the system being measured. The second step is to define at least one PI per KPA, and the third step is to use the PIs to evaluate and improve performance. The whole process is iterated through frequent consultation with users and other interested parties.
Given the complexity of infrastructure systems and the various interdependencies that exists between them (Rinaldi et al., 2001), it is not sufficient to monitor performance solely on an infrastructure-by-infrastructure basis. Performance in one infrastructure can influence that in others. Resilience also emerges as a challenge of system interdependency. For example, smart technologies which may result in more efficient infrastructure may lower the case for additional capacity – that is, applying smart or any modifications at the sectoral level may decrease resilience at the level of the broader infrastructure system. Furthermore, while one infrastructure sector may be internally relatively resilient, if it relies on another that is less resilient, vulnerabilities emerge (Engineering the Future et al., 2011). Therefore, a clear case to define strategic PIs at the whole national infrastructure system level can be made based on the need to manage interdependence and create resilient infrastructure.
Mazur and Zabierek (1997) considered how outcomes could be incorporated into transport planning. They concluded that output measures are important for management and planning activities and can be used for assessing outcomes but PIs that evaluate outcomes are the best tool for assessing the effectiveness of transportation programmes and funding decisions. The Delaware Department of Transportation’s tiered system of performance measures uses both outcomes and output indicators that correspond to goals, strategies and policies in longer-term strategic plans. The outcome performance measures monitor the achievement of goals and strategies, whereas output measures indicate progress towards that achievement (Abbott et al., 1998). In the UK, recent research on multiutility infrastructure operators (Roelich et al., 2015) advocates an approach to infrastructure decision-making that incorporates the needs, behaviours and technological choices of end users and focuses on the services provided as infrastructure outcomes. Similarly, Blom and Guthrie (2015) advocated outcome-based infrastructure and promoted change from a technical-led to a service-led approach to infrastructure management, driven by customer perception of road surface conditions in New Zealand. Thus, both support the idea of reconnecting performance measurement with the end users’ desired outcome(s) because it is the purpose of infrastructure to enable or facilitate such desired outcomes.
In their work relating to Australian infrastructure, Sharp et al. (2014) identified and defined four characteristics of infrastructure systems that PIs can measure. These are (a) inputs (for example, the level of capital investment); (b) outputs (the specific infrastructure built); (c) Outcomes (the purpose of the infrastructure) and (d) impacts (factors occurring as an indirect result of a and b). They recommended that, in situations where the results enabled by infrastructure are of strategic interest, PI sets should be designed to measure outcomes. PIs for measuring outcomes for six Australian infrastructure sectors – water, energy, communications, roads, rail (freight) and rail (commuter) – are then proposed using four groups of outcomes that affect the quality of service as experienced by system users: reliability, stability, safety and resilience.
This approach makes clear the importance of focusing on outcomes and provides a common terminology that may enable collaboration across sectors. However, no attempt is made to develop a higher set of societal-level outcomes or to give infrastructure (a system of systems) a strategic purpose as an enabler of the outcomes society desires.
In summary, a review of infrastructure performance measurement literature and international comparison has drawn out the following lessons of central importance to the process of developing outcome-oriented PIs for infrastructure.
The primary function of infrastructure is to enable desired outcomes. Therefore, the role of strategic PIs should be to evaluate how effectively infrastructure contributes to those desired outcomes rather than to measure directly the performance of the infrastructure itself.
PIs must be aligned with the purpose of the system that they are measuring.
A continual cycle of performance improvement is required, based on understanding past performance, current performance and a clear vision of the expected level of future performance.
Strategic PIs should be constructed through a sequence of purposeful steps, instilling them with traceable foundations.
Indicators need to be able to assess performance and strategy across sectoral boundaries, as well as the performance of the interdependencies between infrastructures that influence them.
Stakeholder workshop to elicit expert perspectives
A workshop was held on 2 March 2015 with 29 UK-based infrastructure practitioners from both the industry and academia. Experts in infrastructure finance, delivery, operation, regulation and the economic, climate change and sustainability issues affecting infrastructure systems attended the event. The main purpose of the workshop was to synthesise a collective understanding of what infrastructure stakeholders need from strategic infrastructure PIs if the indicators are to enable forward-looking decision-making. The workshop was structured using five questions (Table 2). These questions were developed following the critique and review detailed in ‘A critique of infrastructure PIs in the UK National Infrastructure Plan’ and ‘Lessons from performance management literature and international best practice’ sections. An explanation of the purpose of each question is provided in Table 2.
Industry engagement questions and purpose
| Question | Question purpose | |
|---|---|---|
| 1 | As an end user of infrastructure, what are the characteristics of the services it delivers that you value most highly? |
|
| 2 | From the perspective of the major infrastructure industry sectors, what do you think about current infrastructure PIs? |
|
| 3 | What information must indicators for making decisions about the future communicate? |
|
| 4 | From your sector’s perspective, what performance information would be valuable from other infrastructure sectors (e.g. those in your supply chain) to make decisions about improving your own sector’s performance and the experience of your end users? |
|
| 5 | From your sector’s perspective, what performance information would be valuable that makes sense only at the system-of-system level to make decisions about improving your own sector’s performance and the experience of your end users? |
Service characteristics valued by end users
Based on responses to question 1 (Table 2), 12 broad infrastructure characteristics of value to end users were identified: communications, uninvasiveness, availability, safety, reliability, sustainability, inclusivity, flexibility, low impact, affordability, comfort and choice. These characteristics are noteworthy only as an indicative snapshot of the type of outcomes that end users expect infrastructure to enable. The most salient finding arising from this is the difficulty that attendees had in establishing consensus when discussing this question. This observation challenges the assumption that infrastructure practitioners know precisely what their customers want, suggesting that open-ended consultation with infrastructure users is needed if PIs are to be aligned with the needs of end users.
Industry perspectives on the current PIs for infrastructure as published in the NIP
In response to question 2 (Table 2), workshop attendees identified a number of challenges that prevent their organisation or sector from using the NIP PIs in the form in which they are currently published. Significantly, there was consensus on the following.
The purpose of strategic PIs must be clearly stated.
This is unclear for those published in the UK NIP.
PIs must be forward-looking and aligned with a shared strategic vision of national infrastructure if they are to inform strategic decision-making.
The indicators in the UK NIP are a backward-looking statement of observable fact.
PIs cannot meaningfully inform strategic infrastructure system planning if they provide no insight into the gap between current infrastructure system performance and the desired outcomes that infrastructure is expected to enable.
At present this is not the case in the UK.
Additionally, a number of other themes were identified as important challenges. The indicators can be difficult to interpret; are overly constrained in practice by regulatory time horizons; are not connected to adaptation; and are not aligned with existing industry indicators. Furthermore, the presentation of indicator data and their granulation, aggregation and normalisation lack clarity, and little consideration has been given to the perverse incentives unfit for purpose that PIs can potentially cause.
PI characteristics and future strategic decision-making
In response to question 3 (Table 2), consensus emerged among workshop attendees regarding the importance of five general characteristics that they expect future-facing strategic PIs to embody. These are joined-up, transparent, flexible, forward-looking and outcome-focused.
Furthermore, the following characteristics of strategic indicators were also proposed by more than one workshop attendee during discussion in response to question 3
relate to public expectations and needs
relate to controllable parameters
take account of the dynamics of systems, including the behaviours of actors in the system
acknowledge different types of indicators, the influence of standards and their origins
allow for intelligent design
identify opportunities
have purpose and vision
account for different audiences
have sufficient scope
communicate uncertainty
account for the whole system and whole life cycle
account for the wider context.
Understanding cross-sectoral and systemic need
Responses to questions 4 and 5 (Table 2) revealed several additional factors which workshop attendees consider important to the design of strategic national infrastructure PIs. These include the level at which the system is considered (i.e. individual asset, group of co-located assets, subsector, sector or whole system) and the temporal components and geographic scale of the system being measured. Furthermore, there is a need to create shared understanding of the purpose of performance measurement and the potential benefits that can be created for the entire system by sharing relevant information. For example, sharing data can be very important in infrastructure failure recovery. It is also useful for understanding wider benefits and the impacts of decisions and actions taken with the whole infrastructure system.
A conceptual approach to outcome-oriented infrastructure PIs
This section builds on and synthesises findings from the critique, review and workshop sections of this study (‘A critique of infrastructure PIs in the UK National Infrastructure Plan’ to ‘Stakeholder workshop to elicit expert perspectives’ sections). It proposes and gives an overview of a series of interconnected research outputs based on this synthesis. Collectively, these outputs provide a conceptual approach that can be used to develop outcome-oriented PIs for the regular and meaningful evaluation of the extent to which infrastructure enables the desired outcomes to which it is expected to contribute and whether changes to infrastructure strategy are needed to ensure that infrastructure performance remains fit for purpose in the face of future challenges (i.e. any change to the context in which infrastructure is embedded). The components of this conceptual approach are
a set of guideline criteria specifying useful characteristics of indicators for the strategic evaluation of infrastructure performance (Table 3)
a set of design principles to ensure that any process for developing PIs in response to objectives a and b in ‘Introduction’ section produces indicators consistent with the guideline criteria (Table 4)
a high-level process for developing PIs directly aligned with the desired outcomes which infrastructure is expected to enable (Figure 2).
Guideline criteria for evaluating PI design
| Characteristic | Description |
|---|---|
| Meaningful | Easy to interpret and unambiguous. |
| Purposeful | Designed for an explicitly stated purpose. |
| Strategic (linked to outcomes) | Designed to provide meaningful feedback on progress towards desired outcomes. |
| Future focused | Designed to measure elements of performance relevant to the future of the infrastructure system in question. |
| Systemic | Clearly linked to the system that they are measuring. Each PI should be part of a meaningful multidimensional set that collectively gives a view of system performance. Indicators should not be used in isolation for the purpose of optimising individual system elements. |
| Transparent | The underlying data source(s) for any indicator need to be declared, as do methods and justifications for any calculation/aggregation/normalisation performed to create the indicator. Where an indicator is aggregated from multiple data sources, it should be published alongside the indicators that comprise it. |
| Relevant presentation | Relevant information can be hidden by average or normalised values. A range of formats should be considered for publishing indicators such that the indicator meaningfully communicates behaviour linked to its strategic purpose. |
| Geographical scale | Indicators should be presented on a geographical scale relevant to decision makers. Where national data are published, appropriate regional data should be made available. |
| Reviewed frequently | Indicators should be frequently reviewed in terms of whether they remain fit for purpose and aligned with strategic outcomes. Indicators no longer aligned with these elements should be removed or adapted to ensure that the indicator remains meaningful. |
| Not data constrained | Strategic elements of performance or outcomes that need to be measured should be identified before issues of data availability are considered. Where data are unavailable or not, clear justification of the use of alternative indicators should be provided. |
| Objective and neutral | Not framed in terms of specific technologies or infrastructures that can be used to enable the desired outcome. |
| Encourage innovation | Send clear signals to infrastructure industries by being consistent with strategic outcomes. |
| Reflect stakeholders’ needs | Designed to provide relevant information to stakeholder groups who may use them. |
| Capture multiple stakeholder perspectives | Indicators should provide measures of performance relevant to a range of infrastructure stakeholder perspectives. |
Design principles of a process for developing PIs
| Principle description | |
|---|---|
| Principle 1 | Establish an agreed and explicitly stated long-term vision comprising the socially desired outcomes that infrastructure is expected to enable before designing PIs. |
| Principle 2 | Create a traceable audit trail of design decisions (and their justification) at every stage of the process. |
| Principle 3 | Analyse and decompose each desired outcome into a set of outcome dimensions and subdimensions that collectively represent the desired outcome. Note: This is necessary because outcomes are typically multidimensional and difficult to measure directly. |
| Principle 4 | Frame desired outcomes and outcome dimensions in solution neutral terms – that is, when framing desired outcomes, do not make reference to specific technologies or infrastructures that can be used to enable the desired outcome. For example, a desired outcome connected to energy should avoid specifying which technology is used to generate the energy. |
| Principle 5 | Avoid making decisions based on current data availability. The process should aspire to create indicators aligned with those elements of performance that need to be measured, not just those already measured. |
| Principle 6 | Produce a set of indicators for each desired outcome. The indicator set should comprise (a) one or more partial indicators for each of the outcome dimensions identified during desired outcome decomposition and (b) a headline indicator (outcome-oriented PI) for each desired outcome. Explanation of how the headline indicator for a desired outcome is calculated from the partial indicator set must be provided alongside the indicator set. |
| Principle 7 | Whenever indicators are published, each indicator should be accompanied by a statement of their intended value to the audience they are designed for; these statements should be an output from the indicator design process. |
| Principle 8 | Make the process collaborative, such that implementation of the PI development process involves all affected stakeholders at all stages. Understanding of the purpose of infrastructure, how that purpose can be fulfilled and the reason for the PI developed are all likely to be improved by making the process inclusive and transparent in this way. |
| Principle 9 | Ensure that the process is sufficiently flexible to be revisited and modified in the light of data availability, without modifications affecting the underlying purpose of the PI. |
| Principle 10 | Any process for PI creation needs to be flexible such that it can be applied at any geographic scale, in any sector and at a cross-sectoral infrastructure scale, to allow comparisons between different regions and different sectors (where appropriate) and to promote a systemic perspective that recognises actions in one sector to achieve performance targets that can impact on the ability of other sectors to achieve performance targets. |
Guideline criteria
The purpose of these guideline criteria (Table 3) is to provide a checklist of characteristics (a) important to consider when developing new PIs and (b) against which any set of PIs can be evaluated to assess the extent to which they meet the needs of strategic decision makers. By design, the guideline criteria are not 100% independent of one another. Rather they are intended to be a complementary, overlapping and interconnected set that reflects a range of nuanced perspectives on the desirable characteristics of strategic PIs.
Design principles for outcome-oriented PIs
Process for outcome-oriented PI development
The final component of this conceptual approach is a simple process for the development of outcome-oriented PIs that aims to embody the criteria and principles mentioned earlier. Figure 2 gives an overview of this process in terms of five steps, the purpose of each step and the outputs that arise from application of each step. Discussion of process design and application, including the potential for systemic application, is presented in ‘Discussion’ section.
Discussion
Interpretation of outcome-oriented PIs
The outcome-oriented PIs produced through application of this process should not be interpreted in isolation from the set of outputs produced through stepwise application of the process. The conceptual map illustrating decomposition of a desired outcome (step 2), the set of partial indicators for the outcome dimensions and outcome subdimensions (step 3) and the formula used to calculate the outcome-oriented PIs (step 4) all provide a vital context to justify and interpret the strategic significance of the outcome-oriented PI. As importantly, the intermediate outputs listed earlier must not be interpreted in isolation from the desired outcome that they comprise.
Process assumptions
The mentioned process (Figure 2) is underpinned by a number of important assumptions. The majority of these can be justified based on the research presented in this paper; in some cases, further research to test the validity of the assumption would be beneficial.
The purpose of infrastructure is to enable desired outcomes (step 1).
In the absence of clearly stated desired outcomes, strategic measurement of infrastructure performance is not possible (entire process).
It is possible to reach consensus between multiple stakeholder groups on what these desired outcomes are and what their relative importance is (step 1).
Desired outcomes are multidimensional. It is possible to decompose any desired outcome into a meaningful set of lower-level outcome dimensions and outcome subdimensions and to produce a conceptual map of the interconnections between these (step 2).
The process of desired outcome decomposition can continue until a complete set of measurable outcome dimensions and outcome subdimensions has been identified (steps 2 and 3).
The partial indicators for outcome dimensions and outcome subdimensions can be meaningfully aggregated to produce an outcome-oriented PI (step 4).
If presented as a set, the outcome-oriented PI, the conceptual map of decomposition and the partial indicators provide a strategic resource to improve understanding among infrastructure practitioners (entire process).
Systemic and collaborative application
Prior to applying the process (Figure 2), it is necessary to decide at what level (i.e. within what geographical and technical boundaries) infrastructure performance is to be measured. In theory, the proposed process is applicable to a range of geographical (i.e. regional, national, multinational) and socio-technical (i.e. single-sector, cross-sectoral, infrastructure system-of-system) scales. This is because desired outcomes can be framed at any geographical and socio-technical scale, and the process produces outcome-oriented PIs consistent with the level at which the desired outcomes are framed. Therefore, by framing desired outcomes at the chosen scale, outcome-oriented PIs capable of measuring performance at that scale can be developed. For example, if desired outcomes are defined at the sectoral level, infrastructure performance can be evaluated only at the sectoral level. However, if desired outcomes are defined at the infrastructure system-of-system level, it then becomes possible to define meaningful PIs to evaluate cross-sectoral performance.
The flexibility of the proposed process creates the opportunity for a more systemic evaluation of strategic infrastructure performance against higher-level desired outcomes. This is important because many elements of infrastructure performance are best understood, managed, measured and mitigated at the whole-system level (e.g. system resilience, societal carbon dioxide emissions, flooding risk, vulnerability to cascade failures and adaptation to changing weather patterns and sea level rise to an by climate change). Therefore, it is advocated that a set of cross-sectoral desired outcomes be developed at the system-of-system level, as an essential first step in producing system PIs for infrastructure.
Collaboration across infrastructure sectors, with infrastructure users and with those responsible for the regulatory and governance structures that affect infrastructure is essential if such a set of cross-sectoral desired outcomes is to be developed. Furthermore, the process of desired outcome decomposition and conceptual mapping (step 2) will require similar collaboration.
Further benefits
Collaborative implementation of the process (in particular steps 1 and 2) will yield benefits beyond the creation of systemic outcome-oriented PIs. If done effectively, it can increase collective understanding of the desired outcomes that infrastructure is expected to enable and why; the interdependent relationships between these desired outcomes; the drivers of decision-making in different infrastructure sectors; and how society perceives and what society expects from infrastructure.
Additionally, a set of cross-sectoral desired outcomes at the system-of-system level can be beneficial when evaluating other infrastructure challenges. For example, when undertaking strategic need assessment to evaluate what infrastructure will be needed in the future, identification of desired outcomes (step 1) can provide a starting point from which infrastructure need can be assessed (where need is defined as the gap between current and projected future provision and societal expectations of infrastructure). Therefore, this research can support the UK National Infrastructure Commission in the once per parliament exercise of conducting a National Infrastructure Assessment.
Furthermore, if desired outcomes are framed in solution-neutral terms (see objective and neutral criteria in Table 3 or principle 4 in Table 4), any outcome-oriented PI based on that desired outcome will provide a solution-neutral measure of infrastructure performance and avoid the creation of inadvertent bias towards, or lock-in to, specific infrastructure solutions.
Application of the process
A complementary paper applies the process in Figure 2 to two case studies (Carhart et al., 2016). The case studies refine the application of the process and demonstrate that it is able to produces indicators linked to outcomes. The case studies also demonstrate the limitations of a sector-by-sector approach, therefore strengthening the case for the prioritisation of the identification and definition of a system-wide set of desired outcomes.
Conclusions and recommendations for practical application
It is recommended that an outcome-oriented methodology (Figure 2) be adopted in the design of strategic PIs for infrastructure. Such an approach emphasises the interdependencies within the infrastructure system necessary to create desired outcomes and therefore contrasts with the mainstream approach that has largely focused on performance measurement within sectors rather than at the level of the whole infrastructure system.
An outcome-oriented methodology has many advantages, including the opportunity to catalyse debate over desired outcomes; the flexibility to design strategic PIs at a range of different scales; the ability to align strategic performance measurement with future aspirations rather than past performance; and the creation of an evidence base against which indicator design can be defended and reviewed. Additionally, the process defined in Figure 2 provides a transparent framework for decision-making; makes unstated assumptions explicit; aligns performance measures with stated priorities; requires all intermediate decisions made during the process to be recorded and justified; and makes an important distinction between developing a vision and strategy and designing PIs to evaluate the performance of that strategy.
Furthermore, this process proposed in Figure 2 is directly applicable to other areas of infrastructure decision-making. In particular, it can be applied to the development of a methodology to assess what infrastructure will be needed in the future. More broadly still, a transferable methodology that establishes purpose by defining desired outcomes and aligning decision-making to the objective of delivering these outcomes is applicable to all areas of government decision-making.
Therefore, to facilitate the adoption of an outcome-oriented approach to strategic infrastructure PIs and the potential broader benefits of an outcome-oriented approach, the following collaborative actions are recommended to the infrastructure industry (practitioners, owners, regulators, policymakers and governments)
purposeful societal-wide discourse, to identify and structure the desired outcomes that the nation or region of interest collectively expects infrastructure to play a role in enabling
based on the above, an explicit statement of the desired outcomes that define the purpose(s) of the nation or region’s infrastructure
creation of a long-term vision or strategic plan for the nation’s infrastructure that reflects these outcomes that can be developed within the context of enabling them.
From this position, it is possible to develop and implement meaningful strategic PIs that monitor the efficacy of the infrastructure system in supporting the delivery of these desired outcomes and therefore better manage the system in the service of the nation.
Acknowledgements
The research presented here and in a second paper (Carhart et al. (2016) – also published in this issue) was undertaken as collaborative research project between two UK-based infrastructure research groups, funded by the Engineering and Physical Sciences Research Council and Economic and Social Research Council (the International Centre for Infrastructure Futures (ICIF)) (EP/K012347/1) and Infrastructure Business Models, Valuation and Innovation for Local Delivery (iBuild) (EP/K012398/1). The research was initiated following a request to reexamine how strategic performance indicators can be used to evaluate infrastructure performance and improve on the performance indicators used in the UK National Infrastructure Plan. A technical report of findings from this research is available from iBuild (2016) and ICIF (2016). The authors wish to acknowledge and thank all those who participated in the workshop described in this paper. The conclusions and recommendations are those of the authors alone and do not necessarily represent those of IUK or those who participated in the workshop.
