Performance measurement provides critical information for strategic decision-making about the future of national infrastructure provision. The process of developing appropriate performance indicators must be based on an understanding of the high-level desired outcomes that infrastructure systems are intended to facilitate. These outcomes may be complex and dynamic and vary across the spectrum of infrastructure stakeholders. This paper demonstrates a conceptual process for developing outcome-related performance indicators by using case studies from the UK rail and water sectors. The case studies show that the process can provide an industry-specific picture of desired outcomes across the main stakeholders, together with their dimensions and associated performance indicators (so-called partial indicators). The work highlights potential shortfalls of a sector-by-sector approach to outcome-oriented strategic performance indicators and suggests that further work is required to integrate partial indicators to provide a holistic picture that recognises the contributions often made by a variety of infrastructures to a given high-level desired outcome.
Introduction
A suite of infrastructure performance indicators is required to help steer UK infrastructure projects towards meeting society’s desired high-level outcomes – for example, a strong economy, wider home ownership and better communications. Over the past few decades, UK infrastructure investment has been less effective than it should have been in contributing to those outcomes (HM Treasury and Infrastructure UK, 2011). The government is now seeking to address the problem with a pipeline of infrastructure projects out to 2021, valued at £483 billion (IPA, 2016); therefore, it is vital to establish clearly the links between infrastructure projects and the contribution that they make to society as a whole.
Two research projects, iBUILD (Infrastructure Business Models, Valuation and Innovation for Local delivery) and ICIF (International Centre for Infrastructure Futures), jointly funded by the Engineering and Physical Sciences Research Council (EPSRC, 2016) and the Economic and Social Research Council (ESRC, 2016), have investigated the approach that was used by Infrastructure UK (IUK; subsequently merged with the Infrastructure and Projects Authority), resulting in the creation of a five-step conceptual process for the development of new, outcome-oriented performance indicators.
An earlier paper (Dolan et al., 2016) argues the need within the UK for infrastructure performance indicators that can help to measure infrastructure’s contribution to high-level desired societal outcomes and to decide whether strategic changes are necessary to ensure that infrastructure performance remains ‘fit for purpose’. Current indicators used for managing national infrastructure in the UK, and indeed elsewhere, do not meet this need. Through a consultation exercise, Dolan et al. (2016) identified five general characteristics of importance in future-facing strategic performance indicators: joined-up, transparent, flexible, forward-looking and outcome-focused. This process also highlighted the appetite of government and infrastructure practitioners for indicators that relate directly to public expectations and need, allow for intelligent design and account for different audiences.
Dolan et al.’s (2016) five-step process is summarised in Figure 1. It is based on international best practice, emerging theory and the aforementioned consultation. It is essentially forward-looking, in contrast to the UK’s current backward-looking method, which covers the nation’s key infrastructures (major roads, rail network, airports, container ports, electricity network, gas network, flood risk prevention and waste disposal), and uses data to establish historic performance trends (HM Treasury and Infrastructure UK, 2010, 2011, 2012, 2013, 2014).
This paper describes two case study applications of the process within the water and rail sectors. These provide proof of concept for the first three steps of the process. Research on steps 4 and 5 is still ongoing and therefore is outside the scope of this paper. To this end, both applications set out to illustrate the proposed process and, in doing so, to demonstrate that it is possible to construct indicators from existing data that show explicitly an infrastructure’s performance in facilitating a societally desired outcome.
The paper begins with some background: recapping the need for the process, briefly describing the five steps, and exploring the interfaces between government, infrastructure operators and users and the differing points of view that they have regarding infrastructure outcomes. The paper then presents and discusses the two applications before arriving at some conclusions and recommendations.
Background
A nation’s infrastructure exists to facilitate and enable society’s high-level desired outcomes: from health and well-being to prosperity and growth. Long-term decisions about the provision of strategic infrastructure require knowledge of these desired outcomes and the role that infrastructure plays in delivering them. An infrastructure system is performing well if it is effectively and satisfactorily enabling society to achieve its desired outcomes; conversely, it is performing poorly if it does not.
Strategic performance indicators for infrastructure are required that can show the extent to which infrastructure is enabling society to achieve its desired outcomes (Mihyeon Jeon and Amekudzi, 2005). The indicators must be based on a good understanding of those outcomes and the ways in which infrastructure facilitates them. This is largely not the case with the indicators currently used; in many cases, they track infrastructure outputs only (Network Rail, 2015a). For example, in the case of rail, infrastructure failures relative to train-kilometres, while useful for managing day-to-day operations, is of limited use in isolation to national strategy.
The generic, high-level, five-step process for the development of outcome-oriented, strategic performance indicators for infrastructure (Dolan et al., 2016) is outlined in Figure 1. Step 1 involves identifying and drawing together in a coherent way the desired outcomes. Step 2 is an exploration of the dimensions relating to each outcome – for example, if the desired outcome is a secure water supply, what are the factors (outcome dimensions) characterising that? (Note: the dimensions do not necessarily have any direct causal effect on the outcome.) Step 3 involves the selection of partial indicators for each dimension; these are the things that will be measured to assess dimension performance and ultimately synthesised along with other partial indicators to provide a desired outcome performance indicator. Synthesis takes place in step 4; step 5 is about the availability of data required to support the five-step process.
Successful completion of the first three steps provides the basis for assessing the feasibility of the conceptual process. Step 4 concerns a non-trivial decision for those using the indicators to decide the relative importance of each partial indicator to form a single outcome-oriented headline indicator and even whether they wish to aggregate the data in this way at all. Such aggregation can be achieved through many widespread approaches that are not the focus of this study. Step 5 is included as the process recommends the identification of further indicators where necessary. As this study, with its aim of demonstrating the feasibility of the process, draws only from existing indicators, it is somewhat moot here.
The infrastructure performance indicator development process operates in the context of the relationships between the principal stakeholders and the differing points-of-view that each stakeholder has with regard to infrastructure and the desired outcomes. For the purposes of the case studies, the principal stakeholders have been assumed to be the national government, infrastructure regulators, infrastructure managers (otherwise referred to as the industry) and infrastructure users.
Both of the case studies described here are set within the context of the UK’s operational, regulatory and governance systems. A model of this operational context is illustrated in Figure 2, which shows for each of the principal stakeholders the following: components of the existing infrastructure development and supply process (circles shaded grey); the proposed new components (circles shaded white) supporting the development of the new performance indicators; and the relationships linking them together. The stakeholders are each operating under a wide range of different incentives and constraints, with the result that they bring to the issue of high-level desired outcomes a variety of points of view. These different points of view and their associated indicators must be synthesised, as shown in Figure 2. Care needs to be taken with these to ensure that there is no confusion over the terminology employed that could hinder identification of outcomes, dimensions and partial indicators (i.e. steps 1–3 of the process).
National government takes the lead in setting the high-level desired outcomes, driven principally by the aims of the ruling political party, but shaped by appropriate factual evidence, tempered by considerations of implementation feasibility and subject to a process of consultation facilitated by green papers (Civil Service, 2013; Hallsworth and Rutter, 2011; IFG, 2011; PASC, 2007, 2014; Parliament, 2016a; STC, 2006). Desired outcomes are used to inform development of policy, which is reified through the publication of white papers and the making of legislation/regulatory guidance in Parliament (2016b). Policy is implemented through strategy (PASC, 2010); however, there is an increasing tendency for the government not to specify strategy and leave the supply chain to decide how best to deliver.
At the industry level, companies are required to operate according to their licences while at the same time fulfilling their duties to their shareholders. In the case of rail, Network Rail is, broadly speaking, the owner and operator of the national UK rail network. It is a ‘not-for-dividend’ company, so it does not have shareholders in the normal meaning of the word; however, it is accountable to the Office of Rail and Road (ORR), which determines the income received from the government and the outputs that must be achieved during each 5-year control period (Network Rail, 2016a). ORR monitors the performance of Network Rail against these and a wide range of other operational outputs (ORR, 2016a); collectively, they are referred to as industry outcomes in Figure 2. Passenger services are operated on this network by train operating companies granted franchises to operate, through competitive bids, by the Department for Transport. These are further complemented by commercial freight operators.
The situation, with regard to conflation of outputs and outcomes, is broadly similar in the water industry; however, the physical and organisational arrangements are somewhat different. Regional companies providing water and sewerage services to domestic and industrial users own and operate the physical infrastructure, with most having shareholders who expect a commercial rate of return on their investment. The economic regulator, the Water Services Regulation Authority (Ofwat) sets prices in consultation with the operating companies and fulfils a similar role to ORR and steers the industry through a series of 5-year control periods. The Drinking Water Inspectorate regulates the quality of public supplies, while the Environment Agency also regulate in relation to abstraction, reservoirs and other environmental matters.
The end-user viewpoint is challenging to distil given that there are many individuals acting independently. However, groups like Passenger Focus and the Rail Freight Group for the rail industry do collect information on customer perception of service, which is fed back to the industry; as will be seen later in the water case study, the water industry is working with its users under auspices of Ofwat. Otherwise, users are left with their own ideas of what sort of service infrastructure should provide and their perceptions of how well infrastructure companies, and by implication government, are delivering.
Case study applications of the proposed outcome-oriented performance indicator development process
The first three steps of the process are described in the next sections for the water and rail sectors. Identifying the intended outcomes of national government policy presents general and specific challenges. Firstly, the intended outcomes of national policy are not always explicitly stated and may be open to interpretation. Secondly, the timing within a parliamentary cycle can impact on the clarity and availability of national government policy. Finally, as the outcomes enabled by infrastructure can be far reaching, there is no simple way to identify which policies rely on which infrastructure systems. Indeed, when adopting a system-of-system view of infrastructure, it is conceivable that specific systems contribute to outcomes far removed in time and space from the assets and processes of that system – for example, the outcomes enabled by the rail system are not limited to the policies of the Department for Transport. To address these issues, different approaches were implemented for each of the case studies.
UK water sector
Step 1 of the five-step process requires the identification of the outcomes desired from each of the four stakeholder-viewpoints. At the time this exercise was undertaken in mid-2015, the UK government was in a period of post-election transition; therefore, for the purposes of this exercise, policies from the end of the 2010–2015 government were used to identify desired water-related outcomes, in particular, policies owned by the Department for Business, Innovation and Skills and the Department for Environment Food and Rural Affairs (see Table 1).
Ofwat and the water companies have implemented a framework for identifying the desired outcomes of the sector’s end users. For the 5-year control period from 2015 to 2020, the UK’s water companies identified 171 outcomes valued by their customers (Ofwat, 2014). The outcomes covered ‘essential elements’ such as excellent water quality, protection of the natural environment and reliable waste water services. Extensively comparing the 171 outcomes of the different water companies shows a significant overlap, making it possible to condense the list into the 12 generic ‘End user’-desired outcomes shown in Table 1. For example, Northumbrian Water (2014) stated, ‘We protect and enhance the environment in delivering our services’, while United Utilities (2013) said, ‘The natural environment is protected and improved in the way we deliver our services’.
Discussions with those involved in UK infrastructure governance identified ten generic outcomes (see Table 1) that may be important from this viewpoint. While these may have some overlap, they provide important desirable outcomes that are understandably not highlighted by the end users of any individual water company – for example, equality between regions and meeting international standards.
Harmonising the entries in Table 1 from across the four viewpoints, a list of 17 outcomes can be constructed
(a) increased competition within the UK
(b) internationally competitive
(c) protected environment
(d ) dependable water supply
(e) suitably located supplies
( f ) regional homogeneity
(g) safe, clean drinking water supply
(h) safe, dependable sewerage treatment
(i) sustainable freshwater fisheries
( j) efficient customers
(k) improved communities
(l) affordable, accurate and clear bills
(m) satisfied customers
(n) safe operations
(o) skilled workforce
(p) reputable operators
(q) fair and efficient operators.
It is clear that these do not exist in isolation from one another – for example, a ‘dependable water supply’ may have an influence on the achievement of ‘satisfied customers’ as an outcome.
Each individual outcome can be complex and multidimensional, often evidenced by the nuanced ways in which they are described by different stakeholders. Step 2 of the five-step process seeks to identify the ‘dimensions’ of each desired outcome. This is illustrated here using an outcome common to all four viewpoints: the provision of a dependable water supply. While ‘dependable’ was used as a collective term for the outcome expressed by the water companies’ end users, their descriptions of dependability reveal additional facets that aligned with the governance and national policy viewpoints shown in Table 1. For example, 30% of the companies used resilience in their descriptions and 30% used reliability. Other expansions on the meaning of dependable in this context include
sufficient to meet the needs of current and future generations (Northumbrian Water, 2014)
supply meets demand (Anglian Water, 2013)
available when customers require it (Thames Water, 2013)
always there when it is needed (Severn Trent Water, 2015)
keeping supplies on (Severn Trent Water, 2015)
constant (Southern Water, 2013).
Thus, the dependable water supply outcome includes the notion of a resilient, reliable, sufficient and secure supply; this is reflected in the alignment with resilience, reliability and security identified by the national policy and governance viewpoints. These four characteristics therefore become the dimensions for the dependability outcome (see Table 2).
Some of these dimensions may be approaching something close to a measurable variable; some, in fact, may be directly measurable. Step 3 of the process seeks to identify the measures appropriate for each of these dimensions, each measure being a ‘partial indicator’ of the fulfilment of the related outcome. It is important to note that identification of partial indicators is not predicated on the pre-existing availability of the necessary data nor, indeed, if the data are not available, on the identification of ways to overcome problems associated with obtaining it. With that in mind, Table 3 suggests potential partial indicators from those already used by the industry in order to demonstrate the practicality of identifying suitable measures.
The application of the first three steps of the process in this case study results in a transparent and explicit link between indicators already used by the UK’s water industry and the achievement of indicated high-level desired outcomes. The initial identification of desired outcomes shows that while the industry has undertaken valuable work to illuminate these from an end-user perspective (and in the very specific context of the water sector), the broader desired outcomes from a policy viewpoint are less clear.
While the desired outcomes related to the sector can be harmonised to a manageable number across the three viewpoints, the number of potential dimensions for these, and the ways in which they may interact with one another, suggest an increasing level of complexity.
The process outlined by (Dolan et al., 2016) is intended to embody a set of criteria and principles for performance indicators. Reflecting on the process as applied here, it can be said that the indicators are meaningful in that they are unambiguous and relate to both data already recorded and aspirational outcomes already identified and, in a related sense, that they are strategic, reflective of stakeholder needs and future focused. However, as the currently stated outcomes exist at a sector-specific level, there may be some constraints on their neutrality in terms of selecting technology to achieve the outcome. This emerges from the broad lack of an established long-term vision for the societally desired outcomes beyond the sector-specific context.
UK rail sector
The rail case study was also carried out in mid-2015 during a period of political transition. As an alternative to the approach used in the water case study, the government’s desired outcomes were elicited from the incoming government’s election manifesto and the High-level Output Specification (HLOS) for rail developed during the tenure of the previous government. The manifesto provided 21 desired outcomes: these ranged from ‘a strong economy’ and ‘better communications’, through ‘enjoyment of culture’ and ‘the big society’, to ‘prevention of terrorism’ and ‘a dignified retirement’. The principal outcome relating to rail was ‘better roads, trains and modern communications’.
The HLOS (DfT, 2012a) is a statutory requirement as set out by the 2005 Railway Act (DfT, 2005), which requires the Secretary of State for Transport to advise the Office of Rail Regulation (now ORR) what the government wants the railway to achieve during a 5-year control period. Thus, although the HLOS is a government document, it sets out railway industry outputs. However, the HLOS must be read in conjunction with the preceding command paper: Reforming Our Railways: Putting the Customer First (DfT, 2012b). This describes a vision for the future of the railways in England and Wales, which includes a range of outcomes: from ‘an engine for economic growth’, through ‘providing commuters with a safe route to work’, to ‘connecting communities with public services’.
Network Rail’s licence (ORR, 2014) to operate provides some insight into the fundamental outcomes expected of it such as a functioning and maintained cost-effective rail network. Network Rail’s stated vision is to provide a better railway for a better Britain: a railway that is safer, more reliable and more efficient than ever before and that will help to build a thriving, sustainable economy (Network Rail, 2016b).
The ORR is the regulator for the rail sector. It regulates across ten broad areas; perhaps those most relevant to the case study are regulation of Network Rail (the infrastructure operator), competition and investment. In the case of Network Rail, of particular importance is monitoring performance, to ensure that Network Rail delivers what it promised. To do this, ORR monitors a wide range of performance indicators. For competition, ORR aims to ensure that the rail market is competitive and fair – for passengers, freight customers, railway operators and taxpayers; and in terms of investment, ORR controls the process for those wishing to invest in the rail network.
The rail sector has not explicitly engaged with and integrated the views of its end users in the same way as the water sector; however, the ORR’s Consumer Expert Panel meets four times a year to ‘develop clear and robust evidence of consumer outcomes in rail’ (ORR, 2016b). The Consumer Expert Panel has not published their views on consumer outcomes, but Transport Focus has conducted several studies. For the purpose of obtaining the passenger view in the case study, two sources were used and cross-referenced, one looking at a specific aspect of rail and one looking at rail services in general
Future Merseyrail Rolling Stock: What Passengers Want (Passenger Focus, 2014a)
Rail Passengers’ Priorities for Improvement (Passenger Focus, 2014b).
The first identified eight outcomes of rail; the second identified 31. These were consolidated where similar to obtain 13 outcomes, which ranged from personal security and accessibility to reliable services and convenient ticketing.
Freight end users are represented by bodies such as the Rail Freight Group, which published a five-point plan for the future success of rail freight (RFG, 2014)
protect capacity
continue to invest in strategic freight network
stable environment for growth
meet the needs of freight customers
equity of approach to road and rail freight.
Some of these are inputs (investment) rather than outcomes, while others are perhaps too vague to be meaningful without further information (e.g. meeting the needs of customers). Nevertheless, these can be combined with those of the passenger groups to provide an end-user perspective. Table 4 provides a list of 19 outcomes consolidated from those of the main stakeholders described earlier.
The information captured in Table 4 already represents a step beyond the existing strategic performance indicators for national infrastructure (reported in the likes of HM Treasury and Infrastructure UK (2014)). It provides the basis of desired outcomes, from three different viewpoints, upon which forward-focused performance indicators can be based. The subsequent steps described below build on this information to identify the specific measures of system-level variability which contribute to the achievement of these outcomes.
With regard to outcome dimensions. the manifesto sets out projects to be delivered to achieve a transformed railway. Some of these appear in the HLOS along with the projects identified earlier. Network Rail’s outcome dimensions closely align with the HLOS as would be expected. ORR’s dimensions relate to the principal areas for which it has regulatory responsibility, and user dimensions are drawn from the documents referred to earlier. Table 5 shows an indicative example range of dimensions relating to the 19 outcomes in Table 4.
Table 6 provides an example of the sort of partial performance indicators that underlie the dimensions. The example is for Network Rail and the outcome/output ‘a safe railway’. The partial indicators are drawn from Network Rail’s (2015b) Annual Return 2015.
The application of the process here highlights the tacit way in which the high-level outcomes facilitated by the rail sector are articulated. When outcomes are discussed, it is specific to the direct experiences of rail transport. Table 4 shows the similarities and differences between the different stakeholders in terms of the theme and level at which outcomes are articulated. The example shown in Table 6 demonstrates that it is possible to construct a transparent chain linking existing indicators to the high-level outcomes over which they have a causal influence.
Reflecting again on the guideline criteria for evaluating performance indicators outlined by Dolan et al. (2016), while the variables in the right-hand column of Table 6 taken in isolation are not meaningful, or future focused, they take on these characteristics when presented in the context of their contribution to the desired outcomes.
Conclusions and recommendations for practical application
This paper has described the first three steps of a five-step process for the design of strategic performance indicators that help steer UK infrastructure projects towards meeting society’s high-level desired outcomes. Step 1 identifies and integrates the outcomes from the viewpoints of the principal stakeholders involved; step 2 identifies the dimensions characterising the outcomes; and step 3 identifies the so-called partial performance indicators, which measure dimension performance. Two case studies, one each from the water and rail sectors, have been used to explore and demonstrate the viability of the concept.
The research has highlighted two important issues regarding the identification of desired outcomes. Firstly, they can vary depending on the stakeholder viewpoint considered – for example, outcomes desired by end users may be different from those of the system operators or those with an overview of the wider, long-term infrastructure system of systems. Secondly, they are rarely stated explicitly; in many cases the causal relationship between the actions of the infrastructure system and the delivery of policy outcomes is tacitly understood at best. Despite these difficulties, the case studies suggest that with a little effort, the major stakeholder outcomes can be integrated such that there is a traceable path from user and industry outcomes, through governance outcomes, to government’s high-level desired outcomes.
The case studies have demonstrated the complexity potentially underlying the outcomes; this is often evidenced by the nuanced ways in which they are described by different stakeholders. Step 2 of the five-step process recognises this complexity with identification of the desired outcome dimensions. The performance of each dimension is monitored using partial indicators, partial in the sense that they monitor a part of the overall outcome. It is important to note that identification of partial indicators is not predicated on the pre-existing availability of the necessary data nor, indeed, if the data are not available, on the identification of ways to overcome problems associated with obtaining it.
The case studies look only at a sector-by-sector approach; they do not deal with national infrastructure as a holistic system of systems; this provides a focus on outcomes that are limited to that system rather than those which are akin to societal benefits. While this is sufficient for the purposes of demonstrating the process, it is not necessarily the most useful pattern of application. Many high-level societal benefits will require the co-operation of multiple infrastructure systems, and as such, a suite of outcome-oriented measures indicative of those benefits would need to consider multiple sectors. This is clear from the occurrence, even within a sector-by-sector approach, of such outcomes as a protected environment, a strong economy and improved communities. Furthermore, even those outcomes which appear on the surface to be within the boundaries of a single sector (e.g. safe, dependable sewage treatment) rely significantly on interdependencies with other infrastructure sectors. This is the focus of step 4 of the process, which is outside the scope of this paper.
The fact that outcomes are not mutually exclusive is an important issue that extends to consideration of the extent to which a given dimension influences multiple outcomes. The research shows the complex, closely coupled nature of the railway and flags up the need to understand the degree to which outcomes are related and the relative influence of dimensions. To produce a set of outcomes, outcome dimensions and partial indicators that help guide infrastructure development, more information is needed about the nature and strength of these relationships. It is recommended that this be the subject of further research.
Despite the term ‘outcome’ being formally used within the context of infrastructure policy as ‘high-level societal benefits’, it is often interpreted differently within industry. Indeed ‘outcome’ is on occasion used to describe the functional ‘outputs’ of the system. It would be helpful to the application of this process if agreement could be reached between the main stakeholders as to a shared terminology designed to reduce the risk of confusion.
It is interesting to note that while the top-to-bottom (i.e. from the government to industry by way of regulator) dissemination of outcomes, dimensions and indicators is very strong for rail, the involvement of the customer (both passenger and freight) in this is limited. Certainly the views of customers are sought, but it is not clear at what point, if at all, these are incorporated into the top–down process. It is recommended that fundamental to achieving any success in creating meaningful performance indicators, channels should be amended to facilitate a more bottom–up and holistic consideration and communication of desired outcomes.
The process applied here should be of interest to bodies such as the UK’s newly formed Infrastructure and Projects Authority, tasked as it is with developing an understanding of the country’s infrastructure needs and the means by which the fulfilment of those needs will be monitored. It also sits alongside efforts in New Zealand (Beca and Covec, 2013) and Australia (Sharp et al., 2014) to develop a more explicit link between infrastructure performance and societally valued outcomes. Crucially, however, these case studies underline the need for the desired outcomes that society expects infrastructure to facilitate to be defined. Only then can the purpose of the infrastructure systems be understood and their strategic performance meaningfully evaluated. It follows that policies relating to infrastructure systems must clearly define the outcomes that the systems of relevance are expected to facilitate. Without this it is not possible to construct meaningful performance indicators.
Acknowledgements
The research presented here and in a previous paper (Dolan et al., 2016) was undertaken as collaborative research project between two UK-based infrastructure research groups, funded by EPSRC and ESRC (the ICIF (EP/K012347/1) and iBuild (EP/K012398/1)). The research was initiated following a request to re-examine how strategic performance indicators can be used to evaluate infrastructure performance and improve on the performance indicators used in the UK National Infrastructure Plan. A technical report of findings from this research is available from iBuild (2016) and ICIF (2016). The authors wish to acknowledge and thank all those who participated in the workshop described in this paper. The conclusions and recommendations are those of the authors alone and do not necessarily represent those of IUK or those who participated in the workshop.
