A SCOR framework to measure logistics performance of humanitarian organizations Open Access

Global disasters have been increasing in diversity and severity for the past decades (IFRC, 2012). To mitigate the effect of such disasters, humanitarian relief organizations (HROs) across the world are busy rescuing and helping people in disaster-prone areas where the poor infrastructure often makes humanitarian logistics critical. Today, there is a strong demand for greater effectiveness and efficiency in humanitarian logistics operations as almost 60-80 per cent of the expenses incurred in humanitarian operations are due to logistics activities (Van Wassenhove, 2006).

To improve humanitarian logistics operations, performance measurement is a key step. Despite its significance, performance measures and measurement systems have not been widely developed and systematically implemented in the relief chain (Beamon and Balcik, 2008). Abidi et al. (2014), a recent literature review in this field, has observed that few performance frameworks were empirically tested. Another recent work confirmed this observation through case studies (D’Haene et al., 2015). In addition to the typical problems in the non-profit sector such as multiple objectives (O’Neill and Young, 1988), the inherently unique characteristics of the disaster relief environment (e.g. multiple stakeholders and the difficulty of data collection on the ground) make performance measurement of the relief chain even more challenging (Tomasini and Van Wassenhove, 2009).

Following the suggestion of recent humanitarian logistics literature such as Tatham and Spens (2011) and Abidi et al. (2014), we apply a process perspective to develop a performance measurement framework based on the widely used supply chain operations reference (SCOR) model. Noting the need for implementation in humanitarian operations (Tatham and Spens, 2011), we have worked with a large international HRO to examine and map its operation processes in detail, and proposed a set of key performance indicators (KPIs) with a bottom-up approach. After the initial work of identifying 26 KPIs, we then work with seven international HROs to validate the framework and metrics. It is found to be generic and applicable to many similar HROs at the organizational level to measure logistics performance.

Our work contributes to the humanitarian logistics literature by proposing a framework of KPIs for measuring logistics performance along the humanitarian supply chain at the organizational level. By a bottom-up approach and validation from multiple HROs, the framework has been tested on the ground with greater applicability compared to other similar frameworks in the literature. It may contribute to the improvement of relief operation efficiency and the better usage of limited funding to alleviate the suffering of the disaster victims. This paper is also one of a few works on the implementation of SCOR model to humanitarian logistics, a nascent and important research field (Abidi et al., 2014).

The rest of the paper is organized as such. Section 2 reviews the context of humanitarian logistics in the literature as well as the tools for performance measurement in the commercial and humanitarian worlds. After we discuss the research method in Section 3, Section 4 presents the reference framework and performance metrics. Section 5 then reports the KPI validation process as well as the results and recommendations, concluded by Section 6.

Humanitarian logistics is the process of planning, implementing, and controlling the efficient, cost-effective flow, and storage of goods and materials as well as related information, from source to consumption so as to meet the end beneficiary’s needs (Thomas and Mizushima, 2005). While humanitarian logistics shares similarities with its commercial counterparts in activities such as planning and transportation, they differ on demand pattern, objective, stakeholders, structure, complexity, and operating environment (Van Wassenhove, 2006; Ertem et al., 2010). These differences make performance measurement in humanitarian logistics even more challenging (Abidi et al., 2014; Schiffling and Piecyk, 2014).

Performance measurement is defined as an activity, that is, undertaken to attain predefined company goals (Clemens et al., 2004). It affects the strategic, tactical, and operational planning and control of the firm, and has a critical role in performance evaluation and guidance for the future actions of the whole entity (Gunasekaran et al., 2004).

At the supply chain level, Vahrenkamp and Siepermann (2005) defined performance metrics as a consolidated data set to measure the operations of the supply chain for a holistic view. The design and development of performance metrics is aimed at supporting management decisions and improving chain performance, and needs, therefore, to be aligned with supply chain strategies (Neely et al., 1995; Stadtler and Kilger, 2008).

There are many studies on supply chain performance measurement, using various approaches and targeting different objectives (Garcia et al., 2012). The balanced scorecard (BSC), as proposed by Kaplan and Norton (1992), is one of the most popular tools in measuring supply chain performance (Gopal and Thakkar, 2012). It affords users the flexibility of integrating multiple attributes and allows managers to assess the overall competitiveness of the supply chain easily (Lambert and Pohlen, 2001).

For a detailed view on supply chain performance, reference process models are valuable tools (Verdouw et al., 2010). A reference process model measures a specific ordering of activities across time and place, with clearly defined inputs and outputs (Davenport, 1993). The most widely used reference process model for supply chain processes is SCOR, developed in 1997 by the Supply Chain Council to evaluate the overall effectiveness of a supply chain (SCC, 2010). It provides a framework that links processes and metrics into an integrated structure (Huan et al., 2004). There are four levels of supply chain processes under SCOR. Level 1 is the strategic level for a firm to establish the competitive objectives. Level 2 is at the tactical level, including 26 core process categories that are generic components of a typical supply chain. Level 3 dives into the operational level with more granularity on the Level 2 processes, and Level 4 is industry-specific for a firm to customize the metrics for operational improvement. Under SCOR, at each level of the supply chain process, there are five generic performance attributes: reliability, responsiveness, agility, cost, and asset management.

Performance measurement is equally crucial in the humanitarian sector as the increasing impact of natural disasters is putting pressure on HROs to deliver more effective operations (Kaplan, 2001; Van Wassenhove, 2006). Developing the right performance measurements can assist HROs in identifying the bottlenecks in their logistics processes, enhancing preparedness, and emergency operations, alleviating the suffering of disaster victims, and managing donor funds more effectively in both relief and development activities (Beamon and Balcik, 2008).

Much of the early work on humanitarian performance measurement have focused on measuring outcome attributes (Buckmaster, 1999). The BSC approach is a common tool used in the humanitarian field because of its relative ease of use and the ability to balance various performance components (Schiffling and Piecyk, 2014). Moe et al. (2007) modified the BSC to evaluate the performance of natural disaster relief projects. Based on the practice of the International Federation of Red Cross and Red Crescent Societies, Schulz and Heigh (2009) proposed a performance framework using the BSC approach. Schiffling and Piecyk (2014) further developed a conceptual framework with the BSC to incorporate the demands of multiple stakeholders.

Noting the difference between humanitarian and commercial supply chains in objective and operation environments, Davidson (2006) proposed a framework with four performance indicators: appeal coverage, donation to delivery time, financial efficiency, and assessment accuracy. Similarly, Beamon and Balcik (2008) have proposed a performance measurement framework with three components: resource performance, output performance, and flexibility. However, all these works have not been empirically validated. More recently, Santarelli et al. (2015) developed a performance measurement system based on the literature and case studies, and empirically tested its KPIs in 11 HROs.

On a practical level, D’Haene et al. (2015) examined the implementation of performance management in the field, and found that most existing frameworks are not well received, but simple in-house developed tools are instead employed. Data gathering are also challenging for the HROs, and it is difficult to develop a system that can capture data in the same quality and format across multiple levels of an organization (Tatham and Hughes, 2011).

In summary, performance measurement in the humanitarian field is much less developed compared to the commercial world, and there is no universally accepted performance measurement framework for the HROs (Tatham and Hughes, 2011). Most existing measures are results-based and may not be very helpful to improve HRO relief operations, which face much uncertainty due to various factors beyond the control of one organization (Abidi et al., 2014). Thus, the literature such as Abidi et al. (2014) has called for a process-based performance measurement framework and KPIs that can address the above challenges. It can reduce possible bias from results-oriented metrics and help in the continuous improvement of supply chain operations. Blecken (2010) is one of a few attempts in the extant literature to apply supply chain process modelling to measure HRO performance, which used assessment, procurement, warehousing, and transport as key processes to develop the KPIs. However, Blecken’s (2010) model is limited to the strategic level without sufficient KPIs to measure performance attributes such as reliability and cost across multiple levels of the chain. This study is thus initiated to address the above issues by choosing SCOR model as the key framework, seeing it is a comprehensive supply chain process model covering multiple levels of supply chain operations with ready-made KPIs generated from its extensive application in the commercial world. While SCOR had been criticized for being too rigid in performance measurement in the field of humanitarian logistics (Davidson, 2006), it has been successfully implemented in large HROs such as the World Food Programme (Bölsche, 2012). The adoption of SCOR agrees with the suggestion of Tatham and Spens (2011), who propose SCOR as the first step for a unified reference source for the HROs.

Noting the lack of actual implementation of performance metrics in humanitarian supply chain (Abidi et al., 2014), we have sought the support of the HROs since the initiation of the study. Organization A was selected as the main project partner, which is a large secular non-governmental HRO in the field with over 90 years of experience in relief and development operations internationally. Eager to improve the effectiveness of its logistics operations, Organization A decided to join the study as the key stakeholder. During the initial stage of the study, Organization A was entering a process of internal consolidation among its country branches, many of which operated with full autonomy and without consistent policies and procedures. The global headquarters of Organization A thus expected the project to support its effort in streamlining the logistics operation processes across its country branches. Organization A shared its newly developed operation policies and procedures, and met the research team regularly with valuable feedback. In addition, it expected the project to benefit other HROs, and as such removed some peculiar internal processes from the study. The adoption of SCOR as the key framework was thus supported by Organization A, seeing its characteristics of genericity and flexibility from recent improvements.

The purpose of the project is to develop an implementable performance framework for HROs. Over the long-run, it should cover the whole scope of the supply chain to incorporate the view point of the “final customer”, beneficiaries in humanitarian operations (Tatham and Hughes, 2011). However, at the first stage, in considering the constraint of data availability in most HROs and the opinion of Organization A which expects the project to benefit mainly its internal logistics performance improvement, we limit the scope of the study at the organizational level, i.e., measuring the supply chain within the scope of one organization. Organization A (and other similar international HROs) is taken as the focal organization in our analysis, and we focus on the supply chain processes under its control, and examine the supply chain operations more at the macro- or country-level. It is also consistent to Organization A’s intention of branch consolidation. We thus exclude the delivery processes on the ground, or the last mile delivery to make the performance metrics manageable in practice. While this stage is critical in meeting the needs of beneficiaries, the last mile delivery is often done by the local community with limited support from large HROs and is very difficult to measure from their perspectives (Balcik et al., 2008). Indeed, it would be invaluable to include these processes for a more comprehensive framework, but at this stage, we exclude them to make the study more manageable and metrics more implementable for HROs.

The study is conducted in two phases. First, the original SCOR framework is modified to suit the need of humanitarian operation measurements. According to the generic processes in SCOR, we conduct a mapping of current supply chain operations in Organization A. As there are three generic levels in SCOR, the mapping of Organization A is conducted by three levels as well. Moreover, we have selected the relevant performance attributes from SCOR based on Organization A’s inputs as the second dimension of the framework. We then examine the humanitarian relevance of all SCOR metrics under these attributes. Suitable metrics are selected and adopted through a two-pronged process of elimination and modification. In the selection process, we combine the literature scan, industrial insights from commercial experience, and feedback from Organization A based on humanitarian practices. Existing KPIs used in HROs as reported by the literature are considered as well in this process. During this process, we focus on the high-impact metrics that can potentially generate tangible benefits for HROs. An initial set of KPIs is proposed at the end of this phase with details in the next section.

In the second phase, we validate the draft KPIs by surveying and interviewing the HROs. The validation seeks to understand the capability of the HRO’s current information system in capturing relevant data for the proposed metrics as well as the data integrity and quality. A questionnaire was sent out to the HROs from organizational and personal contacts as there is no database on the HROs in Asia. We have tried to contact all international HROs with a presence in Southeast Asia to elicit their feedback on the validity of the KPIs and the availability as well as the quality of relevant data in the field. In addition, we managed to contact some HROs through face-to-face interviews with key logistics personnel in the field for more detailed information. We surveyed 13 international HROs from our contacts, and received seven responses between May and June 2013, yielding a response rate of 54 per cent. In all, seven HROs joined the validation exercise, five through questionnaires and two through interviews. Among the seven participants, two are small HROs with specialties such as medical services, and the rest five are large with annual incomes of over US$100 million, among which one is an UN agency and four NGOs. Concerning the respondent demographics, three are global logistics coordinators, three regional coordinators, and one a country branch logistics specialist. All of them are senior logistics specialists with in-depth knowledge of the logistics operation within the organization. The validation of these HROs is thus a good representation of the current HROs, especially for the large international ones.

To develop a set of logistics performance indicators, we use both the performance attributes and logistics processes for the classification. There are five performance attributes in the original SCOR model: reliability, responsiveness, agility, cost, and asset management. SCOR defines the reliability as the performance of the supply chain in delivering correct products, at the correct time, in the correct condition and quantity, with the correct documentation, to the correct customer. Responsiveness refers to the speed at which a supply chain provides products to the customer, and agility is the ability of a supply chain to respond quickly to market changes.

In the context of humanitarian logistics, all attributes except for asset management are deemed important. Reliability or quality is a key requirement of any supply chain. Measuring the reliability of logistics processes is the first step to improving these processes. Responsiveness or timeliness is related to the response time of the supply chain, a key aspect in humanitarian operations. Cost is the chief measure of logistics financial performance, and the three attributes are widely used in the literature (Garcia et al., 2012). It is especially important nowadays when HROs face greater demand from the donors and the public on accountability and transparency (D’Haene et al., 2015). Agility or the chain’s response to demand surges is a critical aspect in emergency relief operations (Oloruntoba and Kovacs, 2015). Only asset management is less important as HROs are normally resource-light with few assets, and rely heavily on external resources and capabilities for emergency relief operations (Oloruntoba and Kovacs, 2015). Their assets are mainly vehicles and office space for daily operations. As most HROs are still weak in respect of detailed cost measurement, both Organization A and the research team decided to defer the asset-related metrics to future studies, and adopt reliability, responsiveness, agility, and cost as the key performance attributes for the study at the current stage.

Among the three-level processes, the original SCOR processes at Level 1 are plan, source, make, deliver, and return. In the context of humanitarian operations, the return process is less prominent as most HROs do not pay much attention to returned relief items (Peretti et al., 2015). While the environmental concern will become more important and so is the return process, we exclude this process at the current stage to make the project more implementable. As most HROs do not manufacture humanitarian goods except for some simple packing, we rename the process “make” as store to highlight the importance of the storing process in relief operations. Storing takes much more resource compared to packing as HROs often store relief items for months or even years but only do the packing in a few days. The processes under Level 1 of the framework are, therefore, plan, source, store, and deliver. Plan refers to the planning activities in identifying and meeting demand. Source refers to the activities to source and procure goods and services to meet demand from the ground. Store refers to the activities to receive and inspect goods as well as to store and dispatch the stored goods. Delivery refers to the activities related to the management of the received goods as well as the delivery of goods from the HRO.

At Level 2, the processes plan and source are further decomposed into several sub-processes, P1 (plan source), P2 (plan store), and P3 (plan delivery); S1 (source for goods intended for storage), and S2 (source for goods intended for immediate distribution). Based on the importance of these processes, we adopt related SCOR metrics to measure the efficiency of the critical processes. At Level 3, similarly, we map the supply chain of Organization A with the details omitted here.

For each of the four performance attributes, i.e., (A) reliability, (B) responsiveness, (C) agility, and (D) cost, we select and adopt related SCOR metrics in a hierarchy of three levels according to logistics processes of Organization A. They are presented as follows:

(A) Reliability: reliability is the basic requirement of any operation, and it is measured from Levels 1 to 3 by the following eight indicators:

  A1.1. Perfect order fulfilment: it shows the overall reliability of the relief supply chain by measuring the percentage of orders that are fully consistent to requirements with complete and accurate documentation. It is further measured by four metrics below.

  A2.1. Percentage of orders delivered in full: it is the first KPI at Level 2 with more granularity on the logistics process. We can view it as one component of A1.1 with better insights on the delivery process at Level 2. A2.1 is defined as the percentage of orders which meet customer requirements on quantity and accuracy. It measures the quantity and quality of the delivery, i.e., the performance in delivering the right item and amount.

  A2.2. Delivery performance to customer commit date: it measures the timeliness, another important aspect of delivery performance, defined as the percentage of orders that are fulfilled on the customer’s originally scheduled or committed date. The focus of the indicator is in delivering at the right time.

  A2.3. Documentation accuracy: it is defined as the percentage of orders with accurate documentation. It is one important metric in humanitarian operation as goods often pass multiple hands in logistics operations and are prone to errors due to the urgency and partners’ inexperience in relief operations. The order documentation is considered accurate if the documents related to shipping, payment, compliance, and other mandatory requirements are all received in full, correct, and readily available upon request.

  A2.4. Perfect condition percentage: defined as the percentage of orders delivered in an undamaged state that meets specifications, have the correct configuration, are faultlessly installed and accepted by the customer. The four Level 2 metrics A2.1 to A2.4 reflect four different aspects of the Level 1 metric A1.1.

  A3.1. Store documentation accuracy: It is the first Level 3 metric. Noting the complexity of humanitarian supply chains, we only adopt important Level 3 KPIs in SCOR model which have the potential to deliver the tangible benefits to the HRO operation at the organizational level based on Organization A’s feedback. A3.1 is defined as the percentage of orders with accurate documentation supporting the order in the storage process. It is a detailed measure of metric A2.3 and can help to identify the sources of performance deficiency if the metric is below expectation.

  A3.2. Delivery documentation accuracy: it is defined as the portion of orders with accurate documentation supporting the order in the delivery process. Combined with metric A3.1, the two metrics provide a detailed examination on the Level 2 metric A2.3.

  A3.3. Risk mitigation plan: different from the first two metrics, A3.3 is not directly related to the other metrics, but it is a high-impact one which may significantly benefit a HRO’s operation if implemented properly. It reflects the preparedness of an HRO on supply risks, measured by the percentage of critically sourced items with alternative or additional suppliers.

(B) Responsiveness: responsiveness is the second key supply chain attribute, and we adopt the following SCOR metrics on responsiveness for the three levels:

  B1.1. Order fulfilment cycle time: it shows the overall responsiveness of the supply chain, defined as the average actual cycle time consistently achieved to fulfil customer orders. For each individual order, cycle time starts from the order receipt and ends with customer acceptance of the order. It is measured by three Level 2 metrics below.

  B2.1. Sourcing cycle time: defined as the average time associated with sourcing processes. It gives an overall measurement on the sourcing process, starting from identifying the sources of supply and ending with the payment to the supplier.

  B2.2. Assembling cycle time: defined as the average processing time between commencement of upstream processing and completion of all assembling processes including packaging and labelling operations. In the humanitarian context, it mainly refers to the packaging and labelling operations for humanitarian kits during the process of store though the other types of packaging are included also.

  B2.3. Delivery fulfilment cycle time: defined as the average time associated with the delivery processes. It gives an overall measurement on the delivery process. Metrics B2.1- B2.3 collectively provide a comprehensive picture of the Level 1 metric B1.1.

  B3.1. In-stock percentage: defined as the percentage of relief items which are in-stock when needed. The two Level 3 metrics on responsiveness selected are not related to the other metrics, but they measure some high-impact aspects of the logistics operation and may improve an HRO’s performance significantly. If an HRO can maintain a high in-stock percentage, it can meet relief needs much more effectively.

  B3.2. External event response: defined as the mean response time in days to an external risk event from the start of the event, including any detection lags. This metric is related to humanitarian operations as HROs are exposed to various risk events and have to respond quickly.

(C) Agility: agility is also critical in humanitarian operations as HROs face much demand uncertainty and need to adjust their operations rapidly to the needs on the ground. We adopt the following SCOR metrics on agility for the three levels:

  C1.1. Upside supply chain flexibility: it is defined as the number of days required to achieve an unplanned sustainable 100 per cent increase in quantities delivered. The 100 per cent is a general benchmark for humanitarian logistics while the original SCOR sets 20 per cent as the benchmark for the commercial world. It is further measured by two Level 2 metrics below.

  C2.1. Upside source flexibility: defined as the number of days required to achieve an unplanned sustainable 100 per cent increase in the quantity supplied. It measures chain agility of the sourcing process.

  C2.2. Upside delivery flexibility: defined as the number of days required to achieve an unplanned sustainable 100 per cent increase in quantity delivered with the assumption of no other constraints. It measures chain agility of the delivery process.

  C3.1. Current on-hand inventory: defined as all current on-hand inventories, including safety stock required to sustain current order fulfilment. An HRO has to know its actual on-hand inventory as well as the planned on-hand inventory to implement this KPI. It is not related to the other metrics, but may improve HRO’s performance significantly.

  C3.2. Current purchase order cycle time: defined as the time of place a purchase order plus the supplier’s lead time. It gives an overall cycle time from purchase request to procurement, and ended at goods receipt.

(D) Cost: cost is a basic element in any supply chain operations and is important in humanitarian operations as well. We adopt the following SCOR metrics on cost for the three levels:

  D1.1. Supply chain management cost: it is defined as costs associated with operating the supply chain. This metric reflects the overall cost of the supply chain and is further measured by four Level 2 metrics indicated below.

  D2.1. Cost to plan: defined as the cost to manage the plan process.

  D2.2. Cost to source: defined as the cost to manage the sourcing process.

  D2.3. Cost to manage product inventory: defined the total cost to manage inventory, covering all costs at the stage of storage such as warehousing operating, rental, and labour cost. While HROs tend to hold more inventories for agility and responsiveness, they have to balance the cost associated with inventory keeping with this metric.

  D2.4. Cost to delivery: defined as the cost to manage the delivery process. Metrics D2.1- D2.4 detail the supply chain cost for the different processes.

  D2.5. Supply chain risk mitigation cost: defined as the cost associated with activities that are planned to mitigate supply chain risk. It is different from the first five metrics but is also important as the humanitarian supply chain operation is full of risk and the HROs have to balance the need between cost reduction and risk control.

  D3.1. Cost to manage supply chain performance: defined as the total cost associated with assessing supply chain performance, including cost associated to the monitoring of both internal and external players such as suppliers and shippers. While measuring supply chain performance can improve the chain performance, an HRO needs to know its cost and balance the various needs.

Most details on these metrics are presented in Table I, where the original SCOR number, related performance attributes, and processes as well as the calculating formula are displayed.

After proposing the 26 KPIs based on the study on Organization A, we then conduct the KPI validation with the participation of seven HROs. Knowing the difficulty of measuring humanitarian operations on the ground, we focus on two aspects in the validation exercise, importance, and difficulty. The importance of a KPI refers to the relevance of the KPI in humanitarian operations, and the degree of potential improvement by using the metric in the operation. The difficulty of a KPI refers to the implementation difficulty of measuring it from the ground operation based on current processes and existing data. We use a coupled five-point Likert scale ((1, 1)=(not important at all, not difficult at all) and (5, 5)=(very important, very difficult to implement)) to present our survey results. A metric scoring of more than 3 in importance means it should be implemented. Similarly, a metric scoring of more than 3 in difficulty indicates that it would experience significant difficulty in implementation.

In general, we can classify all KPIs into four quadrants of a 2×2 matrix of importance vs difficulty as displayed in Figure 1. Clearly, the proper strategy for KPIs in Region I is implementation, and the right strategy for KPIs in Region IV would be no implementation. The strategies for KPIs in Regions II and III are a bit tricky. For KPIs that are important but difficult to measure (Region II), the HRO may need to explore other means to measure them such as adjusting its current processes to facilitate related data collection, and the proper strategy would be future implementation after a significant system change For the KPIs in Region III, the strategy is unclear. The HRO can either remove them or choose some of them if there is still some value from the adoption. It may need to examine these KPIs individually to decide whether to implement them or not.

Applying the generic strategy of Figure 1, we examine implantation strategies for the draft KPIs from the survey. The KPIs are put into only two quadrants as all of them are valued as important. They are in either Region I (five KPIs) or Region II (21 KPIs), and the initial strategic choices we have are only two, implementing now or later.

To gain more strategic choices and flexibility, we further differentiate KPIs by two more criteria: by importance from very important (mean importance of 4 or higher) to somewhat important (mean importance of between 3 and 4), and by implementation difficulty from very difficult (mean difficulty of 4 or higher) to somewhat difficult (mean difficulty of between 3 and 4). We categorise the 26 KPIs into four groups as shown in Table II.

The first group (upper-left in Table II) consists of KPIs that are very important and easy to measure (implementation difficulty below 3), which can be implemented quickly without much difficulty. It includes two KPIs on reliability (“percentage of orders delivered in full” and “delivery performance to customer commit date”), one KPI on responsiveness (“source cycle time”), and two KPIs on agility (“current on-hand inventory” and “current purchase order cycle times”). As the first two KPIs on reliability could be difficult to measure for last mile delivery, their scopes are limited to the delivery of supplies. The scopes of “source cycle time” and “current purchase order cycle time” are similarly limited.

The nine KPIs in the next group (upper-middle) are very important but are somewhat difficult to measure (mean difficulty of between 3 and 4). This group includes five KPIs on reliability (three KPIs on documentation and KPI “perfect condition percentage” as well as overview KPI “perfect order fulfilment”), three KPIs on responsiveness (“order fulfilment cycle time”, “in-stock percentage”, and “external event response”), and one KPI on cost (“cost to manage product inventory”). The implementation of these KPIs is possible based on the existing reporting systems in the HROs with some modification. We thus investigate each implementation issue. For the five KPIs on reliability, the difficulty in measuring the three KPIs on documentation is the lack of records on the ground, and the other two KPIs face a similar problem in implementation. For the first two KPIs on responsiveness (B1.1 and B3.1), the measuring difficulty lies in the order data checking and tracking, but the difficulty for the other KPI (B3.2) is relevant to the demand data estimation.

The third group (upper-right) consists of very important KPIs but very difficult to measure. Five KPIs in the group are related to cost. It shows that the current accounting system in the HROs does not fit the typical commercial costing for supply chain operations. It is important for HROs to develop more relevant measures on these KPIs though it may be time-consuming.

The KPIs in the last group (lower) comprise the less important KPIs but are also difficult to measure. The implementation of these KPIs is not a priority.

Based on our scrutiny on the implementation issues of the most important KPIs, we would suggest the HROs to install some automated systems for better documentation accuracy. This system could record the sending out of order requests as well as the delivery of supplies, warehouse check-in and out. With support from the suppliers and partners, system automation may help them to easily measure the three KPIs on documentation. Similarly, the other KPIs in the second group (upper-middle) such as “perfect condition percentage”, “order fulfilment cycle times”, and “in-stock percentage” would also be suitable for measurement. Moreover, some KPIs belonging to the fourth group (lower) such as assembly cycle time and delivery cycle time could then be readily measured as well. Such a system would increase supply chain visibility as well and support the efficiency and effectiveness improvement of humanitarian operations. After installing the automation system, the classification of the 26 KPIs is expected to be shown as Table III, with 14 KPIs implementable compared to the original 5 KPIs.

Performance measurement in supply chain management is established in the commercial world (Gopal and Thakkar, 2012), but its usage still lags behind in humanitarian operations. Most HROs recognize the importance of having a suitable performance measurement system and the benefits of its implementation to their organizations (D’Haene et al., 2015). SCOR, being a reference process model that links business processes, metrics, best practices, and technology with wide usage, has great potential in measuring the logistics performance of the HROs (Tatham and Spens, 2011; Abidi et al., 2014).

With a bottom-up approach, we have worked with several HROs to apply the SCOR framework to the humanitarian supply chain at the organizational level. A set of 26 KPIs has been identified for the HROs to measure and monitor their supply chain performance. These metrics cover the key elements of quality, time, and cost in humanitarian supply chains, and can assist the HROs to measure their performance on agility, responsiveness, reliability, and cost effectiveness. By implementing these metrics, an HRO is able to benchmark both internally against its previous performance and externally with the other HROs, and set realistic targets for ongoing improvement. The implementation of multiple indicators may also help the HRO to review its current operation policies and processes for potential improvement. For example, it is noted that the current policies and processes in Organization A tend to be overly cost-oriented. In the case of its branch in Indonesia, all purchases above US$500 at the country level requires at least three quotations, and the exception limits to within five days after an emergency. While these thresholds may be required by the donors, it would be instructive to examine the impact of these policies on the responsiveness and agility of the organization during emergency operations. A review of its KPIs may lead to better policies and processes that balance the performance in cost efficiency and operation responsiveness. For instance, Organization A can propose to extend the quotation waiver period from five days to ten days should the second five days see many orders being placed and current policies hinder supply chain agility.

The validation of the KPIs supports our initial work with all the metrics deemed important. It also highlights the implementation difficulty as only five indicators out of the original 26 can be readily implemented in the HROs based on their current procedures and practices. In view of the operating practices of most HROs, we further suggest the automation of some key logistics processes, including goods ordering, receiving, and delivery in their logistics operations. It could improve documentation accuracy, reduce the workload of the logistics staff, and facilitate the implementation of nine more KPIs on supply chain reliability and responsiveness in humanitarian practices. Nevertheless, there are still 12 KPIs including those on supply chain cost which are difficult to implement despite automation. As such, HROs need to improve their supply chain management by integrating their internal procedures across the other functions of finance, procurement, service contracting, and distribution activities. Such an integration attempt may lead to a more effective measurement of their logistics performance, including supply chain cost estimation.

Our work contributes to performance measurement in humanitarian logistics with a generic metrics set cased on a supply chain process model. Different from many existing metrics in the literature, this work is based on the detailed feedback of one large HRO and validated by several other HROs. It may then be more applicable on the ground with tangible benefits to HROs. However, there are also limitations in this study such as the small sample size of the validation and the exclusion of some important processes like the last mile delivery. Future works can extend the scope of the framework to a full supply chain, including the last mile delivery, which is the critical link to meet the needs of the beneficiaries. Other processes such as the return and asset attributes can be added to improve the metrics as well.

Moreover, we can look at implementing a logistics management system to automate the supply chain processes with real-time dashboards for better supply chain visibility. The platform and the application requirements of an automated system need to be carefully examined as the operating environment of the HROs is very different from the commercial sector, without the support of advanced physical and IT infrastructure. Insights from the implementation practice would be invaluable to improve the work. The KPIs can be validated and improved for a broader community of international and regional HROs, leading to a more comprehensive set of performance measurement metrics.

Qing Lu is an Assistant Professor in the Department of Logistics Management, Izmir University of Economics (IUE). His research interests include supply chain strategy, logistics outsourcing, supply chain security and governance, humanitarian logistics, and supply chain sustainability. Before joined IUE, he worked as a Research Fellow in supply chain management at The Logistics Institute – Asia Pacific (TLIAP), the collaboration between the National University of Singapore (NUS) and the Georgia Institute of Technology for seven years. He obtained PhD in business strategy from the NUS Business School in 2006. Qing Lu is the corresponding author and can be contacted at: lu.qing@ieu.edu.tr

Mark Goh is the Director for Industry Research at TLIAP. He is also a Faculty at the NUS Business School and the University of South Australia. He is currently on the Editorial Boards of the Journal of Supply Chain Management, Q3 Quarterly, Journal for Inventory Research, and Advances in Management Research, and has served as an Associate Editor for the Asia Pacific Journal of Operational Research. His current research interests focus on supply chain strategy, performance measurement, buyer-seller relationships, and reverse logistics. Professor Mark Goh holds a PhD from the University of Adelaide.

Robert De Souza is the Executive Director of the TLIAP. He is a Professor at the Georgia Institute of Technology in USA and a Senior Fellow at the National University of Singapore. He has published extensively and is a sought after Speaker and Consultant. He is a Chartered Engineer and serves on multiple industry, government, and academic committees. He received his PhD, MSc, and BSc Honours in the UK.

The authors express their gratitude and appreciation to all organizations and individuals supporting this study as well as the financial support from the UPS Foundation. In particular, the humanitarian organizations participated the project, and the research assistance provided by Hafidzaturrafeah Binte Othman, Lau Ying Bin, Lee Min Hua, Sylvia Widjaja; Vincent Teoh, Yeo Kuei Peng, and Keterina Chong.