Improving configurator development: a coordinated performance assessment approach Open Access

As technology advances, customers increasingly expect products and services tailored to their specific needs. The requirement of mass personalization demands tools that let users customize products and services effectively and efficiently, such as configuration systems (or configurators). A configurator is a type of expert system for automating product and service specification processes by guiding users to make customized choices among valid combinations (Haug et al., 2019b). Employing configurators brings various benefits, including improved internal business processes, reduced lead times, on-time delivery, improved quality, reduced resource consumption and optimized products (Kristjansdottir et al., 2018).

Many companies benefit from configurators, but many also encounter significant challenges in realizing these benefits, which sometimes leads to abandoning projects after their initiation (Haug et al., 2012, 2019a; Ladeby and Oddsson, 2011; Walcher and Werger, 2011). Common causes of configurator development failure include dissatisfied decision-makers and stakeholders, lack of user acceptance, maintenance difficulties and cost overruns (Haug et al., 2019a). One major indirect cause of configurator development failure—the dissatisfaction of decision-makers and stakeholders—often results from misalignment between organizational and configurator development goals at various stages of configurator development.

The existing literature offers several approaches to configurator development and implementation (e.g. Forza et al., 2006; Haug et al., 2012; Hvam, 2006; Hvam et al., 2008), but such approaches can bring challenges. The first observed challenge is a misalignment between the configurator development process and project objectives. Deviations from project goals have been discovered at various stages of failed configurator projects (Haug et al., 2019a), which reflects a lack of attention to the development of performance measures in accordance with configurator project goals in the existing approaches. Specifically, the existing literature advises choosing performance measures from a common list as opposed to providing a more systematic approach to developing and using such configurator performance metrics. Consequently, configurator developers are left to choose relevant performance measures from a common list of indicators, such as the number of errors, use of labor hours, processing time, accuracy and completeness (Haug et al., 2019b), and the performance measurement structure is rarely discussed holistically. Furthermore, scant attention has been paid to leading indicators and process compliance measures in configurator projects (e.g. Myrodia et al., 2017; Trentin et al., 2012; Zhang and Shafiee, 2022), implying that performance results can be too generic and do not provide sufficient knowledge of performance drivers.

Another observed challenge is the risk of neglecting opportunities to identify and timely eliminate deviations in development that could lead to project failure, which is caused by overlooking the use of performance measures during configurator development. Rather than continually extending performance measures, existing approaches to configurator development advise that the performance of relevant processes should be measured only before and after a configurator is implemented (Hvam et al., 2008). This is problematic due to the practical complexity of predicting all the performance measures at the beginning of a configurator project that will later become relevant. Furthermore, studies based on such approaches tend to emphasize evaluating a configurator’s implications only after it is in operation, and comprehensive development of performance assessment is typically not done in the early phases of a configurator project (e.g. Campo Gay and Hvam, 2022; Schobel et al., 2018; Wang et al., 2019), suggesting that the collection of configurator performance data is not aligned with performance needs. Therefore, insufficient data are available for implementing performance metrics at a later stage, and resources are wasted on collected data that have little use.

The existing literature does not provide a coordinated approach to the development of product configurators and their performance measures, a shortcoming that increases the likelihood of biased choices of performance metrics (Tiihonen et al., 2013) and may lead to misalignment between objectives and evaluation results. Some studies suggest that selecting and applying appropriate performance measures at certain times in a configurator project may play a central role in ensuring its success (Haug et al., 2019a; Hvam et al., 2008; Kristjansdottir et al., 2018). To address the gaps in the literature described above, this paper describes an approach to the coordinated development of product configurators and their performance measures to increase the success of such projects. The research addressed the following research question:

How can the development and application of performance measurements be better integrated into configurator development?

To answer this question, this paper first reviews the relevant literature to identify the metrics used for evaluating configurators, focusing on the coverage of performance aspects and early-stage performance considerations. On this foundation, a coordinated approach is described for integrating performance assessment into configurator projects to better realize configurator development goals and foster continuous improvement. The approach was tested in a longitudinal case study using action research. Finally, the paper offers a discussion of the findings and concludes by addressing the research question.

The use of configurators is reported across various industries. This literature review identifies existing performance metrics for configurator projects by focusing on the alignment between performance metrics and configurator performance objectives. Specifically, it investigates the performance measures of configuration systems in terms of the diverse perspectives on performance indicators, sources and the early-stage planning of data collection for performance evaluation, as well as considering whether indirect evaluation measures have been used instead of more suitable measures due to limitations in data availability.

A search of the Web of Science and Scopus databases was conducted to identify relevant English publications from the past 15 years using the following search strings for titles, abstracts and keywords: “configurator AND (product OR sales),” “configurator AND (performance OR impact) AND (product OR service OR operation),” and “configurator AND develop* AND performance AND (product OR service).” Table 1 summarizes the papers identified in these searches (i.e. those that evaluate the performance of configurators). As shown in the table, most of these studies focus on product and service configurators in the engineering design, construction and production fields. The quantitative data used for performance evaluation in these papers were collected mainly from questionnaires, experiments, and the empirical data of case companies, whereas interviews were the most popular method of qualitative data collection.

Sections 2.1.1 through 2.1.4 summarize the performance measures applied in the reviewed studies, which were organized into four main perspectives—effectiveness, efficiency, process compliance and overall measures—to assemble a comprehensive performance structure for operational services (Ge et al., 2023). Table 2 provides an overview of the summarized performance measures.

Performance indicators, taken from the perspective of effectiveness, identify the extent to which a configurator meets its design objectives. These indicators can be categorized as specification output or product/service quality measures. Specification output indicates the thoroughness of the information output generated by a configurator, which is crucial for fulfilling the features and requirements of deliverables. Mueller et al. (2022) provide a qualitative comparison of the resulting content between existing and configurator-supported commissioning specification processes in an engineering-to-order company. Tiihonen et al. (2013) characterize the attributes of a series of product sales configurators to examine modeling variety and demonstrate the applicability of the presented configurator. Gerth et al. (2016) use the number of deliverables, including blueprints and documents, to quantitatively compare the detail levels of specifications in construction engineering.

Product/service quality, a more common measure in the literature, identifies the effectiveness of a configurator through the quality of its deliverables. A typical way to quantify the quality of deliverables is to measure the number of errors, which can be done by counting returns on production lines (Kristjansdottir et al., 2018) or evaluating deviations in the results of the configurator (Schobel et al., 2018; Wang et al., 2019). The quality of deliverables can also be compared in the form of variations in a final product through qualitative analysis at a higher level (Gerth et al., 2016).

The efficiency perspective focuses on the resources required to support a configurator. A majority of the reviewed studies report performance measures on the productivity of configurators in terms of time and/or cost. Time-based performance measures have been proposed under various scopes, including runtime to complete each task, runtime for an entire configuration process and total lead time of product or service delivery (Bredahl Rasmussen et al., 2021; Gerth et al., 2016; Kristjansdottir et al., 2018; Mueller et al., 2022; Schobel et al., 2018; Tiihonen et al., 2013). Time-based performance indicators can be used to reflect human resource consumption, typically quantified as labor hours (Campo Gay and Hvam, 2022; Kristjansdottir et al., 2018; Mueller et al., 2022). Other common cost factors include configurator development, user training, operations, maintenance and software licensing (Bredahl Rasmussen et al., 2021; Kristjansdottir et al., 2018; Shafiee et al., 2019, 2021).

Another approach to measuring efficiency is evaluating the parameter complexity of both the knowledge input and the operation input that a configurator requires. Knowledge input includes all necessary information that must be gathered before the start of a configuration process. A simple way to compare knowledge input complexity is to compare the document types required for a configuration (Mueller et al., 2022). Shafiee et al. (2017) propose a complexity classification scheme that can be used to quantify configurator parameter complexity via input attributes and constraints. The method was applied in a longitudinal case study to examine the development of integrated sales and technical configurators (Zhang and Shafiee, 2022).

Operation input represents human interactions with a configurator throughout a configuration process. Schobel et al. (2018) evaluate a service configurator that supported the development of mobile data-collection instruments. The authors quantify the operation input by registering and counting the number of operations required to complete each configuration task. Considering that human error is inevitable and is likely to increase as the complexity of interaction increases, clear and simple operation input is preferred for configurators.

This paper uses the term process compliance to describe the degree to which the execution of a configuration process follows designated guidelines and work procedures. This term is distinct from result compliance, which assesses the outcome of a configurator and is therefore categorized from the perspective of effectiveness. In this context, a major implication of introducing a configurator is the need to align specification processes and, consequently, reduce elements that do not add value.

While the reviewed studies focus closely on monitoring the resulting quality improvement and complexity reduction from the perspectives of effectiveness and efficiency, they rarely mention the impact of configuration process changes (see, e.g. Bredahl Rasmussen et al., 2021). Thus, this paper adopts a process compliance perspective to investigate the implications of standardization on the configuration process.

Some studies in our literature review focus on two important aspects of a configurator: profitability and user acceptance. Profitability embraces both effectiveness and efficiency from a financial viewpoint. Return on investment (ROI) is commonly used to measure profitability and is typically estimated or calculated as a cost-benefit ratio over a 3–5 year period to increase reliability (Haug et al., 2019b; Kristjansdottir et al., 2018; Shafiee et al., 2021). Compared to productivity from an efficiency perspective, ROI also includes product quality and its associated benefits in terms of cost reduction and sales increases (Shafiee et al., 2021). User acceptance measures the capabilities and willingness of end users to utilize a configurator, taking subjective human perception into account. Capabilities and willingness are commonly measured by use count and user satisfaction surveys, respectively (Freitag et al., 2011; Shafiee et al., 2019; Trentin et al., 2012).

Freitag et al. (2011) include both potential- and client-related indicators in the performance measurement of service configurations. The authors argue that the potential-related performance of a service provider’s resource availability is also relevant, indicating the provider’s ability and willingness to provide a service, and that client-related performance, typically interpreted as customer satisfaction, reflects the influence of external factors on service productivity.

The reviewed studies consider either one or several perspectives to measure the performance of a configurator but fail to describe a comprehensive performance structure. Instead, they widely favor selecting from common performance indicators that measure quality and cost, as doing so provides a direct representation of the resulting performance of configurators. Without an agreed-upon approach to selecting or deriving performance measures, however, the unguided selection of performance measures is subject to the risk of bias (Tiihonen et al., 2013). Consequently, the link between a configurator’s objectives and evaluation outcomes becomes weak. For example, if a configurator project includes only measures related to reductions in time and resource use, it may achieve such aims at the expense of specification quality, which can result in costs that offset such benefits.

Also, many of the reviewed performance measures served only the purpose of evaluating the implications of configurators after they are in operation, giving inadequate attention to leading indicators and process compliance measures. Performance results become highly generic and do not always provide sufficient knowledge of the performance drivers in configuration processes. Furthermore, while the incorporation of configurator development costs during the scoping phase is acknowledged (Mueller et al., 2022), the consideration of planning for performance evaluation in the early stages of configurator development is absent. As a result, the design of configurator data collection is not aligned with performance needs. On the one hand, such misalignment limits data availability in implementing performance metrics at a later stage; on the other, data collected without a purpose adds unnecessarily to management expenses. Therefore, it is necessary to develop a performance measurement approach aligned with configurator development at an early stage.

The literature review shows that the existing practices of performance assessment in configurator development typically involve selecting performance indicators measured before and after a configurator’s implementation. The process of selecting the chosen performance indicators is, however, rarely clear. At the same time, the use of performance results focuses dominantly on evaluating configurator success after project completion, whereas the use of performance metrics to ensure project success during development is widely overlooked, yet failures are reported in configurator development projects in connection with not achieving their intended objectives as a major cause (Haug et al., 2019a). The reviewed literature reveals a research gap in preventing and identifying such deviations earlier in the stages of configurator development.

To address this research gap, the present paper proposes an approach in which a configurator and performance measures are developed simultaneously, and the measures are applied to support continuous improvement throughout a project. This idea is formalized as a coordinated performance measurement approach to develop the configurator and its performance measures, which includes four stages—scope; develop; implement and measure; and review and refine—as shown in Figure 1. Each stage integrates considerations from both the configurator development and performance assessment perspectives. The early-stage co-scoping of configurator development and performance assessment ensures that performance measures are identified in line with configurator objectives and can support a holistic performance view. The data, resources and ownership of the identified performance measures are then secured by integrating performance resource planning into configurator development cycles. The collection of performance data is coordinated following the launch of the configurator. By reviewing the performance results, targeted insights can be obtained to further support the refinement of the previous stages. The four stages are introduced in Sections 3.1 through 3.4.

The scoping stage focuses on aligning the configurator development scope and performance evaluation scope so that the performance metrics are developed in line with the configurator’s objectives and can be used to guide configurator development even at an early stage. It is important to define a configurator’s scope in configurator projects, given their requirements and goals. Several researchers discuss the scoping of configurator projects (Haug et al., 2019b; Mueller et al., 2022; Shafiee et al., 2014, 2018), which involves identifying and defining a project’s objectives and stakeholder requirements. The outcome of configurator scoping is operational objectives derived from a vision and explicit stakeholder requirements.

The defined operational objectives and stakeholder requirements will be evaluated after the configurator is launched. Such an evaluation must be supported by a set of performance measures that further establish requirements for the availability of corresponding data and resources. To ensure data and resource availability, the need for performance measurements must be specified and addressed within the configurator design. Therefore, the performance evaluation of the configurator should be carefully considered at an early stage, highlighting the importance of identifying the scope of configurator performance evaluation before configurator development. As a result, the initial scoping of the performance assessment must be arranged in accordance with the scoping of the configurator.

The success and usefulness of a process can be evaluated based on its effectiveness, efficiency and process compliance (Ge et al., 2023), and this applies to configuration systems. These factors are related to the operational objectives established during the configurator scoping stage, which include identifying the business processes to be supported. The impact of the configurator should be measured primarily within the scope of these processes.

Direct impacts, such as specification details and the number of configuration attributes, are influenced by the configurator’s design. These direct performance measures need to be specified at the development stage, as the definition of these measures relates directly to the configurator design. They must be specified in specific configurator development steps, which are identified in the scoping stage. However, processes outside the scope may also be indirectly affected (e.g. the indirect effect of reducing lead time). Reducing lead time can indirectly improve processes beyond the scope by exposing underlying problems, such as unclear instructions or a lack of documentation, resulting in improved specifications (Hvam et al., 2008).

An indirect impact, such as lead time, can be measured independently and does not concern a configurator’s development stage. Therefore, the data and resource plans for these measures can be specified before the configurator’s development. Figure 2 provides an example of the direct and indirect impacts of a configurator on business processes.

Relevant performance measures are identified in accordance with the configurator’s defined objectives and requirements. These performance measures fall into three types: measures that can be directly adapted from existing performance metrics, measures that can be derived from existing data and measures that require the collection of new data. Organizations may have existing performance metrics that are also applicable to measuring the configurator’s performance, such as lead time for making quotes, lead times in order fulfillment or the quality of bills of materials for manufacturing. These existing performance measures are identified and adapted during the scoping stage. Some performance measures may not exist but can be derived by organizing and analyzing existing data. Other performance measures may require new performance data, such as hours consumed in making drawings for production or lead time from ordering to finished manufacturing specifications.

An overview of performance measures helps to identify the configurator’s main functionalities and the critical goals it will achieve. Furthermore, the need for data collection and other performance resources is already clarified in the scoping stage, so they can be better incorporated into configurator development. Ultimately, the outcome of the scoping stage provides a clear understanding of performance aspects to be evaluated to ensure the configurator’s success.

The direct performance measures determined in the scoping stage are of a generic nature and must be specified and translated as concrete performance indicators. Such translations are highly case-specific and depend on various conditions, such as data collection potential, depth of analysis and available resources. The definition of direct performance measures also depends on a configurator’s design. Taking these into consideration, the development of performance metrics should be integrated with the development of the configurator to guarantee consistency between performance objectives and indicators. The development stage integrates performance measures and resource planning into configurator development, yielding mutual benefits for both sides.

From the configurator development perspective, a configurator should be designed, modeled and developed with a focus on the most important functionalities. To prioritize the relevant configurator design considerations, well-defined performance metrics must be in place. The performance metrics highlight a concrete configurator development focus relevant to performance goals, such as key functionalities to be incorporated into the configurator, high value–adding product variants to be included and critical operations to be achieved within limits (e.g. achieving cost estimation within a 5% margin of error). These performance metrics provide guidance for focusing on configurator development goals to achieve performance targets, avoid falling into detail traps and reduce unneeded configurator complexity.

From the performance management perspective, the new performance indicators’ detailed data and resource needs are specified and linked to the performance measures identified in the scoping stage. The scope of an evaluation is also subject to project constraints, which could include time, cost, resources and technical limitations. Many aspects can be evaluated, and it is important to select the most relevant ones carefully. The importance and development effort given to each performance aspect should be balanced to allow efficient, realistic planning of performance evaluation.

By integrating performance resource planning into configurator development, performance data needs can be considered a design requirement of the configurator so that data collection can be secured when the configurator is deployed. Meanwhile, the existing measures identified in the scoping stage can be used as benchmarking references during configurator development.

The third stage focuses on the implementation and use of the configurator as well as on collecting performance data and measuring performance. Configurator development activities at this stage include installing, testing and deploying the configurator. The performance management actions embrace data collection, data processing and calculation of performance results. To ensure accurate, effective data collection, the data and resource plans should be followed to collect the correct data promptly using appropriate resources. Data processing involves transforming and cleaning data to prepare them for calculation using designated metrics. The outcome of a calculation should comprise a list of performance results as defined in the previous stage.

The implement and measure stage calculates the configurator’s impact in operation. The performance of the configurator installation can be determined by user-acceptance measures, including the number of users who complete training courses and the correct use rate of the configurator. The performance of configurator testing and deployment can be measured in various aspects, including profitability, productivity, output quality and process standardization. The degree of user support (e.g. frequency of communication with users, handling of user requests and hours consumed in user support) also impacts configurator performance and should, therefore, be continuously monitored at this stage and adjusted as needed.

These activities are carried out in a coordinated manner as planned in the development stage, ensuring that the information conveyed by performance results suffices to support configurator implementation decisions in the next stage.

The final stage (i.e. review and refine) provides valuable performance insights and supports the configurator’s refinement. The performance results obtained in the previous stage are reviewed and compared to the performance targets to examine whether and to what extent the configurator development objectives have been achieved. These performance results can guide the configurator’s improvement via further refinement.

The example in Figure 3 summarizes when performance categories can be specified, measured and used for continuous improvement in relation to the cycles of configurator development. The present research adopted the structured configurator development procedure suggested by Hvam et al. (2002) and Hvam and Ladeby (2007) to represent each configurator development cycle. Each cycle comprises seven phases, from analysis through design and programming to implementation and maintenance. The first phase involves examining and modifying the business procedures impacted by a configurator; this includes defining and setting limits for the configurator’s support of these procedures. Once this is done, the second phase investigates the product/service line and identifies various modules and their connections. The third phase employs object-oriented methods to finalize the configurator design and define the requirements and user interface. The fourth to sixth phases include the creation, programming and deployment of the configurator within the organization. The final phase is centered on maintaining and further developing the configurator, which can be considered the starting point of the next development cycle.

The performance insights obtained from the previous development phases contribute to decision support in subsequent phases of the configurator development cycle. In particular, undesired performance results highlight what should be improved. For example, regarding the product/service quality category, low accuracy of resource estimations indicates a design flaw in resource consumption modeling in the configurator’s modeling step. Regarding productivity, a delay in completing a configuration indicates a lack of user interface clarity or inefficiency of runtime in the configurator’s programming step.

In summary, well-defined performance metrics can reveal performance bottlenecks throughout the configurator development cycle. Based on the performance review’s results, a configurator’s impact can be evaluated against its scope, targets and identified gaps. Performance insights provide support for reaching informed decisions to further develop the configurator.

This section describes the application of the proposed approach in the case company’s maintenance work configurator project through two rounds of development and performance assessment. This empirical study investigated the usefulness of the developed approach for integrating performance measures into configurator projects based on the effects of early considerations and preparation for performance assessment measurements. The high number of variables involved in such an investigation inspired the choice of a longitudinal, action research–based case study using data from information systems, observations and interviews. This enabled testing the proposed approach in a real industrial setting (Checkland and Holwell, 2007) and obtaining an in-depth understanding of the research phenomenon (Voss et al., 2002).

The case company operates large offshore oil and gas production plants in the North Sea, and a great number of maintenance jobs must be done promptly to ensure these plants’ safe, reliable production. Maintenance work orders specify the details and requirements of maintenance jobs to ensure that they are carried out efficiently, effectively, and in compliance with applicable standards and regulations so that a suitable crew can be planned and spare parts ordered in a timely manner. In this context, the company aimed to configure the scope and content of maintenance work orders, including required tasks and resources, parts and materials needed, and estimated duration. Notably, the case company did not have existing practices for developing a maintenance work configurator. Maintenance is a new area of application for configurators, which elevated the importance of properly scoping the configurator, supporting its development with performance measures and accurately evaluating the impact of introducing configurators to this field.

Figure 4 provides an overview of the case study timeline over its 12 months. The performance results from the first configurator development cycle contributed to refinement in subsequent development cycles. The proposed approach was evaluated at the end of the case study. Sections 4.1 through 4.7 present the implications of using the proposed approach for configurator development.

In the case study, a maintenance work configurator was developed in collaboration with the case company using the proposed approach, and the effects of the approach were observed. The case study had two main objectives: (1) to evaluate the usefulness of the proposed approach for supporting the development process and (2) to evaluate the effects of integrating performance measure development and assessments into configurator development processes.

The configurator project in the case company was studied from August 2021 through July 2022. During this period, four data collection processes took place that employed multiple methods to improve the validity of the study (Jick, 1979; Voss et al., 2002). First, data were extracted from the case company’s enterprise resource planning (ERP) system and configurator event log to determine the configurator’s performance before, during and after the project. Second, during the project, the researchers attended company meetings and workshops to observe discussions among those involved in the configurator’s development. Third, semi-structured interviews were conducted with managers to clarify the scope, aims, and targets of the project and with future users to understand their experiences in evaluations of the configurator’s usefulness and applicability. Fourth, data were collected via semi-structured interviews with managers to determine the extent to which the goals and objectives of configurator implementation had been achieved. Specifically, the managers were asked to evaluate the proposed approach, estimate the impact of the configurator and assess the alignment between the configurator’s performance and scope. Table 3 summarizes the data collection process.

The data acquired from the workshops and interviews were analyzed by (1) identifying relevant statements, (2) categorizing statements according to subject and (3) synthesizing categories of statements into generalized descriptions (Kvale and Brinkmann, 2009). Such descriptions concerned the events occurring during the project as well as the effects of the configurator and the developed approach.

The case company’s initial situation was analyzed before the configurator was developed so as to identify concerns in its business processes and cultivate initial conjectures and observations. The findings of this analysis were then used in scoping the configurator project.

The initial observations and conjectures in the case company highlighted issues in maintenance preparation, including repetitive tasks, work piling up, long lead times, inconsistencies in content and scope, and great variations in quality. The preparation team responsible for pumps and valves provides a representative example, as its work was typical of most maintenance tasks. It was known that the team followed no specific compliance guidelines other than legislative rules and regulations (i.e. the work was largely based on experience), indicating the potential to develop a configurator for maintenance preparation with the aim of making the time-consuming preparation process quicker and enabling faster resolution of failures. Additionally, the quality of specifications was improved. Variance that did not add value was reduced by introducing consistent content, structure, comprehensiveness and descriptions so as to minimize, for example, the need for later changes, revisions and reworking.

Since maintenance is a relatively new domain for configurators, it can be challenging to establish specific performance objectives and targets to be achieved by implementing a configurator, as the impact is not yet fully understood. Thus, objectives and targets were discussed extensively among the company’s employees, managers and configurator project members. To improve specification quality, expedite the time-consuming preparation process and decrease the non–value-adding variance of maintenance work descriptions, the following targets for the configurator project were formulated:

1. Effectiveness target: Reduce maintenance work preparation errors from 5% to below 2%.

2. Efficiency target: Lessen time spent on completing maintenance work orders by 80%.

3. Process compliance target: Attain a 95% reduction in the variability of work descriptions.

Regarding the targets, existing performance measures from the case company were identified and adapted for configurator performance assessment. The two effectiveness measures shown in Table 4 were already established by the case company and were adapted to the performance metrics for configurator assessment. The indirect efficiency and process compliance measures were not present, but the data required for the performance calculations were available. Other measures and their required data were not available at the scoping stage. The high-level objectives and requirements for developing the configurator and its performance metrics were identified at the end of the scoping stage.

The scoping stage revealed that the configurator should generate maintenance operation lists that include task descriptions, the necessary competencies to complete a task, the required number of people, task duration, and the necessary materials and tools. Based on the initial investigation of the existing situation, the integrated development of the configurator and performance metrics commenced.

The initial development of the configurator was carried out simultaneously with the development of the performance metrics. Table 4 provides an overview of the performance metrics specified in the development stage. The selection of measures was subject to time constraints (i.e. being able to make valid measurements within the time available) and data availability (i.e. being able to acquire certain data). The data requirements of the specified performance metrics were incorporated into the corresponding phases of configurator development. The collection of other performance data from the company’s enterprise IT systems was also specified.

The configurator development was iterative and aimed to develop an initial configurator with basic functionality to test and improve. Two months were dedicated to initial development before launching to key users, and four months were spent on further development and refinement based on the performance results from the initial launch.

Once the configurator was launched, its performance could be measured, with a focus on evaluating the use of the configurator and its effectiveness, efficiency and impact on process compliance. Figure 5 provides detailed quantitative measurements of these aspects.

In terms of the configurator’s use, the number of users of the system was an important metric for measuring the adoption rate. Another useful measure was the number of work orders configured relative to the total number of work orders created, which enabled measuring the extent to which the configurator was used over time.

Regarding effectiveness, an important measure identified was the amount of rework required later in the maintenance delivery process. Specifically, failing to prepare materials was measured, for example, in cases where the scope of a work order involved replacing a piece of equipment, but a spare part was not accounted for. This measurement required complicated manual data extractions, resulting in only two measurements being made. Another measurement was of the number of attempts made to schedule work orders for execution, which reflected issues such as uncertainties about the work scope, imprecise estimations of duration and necessary tasks that had not been prepared. Both measures showed that the configurator positively impacted effectiveness; notably, non-configured work had required up to 23 attempts before completion.

To measure efficiency, the time spent on configuration was used to reflect the direct impact on efficiency. Users’ sessions on the configurator were measured from start to finish, and the results showed that a majority of users took only a few minutes to configure work orders (average: 2.5 min), and 85% of work orders were generated in five minutes or less. While a direct comparison with non-configured work orders could not be made because the time consumed by the latter had not been logged, maintenance planners stated that it typically took 10–30 min to prepare such work orders (shown in Table 5). Therefore, it can be concluded that the configurator had a positive direct impact on efficiency.

Another measure was the processing time of work orders from pending preparation to preparation completed. It was found that many configured work orders were prepared faster than non-configured ones, indicating that the backlog of pending work orders was cleared more quickly. Users also reported that repetitive and simple work orders could be completed quickly and easily using the configurator, allowing them to focus their efforts on more complex tasks.

Regarding process compliance, the configurator inherently enhanced compliance via transparent, traceable knowledge-based decision support. The work orders generated by the configurator were based on the content of a knowledge base, which was assembled based on historical data and discussions with domain experts regarding best practices, regulations, standards, guidelines, etc. As shown in Table 5, the use of the configurator significantly reduced the number of maintenance activity descriptions that could be combined and matched for 581 defined equipment types. If necessary, users could make changes to the standardized output (e.g. revise wording, adjust estimated workload or add new operations). These changes were measured and manually evaluated for noncompliance or for the need to update the knowledge base.

Throughout the configurator’s development, relevant measurements were obtained and secured to continuously monitor and evaluate performance after deployment. This involved examining whether usage remained consistent and whether there had been any changes, either positive or negative. The monitoring involved overseeing performance targets and identifying potential gaps, as well as evaluating whether the current version could sustain the targets or whether adjustments were necessary, such as setting new targets, further developing the configurator, or introducing new metrics.

The performance results in the case study identified configurator performance bottlenecks. Specifically, user adoption showed significant potential for improvement, as the low number of active configurator users reflected an insufficient provision of support in user training during the implementation phase. Next, the low percentage of configured work revealed flaws in the specification process. Further investigation identified lost and crashing sessions in the data and revealed that the application was slower than the users had expected. Additionally, a one-time measurement revealed that some users did not utilize all the intended functionalities to configure maintenance activities due to a lack of understanding of the configurator concept.

Targeted actions were taken to refine the configurator in subsequent development cycles. Updates were made to the configurator and its knowledge base to resolve performance issues. The user training procedures were improved based on user feedback. Improvements were also made in response to general observations of the output data, including fixing general bugs, such as outliers, irregularities and blank fields. The data necessary to make these measurements were made available by considering the data collection early on and incorporating data collection into the configurator design. The insights from these adjustments contributed to continuous improvement in subsequent configurator development cycles.

Table 6 compares the performance results before configurator development, after the initial configurator launch (without continuous improvement enabled by the proposed approach), and after the final configurator launch (with the continuous improvement enabled by the proposed approach). To account for delayed effects on indirect performance measures, the performance results after the initial and final configurator launches were summarized four months after the changes took place. The results show that user adoption and effectiveness from the final launch were further improved after refinement, indicating a positive effect of the coordinated performance assessment and configurator development approach.

In conclusion, the case company achieved its initial targets for the configurator. The number of errors related to materials was less than 1%. As summarized in Table 5, the average time spent on work preparation using the configurator was measured and compared to the times reported by maintenance preparers, revealing an average time reduction of 88%. Additionally, the variability of work descriptions was reduced by more than 99%, indicating increased consistency and standardization.

To evaluate the effects of using the approach developed in this study rather than existing approaches (e.g. Forza et al., 2006; Haug et al., 2012; Hvam, 2006; Hvam et al., 2008), the benefits of the developed approach were evaluated in relation to (1) its support of the development process and (2) the completeness and accuracy of the performance assessments. Table 7 summarizes the value of the approach on the basis of statements made by employees in the case company during the interviews.

As suggested by Table 7, a strong focus on performance measures helped the company steer the project in a fruitful direction, as the configurator was continually modified to increase the benefits produced. By following the development process and adopting an approach incorporating performance assessment, the progress of development could be monitored and evaluated. The performance results, in turn, supported further refinement of the configurator to achieve continuous improvement. This ensured that the focus was on achieving the targets and taking informed action to reach them. The benefits of continually improving the configurated process, in turn, contributed to creating financial value. Taking material change as an example, errors in planning material for maintenance work can be costly, as they introduce non–value-adding activities related to ordering the wrong materials and shipping them back and forth to offshore platforms. By improving the quality of the configurator over development cycles, subsequent material changes can be reduced, which, in turn, contributes to cost reduction.

In summary, the benefits described in Table 5 would be difficult to achieve without a strong emphasis on continuous performance management. In other words, the developed approach can produce considerable benefits in cost reduction and efficiency improvements.

The present study was inspired by the findings in the reviewed literature, which showed that inadequate attention is paid to the early-stage development of configurator performance metrics for continuous improvement. To address this, this present study developed a coordinated approach to assessing performance in configurator projects that embraces configurator scoping, development and post-deployment phases.

Regarding academic contributions, this study’s findings extend the existing research on the development and implementation of configurators by providing guidance for integrating performance measurement development and assessment into configurator projects (Forza and Salvador, 2006; Haug et al., 2012, 2019b; Hvam, 2006; Hvam et al., 2008; Ladeby and Oddsson, 2011; Walcher and Werger, 2011). The case study demonstrated the benefits of the proposed approach in the process of developing and refining a maintenance work configurator. Specifically, using the approach inspired performance insights that informed decision-making throughout the development process, which was achieved by providing a structured approach from an early stage. Furthermore, the approach helped overcome data availability issues by promoting performance assessments throughout the development phase. Overall, the case study demonstrated the benefit of coordinated performance assessment in configurator development as well as the positive prospects of exploiting performance results for the continuous improvement of configurator projects. This extends the existing knowledge on the role of performance management in configurator projects.

The proposed approach offers practitioners a structured process that supports them in developing configurators and evaluating their impacts. In the case study, the approach proved useful in scoping the configurator project and carrying out measurements for development in an agile, iterative way. Thus, the approach will enable practitioners to iterate configurator development proactively to better achieve its objectives and requirements. The case also demonstrated that configurators can be applied in the maintenance field to configure work orders, which can significantly improve a firm’s maintenance performance. Given the life cycle of IT systems, a developed configurator must be maintained and regularly reviewed for further development. In this regard, assessments can be made to ensure ongoing effectiveness and alignment. As shown by the case study, companies can expect to benefit from applying the developed approach, especially in terms of ensuring that holistic performance results are available and can support the continuous improvement of configurator projects. Finally, the guidance offered by this study on how to ensure better integration of performance management into configurator projects represents an obvious element to include in education involving the development and use of configurators.

The literature review highlighted the need for a comprehensive performance structure and a coordinated approach that integrates development with performance assessment. To address this need, this study proposed a four-stage approach. The first stage, Scope, focuses on aligning the configurator development scope with the performance evaluation scope; the second stage, Develop, emphasizes the integration of performance resource planning into configurator development; the third stage, Implement and Measure, focuses on coordinating configurator implementation with performance data collection; the fourth stage, Review and Refine, prioritizes improvement actions based on performance insights. The proposed approach ensures data availability early in development, provides continuous evaluation for focused development and promotes a reliable, transparent assessment of the impact achieved by implementing a configurator.

An action research–based case study using data from information systems, observations and interviews was conducted to confirm the usefulness of the approach. In the case study, a configurator was scoped, developed, implemented and refined for the preparation process of maintenance work orders. Analysis of case data showed that the configurator significantly reduced time consumption and errors, and that it helped standardize maintenance activity descriptions in the preparation process. The case showed that the approach can help companies make informed decisions on configurator scope, provide useful performance information, and ensure a fit between a configurator’s scope and performance evaluation. Conclusively, this can lead to more effective, efficient configurator projects that align with intended aims and goals, which in turn reduces the chances of project failure and produces greater benefits for a company implementing the configurator.

This present study has some limitations that present opportunities for future research. The proposed approach was tested in only one case company, which calls for further testing in multiple case studies to assess its generalizability and validity (Voss et al., 2002). A potential area for investigation is the application of the approach to develop configurators in new fields, such as service configurators (Mueller et al., 2022), to explore whether the same benefits can be achieved in those domains. The present study focused on aligning configurator development processes to their objectives through the integration of performance management; however, due to the study’s scope, details regarding the root causes of potential misalignment were not fully discussed. Further research could investigate potential deviations in configurator development and their root causes in detail to further improve the design of configurator performance measures.

The authors acknowledge the funding received from the Danish Offshore Technology Centre. The authors would also like to thank the contributions of the case company’s employees.