Project portfolio selection and prioritization through supervised machine learning in continuous improvement projects

Despite the widespread implementation of continuous improvement projects (CIPs) such as Lean Production and Six Sigma, a significant number of them do not meet organizational expectations (Antony et al., 2022; Chakravorty, 2009). Empirical studies indicates that numerous initiatives result in suboptimal local improvements, fall short of anticipated outcomes or fail to generate a positive return on investment (ROI) (Antony et al., 2022; Easton and Rosenzweig, 2012). Analysis of why these failures occur indicates that many organizations often overlook the need for a comprehensive strategy to improvement, overly complicate their methods, misuse their chosen improvement techniques or choose inappropriate projects for improvement (Antony and Gupta, 2019; Kalashnikov et al., 2017; Kornfeld and Kara, 2011). Consequently, their efforts are misdirected and fail to align with and execute the organization’s strategic goals (Antony et al., 2022). Strategy execution involves transforming strategic objectives and themes into initiatives or collections of initiatives tailored to uphold the strategic intent and optimize potential strategic outcomes (Padhy, 2017). Consequently, operations management literature has consistently focused on the CIPs portfolio selection phase to examine the relationship between this selection and the success of implementing improvement projects (Kalashnikov et al., 2017; Kornfeld and Kara, 2011). During the project portfolio selection phase, portfolio managers identify, rank, prioritize and select CIPs to generate an optimal project portfolio selection under a complex multiconstrained problem affected by high uncertainty (Archer and Ghasemzadeh, 1999; Ghasemzadeh and Archer, 2000). This selection, aimed to maximize projects’ ROI while respecting resource constraints, is usually performed defining accurate projects critical success factors (CSFs) that allow to measure the potential success of a CIP (Padhy, 2017; Padhy and Sahu, 2011). According to the extant literature, main CSFs for CIPs typically include leadership commitment, employee engagement, clear communication of goals, process efficiency and customer focus (Gonzalez Aleu and Van Aken, 2016). As a matter of fact, being able to early assess CSFs improves the selection process allowing for:

• reducing the time from the business idea to market deployment;

• performing the early elimination of any nonstarters, reducing sunk costs;

• assessing earlier the identification of stakeholders’ and company’s requirements; and

• anticipate the implementation phase in the portfolio management, reducing errors in the planning and managing processes (Gonzalez Aleu and Van Aken, 2016; Hansen and Svejvig, 2022).

The high strategic relevance of the project portfolio selection phase gained interest in the operations management literature, and several authors have proposed alternative selection and prioritization methods ranging from simple methods like Project Ranking Matrix and Pareto analysis or complex methods like mathematical models and analytic hierarchy process (AHP) (Padhy, 2017). Although the literature on selection methods is extensive, certain aspects remain underexplored. Most of the proposed methods rely on expert judgment, which is often subjective and biased by managers’ personal experience on past projects (Kornfeld and Kara, 2011).; On the other hand, mathematical methods, such as optimization algorithms and AHP, are frequently too complex and difficult to apply in real-world business decision-making contexts (Kalashnikov et al., 2017). Moreover, these methods are inflexible, struggling to accommodate shifts in organizational priorities or adapt to external changes like market fluctuations or technological advancements (Padhy, 2017). In contrast, CIPs demand methodologies that are flexible and adaptive, capable of responding dynamically to evolving business environments and rapid technological advancements (Marzagão and Carvalho, 2016; Näslund, 2013). Emerging research indicates that recent advancements in artificial intelligence and machine learning techniques have the potential to overcome some of the limitations inherent in traditional project portfolio selection methodologies (Costantino et al., 2015). These technologies enhance the adaptability of selection processes, allowing for real-time data integration and analysis. Machine learning can analyze vast amounts of company data, enabling a more accurate and strategic decision-making by identifying the most promising projects based on historical company data and performance indicators. This allows companies to leverage their internal knowledge and insights for optimal project portfolio selection (Ban et al., 2018; Relich and Pawlewski, 2017). Literature has shown some applications of ML for project portfolio selection in different contexts (see, for example, Asawa, 2022; Costantino et al., 2015; Paiva et al., 2019). However, current literature focus on the accuracy of a single proposed method without assessing the impacts of technology application on selecting projects with the highest return ROI; often use synthetical and purposefully build databases, not grounded in real empirical settings; often rely on a single classifier or neural network without comparing different algorithms’ performances and to the best of the authors’ knowledge none of them focuses on the selection of continuous improvements projects in the manufacturing industry. More research is needed to explore the advantages of adopting ML technique to the selection of new CIPs. Therefore, coherently with the stated objectives, this paper aims to address the following research questions:

To answer these questions, the authors have structured the paper in the following way: the background section discusses the strategic relevance of selecting CIPs, the current methods for their portfolio selection and the advantages offered by machine learning algorithms.

Then, the authors provide a demonstration of an application of supervised machine learning to CIP portfolio selection in an Italian manufacturing company. The method proposed, collecting and consolidating implicit knowledge form past projects, predicts a degree of expected success for incoming projects. Furthermore, in developing the method the authors propose a novel data augmentation technique based on fuzzy sets theory to overcome the difficulty that companies have in structuring consistent and large data set. By comparing the results predicted, with the ROI of the final project and the company WACC, the authors were able to discuss the benefits and limitations of applying machine learning to the project portfolio selection phase, and to propose suggestions for further research development on the topic both under a research and practical perspective.

CIPs, such as Kaizen events, Six Sigma projects and Lean Six Sigma (LSS) projects, are structured initiatives aimed at enhancing processes, reducing inefficiencies and delivering measurable value within organizations (Bessant et al., 1994; Khan et al., 2019; Marzagão and Carvalho, 2016). These projects focus on systematic problem-solving, data-driven decision-making and incremental or breakthrough improvements in areas such as quality, cost, delivery and customer satisfaction (Jonsdottir et al., 2014). To maximize their effectiveness, CIPs must be aligned with the overarching strategy of the organization (Jung et al., 2009; Kornfeld and Kara, 2011). This alignment ensures that the improvements achieved directly contribute to the organization’s strategic goals, such as competitive advantage, market positioning and long-term sustainability (Jimoh et al., 2019; Kaye and Anderson, 1999). When CIPs are integrated with strategic priorities, they address immediate operational challenges and build organizational capabilities, drive innovation and support the broader mission and vision of the organization (Anand et al., 2009; Gutierrez-Gutierrez and Antony, 2020). Misalignment, on the other hand, can lead to wasted resources and fragmented efforts that fail to generate meaningful or lasting impact (Chakravorty, 2009). For this reason, the project portfolio selection phase is essential for these types of initiatives, as it ensures that only projects aligned with strategic priorities are undertaken, enabling optimal resource allocation and avoiding fragmented or misaligned efforts, which are likely to result in local and suboptimal improvements (Kalashnikov et al., 2017; Kornfeld and Kara, 2011). Consequently, project portfolio selection in this field have become increasingly significant as strategic tools for delivering projects, thereby garnering substantial academic interest. Ghasemzadeh and Archer (2000) propose a framework for project portfolio selection based on two levels:

1. individual project analysis, prescreening and screening; and

2. the systemic optimization of portfolio selection.

In the first stage, evaluation parameters are defined and calculated for individual projects, leading to initial project prioritization; subsequently, in the screening stage, nonstarters are eliminated, and the previously selected projects are prioritized to be included in an extant portfolio. In manufacturing companies aiming for continuous improvement, the selection criteria for new projects primarily involve assessing each project’s potential to meet the CSFs specific to critical performance indicator (CPI) (Padhy, 2017). Indeed, these projects differ from traditional R&D (Research and Development) projects in several aspects. They focus on practical application and strive for swift, result-oriented outcomes within a much shorter timeframe; they require more predictable outcomes and necessitate active participation from all stakeholders to ensure success; they need for clear goals and efficient project management to align all involved parties toward immediate and applicable results (Padhy and Sahu, 2011).

Gonzalez Aleu and Van Aken (2016) through a systematic literature review, mapped the CSFs for CIPs, categorizing them across several dimensions. These dimensions include task design, team design, organizational and strategic support and CIP processes clear set up. Within task design, key factors focus on goal clarity, alignment and the development of objectives that resonate with both team members and stakeholders. Team design CSFs emphasize role clarity, stakeholder representation, team autonomy and the integration of external expertise. Organizational support plays a critical role, encompassing the availability of essential resources (e.g. time, materials and training), facilitation, and the alignment of organizational policies and structures with CIP objectives. Finally, effective CIP processes are characterized by clear methodologies, structured planning, follow-up activities and mechanisms for institutionalization, ensuring sustainable impact. Therefore, to ensure a proper selection of projects, these criteria should be carefully considered during the project portfolio selection phase and incorporated into the methods used by organizations to evaluate and prioritize CIP projects (Jimoh et al., 2019). The literature categorizes project selection methods into two main groups: simple approaches and complex approaches. Simple approaches are commonly used in practice and are often incorporated into commercial software tools. In contrast, complex approaches rely on a more rigorous scientific foundation. The following subsections lay the groundwork for understanding CIP portfolio selection.

Simple methods for portfolio selection of CIPs include techniques such as pareto analysis, matrix-based techniques like assessment matrix or prioritization matrix which involve setting up a grid to compare different projects against set criteria to determine priorities; quality function deployment which uses matrices to convert customer needs into actionable technical advice for CIPs selection and other methods like Balanced Scorecard evaluation and brainstorming, which are based on managerial judgment (Padhy, 2017). Due to their simplicity, practicality and ease of implementation, these techniques are widely used in organizations and are often embedded in commercial software. However, these methods also have several limitations. Reliance on managerial judgment can lead to highly subjective evaluations, which may not consistently reflect objective reality or be replicable. Furthermore, these techniques might oversimplify complex CIPs decisions, overlooking factors or interdependencies between criteria and projects (Kalashnikov et al., 2017; Padhy and Sahu, 2011). Literature also reports that they are generally less effective in dynamic environments where conditions change rapidly, as they typically do not account for real-time data or emerging trends (Kornfeld and Kara, 2011). Table 1 provides a summary of the literature on simple methods for the selection of CIPs.

Complex approaches include various techniques like the AHP, Options Pricing, Fuzzy Logic and Mathematical Programming. The AHP, which falls under the category of multiple criteria decision-making methods, breaks down the main problem into smaller, more manageable problems based on a predefined hierarchy of importance. Kumar et al. (2009) highlighted that the AHP is highly effective for LSS prioritization decisions that involve both tangible and intangible factors. Among the options pricing techniques, the real options framework (ROF) stands out as particularly valuable. The ROF offers decision-makers a range of choices for investment, expansion or withdrawal from a project based on new information. All authors concur that the ROF is a superior alternative to traditional evaluation methods because it explicitly incorporates the value of future flexibility into decision-making (Condé and Martens, 2020; Pakdil, 2022). Indeed, the option values within a project not only reflect its actual worth but also enhance decision-making flexibility. Decisions regarding the selection of CPIs often face constraints stemming from internal organizational policies and external system requirements. In these scenarios, data typically exhibit a high degree of uncertainty or “fuzziness.” To address this, fuzzy set theory is commonly used in the literature as a framework for managing uncertainties of this nature. Finally, the final category of models within the complex approach is grounded in Mathematical Programming techniques (Bilgen and Şen, 2012). Authors like (Kalashnikov et al., 2017; Saghaei and Didehkhani, 2011) use mathematical formulations to solve optimization problems in the selection of CPIs. They set up an objective function, which is what needs to be maximized or minimized (e.g. costs, time, resources). Constraints related to the problem are then incorporated, and the programming method reports an optimal decision based on the defined criteria and constraints. Table 2 provides a summary of the complex methods for CIPs reported in literature.

Literature has shown that both simple and complex methods for CIPs portfolio selection have limitations. Simple methods rely heavily on managerial judgment, which can lead to subjective and potentially inconsistent outcomes (Kalashnikov et al., 2017). On the other hand, complex methods, like Mathematical Programming or AHP, can be rigid, requiring extensive data input and computational resources, and may not adapt well to changes in project evaluation or external conditions (Condé and Martens, 2020; Pakdil, 2022). These methods often struggle to accommodate real-time data, potentially making them less flexible for portfolio decisions in fast-paced environments. According to some recent literature external to CIPs, the adoption of artificial intelligence in project portfolio selection can lead to improved business decision-making processes and better exploitation of the information stored in corporate databases (Aigner et al., 2023; Costantino et al., 2015; Ma et al., 2021). Specifically, using machine learning algorithms has been shown to boost the reliability of selecting project portfolios (Mariani and Mancini, 2025). This improvement occurs because machine learning extracts implicit knowledge from historical data, thereby reducing the reliance on subjective and potentially erroneous managerial judgments (Zhang et al., 2020). To better illustrate these differences, Table 3 compares traditional approaches (both simple and complex) with machine learning–based approaches, summarizing insights derived from the reviewed literature and highlighting the distinctive contributions of our study.

Despite the capabilities of these algorithms in aiding the portfolio selection process, it appears that no existing research specifically addresses their use in selecting CIPs. Moreover, according to literature there is a growing need to explore how advanced data-driven tools can enable more effective governance in project portfolio management – helping organizations not only to prioritize high-impact initiatives, but also to institutionalize more transparent, objective and learning-oriented decision-making processes (Hanisch et al., 2023; Manoharan et al., 2023). Consequently, there is a significant need for further research in this area. This is essential because the continuous improvement domain involves dynamic and ongoing adjustments where AI (Artificial Intelligence) could significantly enhance efficiency and effectiveness by identifying patterns and predicting outcomes from complex and evolving data sets, leading to more informed and strategic decision-making for incoming projects (Najafi et al., 2024). To bridge this gap the authors sought to specifically to demonstrate the application of supervised machine learning classification for CPIs project selection for the following reasons:

• Supervised machine learning classification produces an inductive hypothesis, i.e. a function capable of “learning” from the results provided during the training phase, which is then able to predict results for all the examples not provided. Thus, the predictive capability of machine learning enables the prediction of the future values’ classes by learning from the historical knowledge of company data, which is used as the training data set. This leads to increased objectivity in terms of the results, which are based on the historical experience of the company and not on individual managers’ judgments (Ray, 2019).

• The method brings with it ease of use and implementation both in relation to programming and once the interface has been created, as well as regarding the interpretation and analysis of the results (Ray, 2019).

• Supervised machine learning includes classification algorithms that are, currently, the most precise and accurate in terms of making predictions. In addition, the large number of classification algorithms at one’s disposal allows the selection of the best ones and comparison in terms of classification accuracy (A. Singh et al., 2016).

• The method allows scalability and adaptability to different projects and companies by modifying the input/output variable forming the training set with respect to the data available within the company (Sarker, 2021).

The dynamic learning capability of machine learning enables the method to be adapted not only to the portfolio selection phase but also to the subsequent management phase, in which decisions about corrective actions and reevaluations of the defined portfolio may be needed. Moreover, completed projects immediately become part of the training set, allowing the creation of a self-updating system, which adapts new predictions to the updated training set.

A single case study approach was used to investigate the use of supervised machine learning in CIPs portfolio selection. This approach involved selecting a specific company case and using multiple perspectives rooted within the context to explore and illustrate the dynamics of machine learning adoption in CIPs project portfolio selection. The choice to adopt a single case study design was deliberate and grounded in methodological considerations. The selected company offered a unique opportunity: it had implemented a substantial number of CIPs over the past decade and had recently launched a digital transformation initiative explicitly aimed at enhancing its project selection processes. This context provided both the motivation and the infrastructure needed to pilot a novel data-driven approach. Furthermore, the single-case approach allowed for deep, contextualized insights into the feasibility and organizational implications of adopting machine learning for portfolio decisions – insights that would have been diluted in a broader, multicase comparison (Dul and Hak, 2007; Yin, 2012). This aligns with the exploratory and demonstration-oriented nature of our study Alvesson (2011). Figure 1 shows the research design of this study.

The Italian division of a multinational manufacturing group, employing over 13,000 people globally, specialized in the production of high-quality automotive components. This division is particularly known for its innovation in developing eco-friendly and energy-efficient vehicle parts, catering to both consumer automobiles and commercial vehicles. Our selection of this division was influenced by two primary factors. First, the division demonstrates a significative commitment to continuous improvement, having instituted a multitude of projects (completed and ongoing) aimed at enhancing operational efficiency and sustainability. Efforts include the integration of advanced robotics to streamline production processes, the utilization of lean manufacturing principles to minimize waste, and the adoption of eco-friendly practices to reduce energy use and carbon emissions. Second, the division has recently initiated a comprehensive digital transformation strategy – intensively supported by the direction board, integrating big data analytics to refine the selection and management of its CIPs. This strategic orientation toward CIPs implementation and digitalization have fostered a strong interested in collaborating in this research.

The data gathering process followed for the design of this research aimed to maximize the validity and reliability of the collected data and to enhance the accuracy of the obtained results. To structure the data set, the authors relied upon a combination of primary and secondary information sources from the above-mentioned manufacturing group. The primary sources were derived from 12 structured interviews with project and portfolio managers. In parallel, the secondary sources came from internal CIPs portfolio management documents, such as internal reporting on single project progress and achievements compared to fixed milestones. It is important to underline that only CPI initiatives and projects were considered for the analysis. In addition, general information was gathered on the organization and management of projects (e.g. internal procedures and best practices) that the company implemented in the past 10 years. To involve the managers of the company and gather data, the following steps were performed:

• The research and its objectives were presented to the Project Management Office and project portfolio managers of the company’s Italian division. Thus, permission and sponsorship for data collection from 28 projects in the Italian division of the company were obtained. The projects’ budgets ranged between 300k and €1m, and their relative durations lasted between 12 and 36 months.

• A meeting was held with a Project Management Professional (PMP) certified portfolio manager, during which 13 CSFs for CIPs success were defined ( Appendix 1), and the final ROI achieved by each project was set as the parameter to evaluate the level of success achieved. To convert continuous ROI values into discrete success classes for classification purposes, we defined five threshold intervals that reflect both statistical distribution and economic relevance. The classification boundaries were designed in collaboration with company managers and aligned with the firm’s internal investment evaluation logic. Specifically, the “Very Low” (VL) class was assigned to projects with ROI below 5%, indicating poor economic performance; “Low” to those between 5% and 7.9%; “Medium” to projects between 8% and 9.9%, closely aligned with the company’s WACC of 8.7%; “High” to projects between 10% and 13.9%; and “Very High” (VH) to those equal to or exceeding 14%.

• The data collection phase took place, where, through a Microsoft Forms questionnaire, the managers outlined the level of each project’s success and the relative weight of the CSFs using a five-point linguistic evaluation scale. For some quantitative criteria (e.g. budget size, workload), primary project data was available, while for other qualitative criteria (e.g. level of technical and organizational complexity), managers’ judgments were still necessary. However, to comply with a non-disclosure agreement established with the company, the primary data relied upon linguistic evaluations expressed by managers, and the secondary data was collected to validate the experts’ judgments.

• Fuzzy logic combined with a data augmentation procedure was then implemented with regard to the collected data set to train and test the classifiers for project success prediction.

• The obtained results were then presented to the global project management office (PMO) and to project portfolio managers, and results were discussed, and compared with the actual project selection performed by the company. After this process the PMO, discussed with the team the possibility of implementing the method in the company as a decision support system for selecting incoming CIPs, highlighting potential benefits and organizational barriers.

Due to confidentiality constraints and internal policies, the use of linguistic assessments was the only viable method for collecting expert-based evaluations. This introduces a degree of subjectivity, but it also reflects realistic conditions under which many industrial organizations operate. This justifies our adoption of fuzzy logic as a means to model this epistemic uncertainty and make it tractable for machine learning applications, as already done in extant research (Nassif et al., 2019; Wu et al., 2021). The next section presents the procedure adopted to perform a data augmentation on the limited data set at disposal, to increase classifiers results’ accuracy and the result coming from the machine learning projects results prediction.

When applying supervised machine learning to a data set, the greater is the amount of data collected, the greater will be the accuracy of the classification results obtained (Ray, 2019). In the case analyzed the company gave permission to collect data on only 28 projects, allowing the authors to build a rather reduced data set to apply supervised machine learning and obtaining significant results. Thus, the procedure was divided in two sequential steps. The first one, entirely developed on MATLAB, aimed to generate a valid and reliable data augmentation procedure, able to account for the epistemic uncertainty (imprecision) connected to the linguistic evaluations. The second phase, developed in a Python 3.0 environment, was designed to classify the predicted ROI levels of upcoming projects using supervised machine learning algorithms.

Much of the theoretical foundation for decision-making processes relies on fuzzy numbers. Fuzzy logic is a powerful tool for representing imprecise, ambiguous and vague phenomena that are better expressed in linguistic rather than numerical terms. Fuzzy numbers – typically modeled as triangular or trapezoidal functions – allow for the mathematical representation of uncertain or subjective concepts, capturing the idea that a value can belong to a range with varying degrees of membership. This is particularly relevant in the evaluation of CSFs during the initial project portfolio selection phase, where expert judgments are often more reliably expressed as qualitative ranges than as exact values (Relich and Pawlewski, 2017). As in other recent work, the data for this paper was collected in the form of linguistic evaluations, thus modeling them as fuzzy triangle numbers appeared to be the most sensible choice (Jafarzadeh et al., 2018). Five linguistic expressions ranging from VL to VH were established and mapped to fuzzy triangular numbers through Table 4.

However, fuzzy logic representations cannot be directly processed by most machine learning algorithms, which typically require large data sets in numerical format. To address this limitation and strengthen the training data set, we developed a data augmentation procedure. Once the fuzzy numbers were defined, they were converted into corresponding probability distributions. This transformation enabled the generation of synthetic numerical data that preserved the uncertainty embedded in the original expert evaluations. At the same time, it provided a systematic and intuitive approach to augment the data set, making it more suitable for effective training of machine learning classifiers.

The proposed methodology consists of transforming each input triangular fuzzy number $λ$⁠, obtained through the mapping procedure of Table 1, into a Gaussian distribution p. This process is performed by assuming a certain standard deviation which can be regulated through an input parameter $α$⁠. The choice of the type of probability distribution is arbitrary and is up to the analyst, who must select the one able to better represent the modeled phenomena. In fact, whichever the chosen distribution is, the critical point is that the extracted sample should represent as best as possible the initial uncertainty contained in the linguistic evaluation and, therefore, in the original fuzzy number $λ$⁠. This verification can be done by antitransforming the sample obtained from the distribution, verifying that the fuzzy number obtained (induced fuzzy number) corresponds to the starting fuzzy number (intended fuzzy number) and measuring the error e associated with the overall transformation process. The procedure is then repeated R times to choose the sampling set of N points that best represents each evaluation criteria. N is the fundamental parameter regulating the data augmentation procedure, since every project is augmented to N ones with the same class of ROI, but with slightly different combinations of CSFs. In this way it was possible to overcome the problem of the small amount of data provided by the company and at the same time to develop a procedure that considers both the human imprecision and the uncertainty of the assessments on the predicted success level of incoming CIPs.

The parameters used in the data augmentation process were selected based on a trade-off between preserving the semantic meaning of the original linguistic evaluations and introducing enough variability to improve model generalization. Specifically, the number of samples generated per project (N = 200 in the initial experiments) was chosen to ensure a sufficiently large training set while maintaining manageable computational complexity. The standard deviation (σ) of the Gaussian distributions was set based on the spread of each fuzzy triangular number, ensuring that the sampled values remained within the support of the original linguistic variable. These choices were empirically validated through an iterative process aimed at minimizing the reconstruction error between the intended and induced fuzzy numbers (as detailed in Figure 2 and  Appendix 2). The implemented methodology allows for the generation of outputs that go beyond simple linguistic evaluations, producing a probability mass function that reflects the predicted ROI level for each CIP. This transformation enables the problem to be effectively modeled as a supervised machine learning classification task. Using the augmented data set, seven different machine learning classifiers were trained and evaluated. Their performance was carefully monitored to identify the algorithm with the greatest potential for practical implementation.

The outputs of the data augmentation process were used as inputs for the machine learning classification. The choice of the seven classifiers [Decision Tree; Gaussian Naïve Bayes; K-Nearest Neighbors (KNNs); Logistic Regression; Multilayer Perceptron (MLP); Random Forest; Support Vector Machine] was guided by their widespread use in supervised classification tasks, their interpretability and their varying levels of complexity (Kotsiantis, 2007; A. Singh et al., 2016). This diversity allowed us to compare simple probabilistic models (e.g. Naïve Bayes), distance-based classifiers (e.g. KNN) and more complex ensemble and neural approaches (e.g. Random Forest, MLP). Specifically, Decision Trees build flowchart-like structures for decision rules; Naïve Bayes relies on probability theory and strong independence assumptions; KNN classifies instances based on proximity to known examples; Logistic Regression estimates probabilities using linear decision boundaries; MLP is a feedforward neural network capable of capturing nonlinear relationships; Random Forest combines multiple decision trees to reduce overfitting; and SVM constructs optimal separating hyperplanes, particularly effective in high-dimensional spaces (A. Singh et al., 2016) .

Once the linguistic evaluations of the 13 CSFs and the corresponding project success classifications for a given number of projects were uploaded into MATLAB, the classification algorithm generated N numerical observations to accurately describe each of the 13 CSFs. Here, N is a variable parameter adjustable by the analyst; for this study, it was set to 200. The results were then formatted into a table, prepared for uploading into a Python environment for further analysis. Simultaneously, a matrix was generated to record the minimum error associated with each of the CSFs, as determined through their respective optimization cycles (see  Appendix 2). In the testing phase of machine learning classification, model performance is typically assessed by randomly splitting the data set into training and testing subsets. Accuracy, defined as the ratio of correctly classified instances to the total number of predictions, offers a general indication of a model’s overall performance. However, when class distributions are imbalanced – as in our case – accuracy alone can be misleading, as it may not adequately reflect a model’s ability to correctly classify minority classes. To address this, we also report precision, recall and F1-score: precision measures the share of true positives among predicted positives, recall captures the share of true positives among all actual positives, and the F1-score is the harmonic mean of precision and recall, offering a balanced assessment that accounts for both false positives and false negatives (Kotsiantis, 2007). The results from the analysis performed on the seven selected classifiers are documented in Table 5.

The results in Table 4 indicate that KNN, Random Forest, Support Vector Machine and MLP all achieve extremely high accuracy (> 0.99). This performance is confirmed by equally high values of precision, recall and F1-score – above 0.94 – signaling that these models maintain both high sensitivity and specificity across most ROI categories. MLP attains a precision of ∼0.96 and recall of ∼0.94, yielding an F1-score of ∼0.95, which suggests it provides a particularly balanced classification outcome even under class imbalance conditions. However, the augmented testing set was generated from specific sequences of probability distributions, which also produced a large number of similar samples within the training set. As a result, a noticeable statistical correlation emerged between the training and testing sets – one that the top-performing algorithms were able to exploit. To more accurately assess the performance of these four classifiers, it was necessary to evaluate them on projects whose linguistic assessments were entirely uncorrelated with those in the training data. To achieve this, one of the 28 projects provided by the company was randomly selected and reserved exclusively for testing, while the remaining 27 projects were expanded – via the data augmentation procedure – into a training data set of 5,400 samples. To further evaluate the robustness of the models, one project from each ROI class was iteratively removed from the original data set and used as an independent test sample. Each of these test projects was then augmented to 200 data points to assess the classifiers’ accuracy. The results revealed that most models exhibited significant overfitting, performing well on training data but poorly on these unseen samples – except for the MLP. Overfitting is a common challenge in machine learning, occurring when a model becomes too tailored to the training data and loses its ability to generalize effectively to new, unseen inputs.

The aim of using supervised machine learning algorithms in CIPs portfolio selection is to predict the success level of future projects by training the algorithm on historical project data. To address the risk of overfitting resulting from the data augmentation process, we tested the MLP model on previously unseen projects, excluded from the training set. To ensure that the model was not simply memorizing the augmented data, we carefully designed the validation procedure such that all augmented samples derived from a given project were included only in either the training or the testing set, never in both. This partitioning helped preserve the independence between the sets and ensured that performance metrics reflect genuine generalization ability. Also, it allows for an evaluation of the algorithm’s predictive accuracy by comparing the predicted success levels with the actual outcomes.

The MLP was the only algorithm that achieved satisfactory accuracy across all classes except for “Very High.” As illustrated in Figure 3, the MLP accurately classified projects with “Very Low” success at over 60% accuracy, “Medium” success projects at over 80% and “High” success projects at more than 90%. It perfectly predicted “Low” success projects. However, its performance on “Very High” success projects was notably poor, predicting them as “Low” 64% of the time and as “High” 36% of the time, demonstrating a significant challenge in accurately classifying projects with the highest success potential. These findings are particularly relevant for practitioners: even though the model does not yet capture top-performing projects with full reliability, it already provides concrete value by identifying low-return projects at an early stage, helping managers to avoid allocating resources to initiatives below the company’s WACC threshold and redirecting efforts toward more promising opportunities.

This is due to the fact that MLP also suffers from overfitting, especially in the “Very Low” and “Very High” classes, with the latter not being recognized at all. This problem can be partially solved by regulating the parameter N of the implemented data augmentation technique. In fact, when increasing the number of samples from each distribution in the training set, the model is more likely to generate project feature combinations never experienced before, allowing the neural network to also recognize unseen patterns. Figure 4 shows a comparison between the accuracy of the model regarding a “Very Low” unseen project, the most misclassified after “Very High” ones, with N = 200 (5,400 elements in the training set) and N = 800 (21,600 elements in the training set).

By increasing the size of the data, the accuracy in relation to the class rises from 60% to approximately 85%, confirming the above-mentioned hypothesis. However, consistently raising the parameter N to enhance the training set size also substantially increases computational efforts and the time required for data augmentation. In general, it is possible to state that the effectiveness of this approach is contingent on the size of the historical data set; a larger data set reduces the need for higher N values to prevent overfitting. Unfortunately, for this specific case the limited number of projects classified under the “Very High” success category reduced the ability to boost accuracy through data augmentation, as the small sample size does not sufficiently represent the diversity needed for effective model training.

To evaluate the practical relevance of the proposed method, we compared the predicted ROI classes of the 28 projects – generated using the MLP classifier – with the actual ROI achieved. All projects were assessed based on a uniform set of 13 CSFs, ensuring consistency and comparability across the data set. Table 5 summarizes the results of this comparison. As previously noted, the database consisted solely of projects that the company had already completed. In fact, for building the data set, the management of the manufacturing company was asked to indicate the final ROI achieved by the projects and the relative weight of each CSFs in achieving the final ROI. A straightforward way of assessing the effectiveness of the presented model is to compare the ranking of projects obtained adopting the MLP classifier and the Weighted Average Cost of Capital (“WACC”) which is a weighted average of the cost of equity capital and the cost of debt net of tax shield on interest expenses. WACC is the rate at which companies must remunerate its financiers, and it is often referred to as “opportunity cost,” meaning that the minimum remuneration for projects undertaken by companies should always be at least higher than the WACC, to allow them to earn as well as cover the debt (Miles and Ezzell, 1980). Since the company is listed on the stock exchange, it was possible to estimate a WACC equal to 8.7%, using the following formula:

Where Ke, the cost of equity capital was estimated to be equal to 9.6%, E/(D+E) was equal to 85.9% and D/(D+E) was equal to 14.1% (based on the company capital structure in 2021); the Corporate Tax Rate was equal to 24% (Italian Corporate Income Tax rate) and the Cost of debt (“Kd”) was equal to 3.7%, estimated as the 12-months average of the daily rate of return of Euro Interest Rate Swap in 2022 (1.9%) plus the 12-months average of the daily spread for BBB-rated corporate bonds in 2022 (1.7%) (Damodaran, 2007). Once the company’s WACC was calculated, it was used as a benchmark to evaluate the actual financial performance of each project. We then compared the final ROI achieved by the projects with the ROI class predicted by the machine learning model. This comparison allowed us to assess the model’s practical usefulness in identifying which projects were likely to exceed the economic threshold represented by the WACC. The detailed results of this comparison are presented in Table 6.

Since all the projects listed in Table 5 were actually completed by the company, applying the MLP classifier during the project selection phase could have been beneficial. Specifically, the model might have helped the company avoid selecting projects whose final returns were significantly below the firm’s WACC, set at 8.7%. By identifying these low-performing projects early, the organization could have redirected resources toward more promising initiatives. Although the limited data set reduced the model’s ability to accurately classify projects with VH ROI, the comparative analysis of actual ROI, WACC, and predicted success classes demonstrates the practical value of using classification algorithms for CIP portfolio selection. In fact, the model correctly identified over 75% of the projects with low ROI. Such predictive insights can enhance decision-making by enabling more effective prioritization of high-return projects.

Selecting the right project portfolio is crucial for the successful execution of new CIPs. Incorrect project selection can significantly hinder an organization’s overall efficiency and productivity (Pakdil, 2022; Singh et al., 2023; Zhang et al., 2020). This study introduces a methodology that combines machine learning techniques with CSFs to address this challenge, offering several theoretical implications. First, while the literature provides a variety of methods for project portfolio selection, most have notable limitations. Simple methods, which rely heavily on managerial judgment, often suffer from subjectivity and inconsistent results (Kalashnikov et al., 2017). In contrast, complex methods, such as Mathematical Programming or the AHP, require extensive data and computational resources. These approaches are often too rigid to adapt to evolving project conditions or external changes such as market fluctuations or technological advancements (Hsieh et al., 2012; Pakdil, 2022). Our paper builds on this literature by proposing that AI-based techniques – specifically supervised machine learning – can systematically support the analysis and selection of CIP portfolios. Compared to previous studies using machine learning in other domains such as Costantino et al. (2015) and (Zhang et al., 2020) this work introduces an innovative data augmentation technique designed to improve accuracy when data sets are small or incomplete. In addition, the use of fuzzy logic accounts for the epistemic uncertainty that often characterizes expert-based project evaluations. This study also contributes to the theory of rational decision-making (Simon, 1979). Traditional selection approaches often depend on subjective or intuitive judgment (Constantiou et al., 2019), whereas our method supports the move toward data-driven decision processes advocated by March (2006). By leveraging past project data, the model enhances the objectivity of project selection and enables more effective management practices. The findings also connect to organizational learning theory. Organizations should “learn from the past” by using accumulated knowledge to inform future strategic directions (Crossan et al., 1995); literature has extensively demonstrated that analyzing lessons learned and other insights from CIPs fosters organizational learning (Gutiérrez et al., 2012; Savolainen and Haikonen, 2007; Sony and Naik, 2012). Our research supports this perspective by using an algorithm that analyzes past project data – specifically, CSFs and ROI – to inform future project selection. This ensures that strategic CIPs portfolio selection is not just based on intuition or sporadic data but is grounded in a systematic analysis of historical performance, thereby increasing the likelihood of future success and alignment with long-term strategic goals. This approach exemplifies, as already mentioned in literature (Balasubramanian et al., 2022; Ransbotham et al., 2020; Zhou et al., 2020), how organizations can leverage on machine learning to turn past project data into actionable insights, promoting a culture of continuous improvement and informed decision-making. Recent research has emphasized that the integration of algorithmic systems into managerial processes extends beyond technical optimization, influencing how decisions are legitimized, responsibilities are distributed and governance structures evolve (Al-Surmi et al., 2022; Balasubramanian et al., 2022; Buçinca et al., 2021). By replacing or complementing human judgment, data-driven models – particularly those built upon black-box algorithms – introduce new challenges related to transparency, interpretability and accountability. Our work reflects this shift by proposing a model that, while technically accurate, also demands critical reflection on how predictions are integrated into project selection routines. In doing so, we align with ongoing discussions on algorithmic governance and organizational change (Ransbotham et al., 2020; Zhou et al., 2020), highlighting the importance of combining technological innovation with adequate institutional safeguards and managerial oversight.

In addition to the literature on project portfolio selection methods, this work makes a contribution to the existing research on CIPs CSFs by providing a method that automatically relates different combinations of CSFs to project success. Historically, the literature has identified various frameworks for CSFs in CIPs that emphasize the relevance of project management approaches, optimization processes and alignment with strategic objectives (Gonzalez Aleu and Van Aken, 2016; Marzagão and Carvalho, 2016; Näslund, 2013). Our research contributes to this body of knowledge by enabling each organization to tailor project selection based on the CSFs that have historically been most impactful in driving ROI growth for individual CIP. Our work provides a method for selecting the combination of CSFs that best supports strategic alignment and enhances the potential for future project success. This approach reinforces the importance of learning from past experiences and promotes a more tailored application of CSFs in CIP portfolio management.

As for practitioners, the research findings should promote an alternative and predictive method for individual project analysis and selection in project portfolio management. Drawing on current studies on novel perspectives for machine learning use in project portfolio selection (Asawa, 2022), our research findings suggest that a company can effectively implement supervised machine learning as a decision support system for CIPs portfolio selection. By training on past projects’ CSFs (e.g. leadership commitment, scope clarity, team competences, process standardization and expected client impact), the machine learning model predicts the likely ROI class of new initiatives, enabling managers, Black Belts and Master Black Belts to prioritize those projects that are most likely to exceed the company’s WACC. This predictive capacity offers three concrete advantages for LSS programs. First, it helps reduce the number of “failed” projects by filtering out, already in the selection phase, initiatives with limited strategic value – such as defect reduction in non-critical processes or cycle time reductions with minimal impact on customer satisfaction. Second, it enables a sharper focus on high-impact opportunities, for example projects aimed at variability reduction in core manufacturing lines, strategic supplier quality management or energy efficiency improvements, which are more likely to deliver substantial ROI. Third, it strengthens deployment transparency and legitimacy: because project ranking is grounded in historical organizational data rather than solely in managerial judgment, portfolio decisions become easier to justify to leadership teams and cross-functional stakeholders.

Further, the suggested method is adaptable, efficient in terms of computation, user-friendly and uses historical data from the company to support decision-making. It is adaptable, since it allows different companies to select various types of CSFs that align with their specific needs. This flexibility is also enhanced by the system’s capability to integrate new project data into the database immediately upon project completion, thus continuously updating the algorithm’s knowledge base. Excluding the necessary data augmentation phase – required to ensure a sufficiently robust data set for meaningful demonstration – the machine learning component of our approach is computationally swift (processing time was less than 15 seconds). In addition, the system is user-friendly, designed to operate on commercial software platforms such as KNIME or Orange. This combination of adaptability, real-time data integration, speed and ease of use makes the method applicable across a wide range of industries where structured project portfolio decisions are critical – such as health care, energy, logistics and public administration. In such contexts, where resource constraints and strategic alignment are critical, the ability to efficiently prioritize and select the most impactful CIPs projects becomes essential to ensure that continuous improvement initiatives deliver tangible operational and organizational value. For example, in manufacturing, the model could support the early prioritization of projects aimed at reducing defect rates in high-volume production lines, or energy efficiency initiatives that directly cut operating costs and environmental impact. In health care, it could help select projects focused on lowering patient waiting times or reducing errors in diagnostic processes, where both efficiency and quality of service are critical.

In addition, it also allows for the simulation of various scenarios, testing how changes in the combination of CSFs might positively or negatively affect the predicted outcomes. By expanding the training set with more projects and thus enhancing the data set through a data augmentation process, the method significantly improves its classification capacity. This increase in data volume helps reduce errors and elevates the accuracy of project success predictions, making the system increasingly reliable in its assessments.

The rising interest of companies in machine learning implementation and the lack of research showing its effectiveness in terms of project portfolio selection for CIPs, led to the conceptualization of this research. Even though the case study revealed satisfactory classification results, some structural limits persist. The main one is that the data set turned out to be too restricted, with one of the success level classes not being adequately represented by the historical data. The paper suggests a data augmentation procedure to overcome this problem and to achieve better results in the classification task. A first improvement would be to broaden the range of historical projects used in the training set, ensuring that all the classes are well characterized and balanced. Furthermore, data was only collected from one company for the analysis, thereby limiting the generalizability of the results. We recognize that the use of a small sample of 28 projects and the reliance on linguistic evaluations represent key methodological limitations. These constraints may introduce bias and limit the external validity of the results. However, we explicitly designed our fuzzy-based data augmentation technique to mitigate these challenges by modeling the uncertainty inherent in expert judgments and expanding the training set in a controlled manner. Still, the classification performance remains sensitive to the distribution of success classes, especially in underrepresented categories (e.g. “Very High”). Future research should aim to validate the proposed method on larger and more diverse data sets, ideally sourced from multiple organizations and using quantitative project performance metrics where available. Another limitation is the lack of external validation. While the results are promising within the studied company, the method has not yet been tested on data sets from different organizational or industrial contexts. Future work should focus on testing the proposed approach in other companies to assess its transferability, robustness and sector-specific performance. A further consideration concerns model interpretability. While the MLP demonstrates high predictive accuracy, it functions as a black-box model, making it difficult for decision-makers to understand the reasoning behind its outputs. This can hinder adoption in organizational contexts where accountability and explainability are crucial. Future implementations could integrate model-agnostic interpretability tools such as SHAP or LIME to provide transparent explanations of how each CSF contributes to the prediction. This would increase user trust and enhance the applicability of machine learning in project governance. Finally a significant challenge is the computational load associated with the data augmentation process, which presents an opportunity for future enhancement. This is particularly relevant for potential implementations within companies.