An integrated methodology for startup assessment in Quality 4.0 using Thurstone’s Law of Comparative Judgment

Traditional evaluation methods for large corporations, such as financial statement analysis, discounted cash flow (DCF), market comparables (Comps), asset-based evaluation, and risk-adjusted return methods, face significant limitations when applied to startups. These methods often fail to account for the unique potential of startups, given their lack of reliable historical data and their operation in volatile and uncertain markets (Köseoğlu, 2023).

Recent data make the extent of the problem clear: about 90% of startups eventually fail, 10% within the first year and almost half by the fifth year. Even among startups financed by venture-capital funds, about 75% never return investors’ capital (Gompers and Kaplan, 2019), and mutual-fund mark-to-market data indicate that early-stage shares are on average overvalued by about 40% (Brown and Gredil, 2022). These figures illustrate both the economic stakes and the shortcomings of conventional valuation metrics, reinforcing the need for tailored assessment methods. Moreover, startups frequently innovate with new technologies and business models, which can escape the confines of traditional metrics (Armstrong, 2006).

In this context, there is a clear need for an assessment method that adapts to the rapid changes and evaluates startups based on both quantitative and qualitative potential. The principles of Quality 4.0 – emphasizing continuous improvement, integration of modern technology, data-driven decision-making, together with the synergy between human and system intelligence – offer a robust framework for this challenge. Quality 4.0 employs advanced data collection and analysis techniques to capture the dynamic and innovative essence of startups during assessment. Specifically, this study addresses the following research questions:

To address these research questions and evaluate startups, the methodology presented here utilizes Thurstone’s Law of Comparative Judgment (LCJ). LCJ is not just a comparative method; it is a dynamic, adaptive process that aligns seamlessly with the continuous-improvement ethos of Quality 4.0. LCJ’s statistical rigor allows to effectively discern relative perceptions, making it particularly suitable for environments where traditional metrics may be less effective. By enhancing traditional evaluation models with an early-stage flexible framework, this approach ensures the relevance of the startup evaluation process across various contexts and sectors. Figure 1 presents a conceptual scheme that summarizes the methodology’s key phases, which are examined in detail later in the article:

• identification of the startups;

• definition of the evaluation criteria;

• evaluation of startups by an expert team; and

• aggregation of the evaluations, so as to find the most promising and incentive-worthy startups.

The remainder of this paper is organized as follows: Section 2 provides an overview of state-of-the-art techniques for assessing startups, as alternatives to the proposed method. Section 3 proposes a general formulation of the startup assessment. Section 4 revisits Thurstone’s LCJ, outlining its basic assumptions and practical applications. Section 5, through a case study, shows how LCJ can be successfully applied to assess startups. The concluding section 6 summarizes the key findings, practical implications, limitations, and prospects for future development.

Assessing early-stage startups presents unique challenges due to limited data and high uncertainty, making traditional evaluations inherently subjective (Yildirim and Maz, 2025). Nevertheless, effective benchmarking of startups is crucial for stakeholders like investors, incubators, and accelerators, as it can yield significant economic returns and stimulate the development of new ideas (El Hanchi and Kerzazi, 2020).

Integrating Quality 4.0 principles into this context enhances the evaluation process with modern technology and data-driven insights, thereby improving decision-making, managing risk and strategically allocating resources. In the dynamic and competitive startup environment, where resources are limited and challenges are complex, accurate assessment can mean the difference between success and failure.

At the risk of oversimplifying, the scientific literature on early-stage startup evaluation can be schematized from three different angles, as illustrated below:

1. Traditional vs. modern methods. Literature extensively addresses methodologies for evaluating and prioritizing startups based on various criteria (Valiris et al., 2005). This area of research has evolved through contributions from numerous authors, each offering unique perspectives and innovations. Prominent scholars such as Blank and Dorf (2020), Kotter et al. (2015) have explored various aspects of startup evaluation, from validating business ideas to developing competitive strategies. However, many of these earlier approaches now require revisiting in light of today’s digital analytics capabilities. There is a growing need to update traditional evaluation models by harnessing big data, machine learning and real-time metrics (i.e. some of the pillars of Quality 4.0), to better predict long-term startup success. Indeed, recent studies have begun to integrate such tools – for example, Wei (2025) proposes a hybrid quantitative framework that uses entropy-based weighting and simulation to evaluate startups on strategic criteria like scalability, adaptability and risk exposure – but overall the literature is still evolving toward fully data-driven startup assessments.

2. Context-specific frameworks. A recurring theme in the literature is the adaptability of assessment methodologies to the diverse business needs and contexts in which evaluations are conducted. The challenge lies in evaluating a variety of startups, each with unique characteristics. Evaluation criteria vary according to business needs and market contexts, making prioritization a dynamic and personalized process (Galende, 2006). For companies conducting these assessments – be they investment firms, corporate innovation departments, or technology incubators – the goal is typically to identify startups that best align with their strategic objectives and values. This has led to the development of structured frameworks targeting context-specific performance indicators. For example, Arshi et al. (2021) introduce the SECURE model as a dedicated framework for measuring startup performance in emerging economies, underscoring the need to capture unique dimensions of startup success (such as scalability and resilience) that traditional models might overlook. Overall, literature emphasizes the importance of a multidimensional approach, considering not only financial and market aspects but also innovation, sustainability and social impact (Smith and Cordina, 2014; Manigart et al., 1997; Bottani and Rizzi, 2008).

3. MCDM and quantitative tools. To systematically compare startups on such multiple criteria, researchers have increasingly turned to multicriteria decision-making (MCDM) techniques and other structured analytical tools. Key analytical approaches discussed in the literature include:

   • Balanced scorecard combined with a multi-attribute rating technique, which has been adapted to startup decision contexts as a way to link strategic objectives with evaluation criteria (Valiris et al., 2005).

   • Analytic hierarchy process (AHP), which has been applied to startup investment decisions to derive weighted rankings of alternatives. For instance, Kyrylych and Povstenko (2023) demonstrate an AHP-based model that organizes evaluation criteria into thematic clusters and calculates a global priority for each startup, allowing investors to see which venture scores highest overall on a combination of quantitative and qualitative factors. Such an approach not only introduces more rigor into the comparison process but also makes the rationale behind decisions more transparent.

   • Fuzzy logic and linguistic models have gained traction to handle the uncertainty and vagueness in expert judgments. Lin et al. (2021) propose a hesitant fuzzy linguistic MCDM model for startup evaluation, enabling assessors to use linguistic terms (like “high” or “low” performance) which are then converted into numerical scores for comparison. This fuzzy approach accommodates the imprecise nature of evaluating novel ventures, where crisp data may be unavailable. In fact, by using tools like fuzzy sets or information envelopment analysis, these models can gauge a startup’s efficiency or potential even with scant historical data.

   • Hybrid data-drive frameworks combine multiple techniques to exploit their complementary strengths. As noted above, Wei (2025) integrates entropy-based weighting (to objectively determine criterion importance from data variability) with a simulation component (to test how robust startup evaluations are under different scenarios). By blending qualitative judgment with quantitative analysis, these hybrid frameworks aim to improve the accuracy of predictions and provide deeper insights into a startup’s potential under uncertainty.

Together, the incorporation of these analytical techniques is making startup assessments more evidence-based. An added benefit of quantitative scoring models (like AHP or scorecards) is the ability to compare and even portfolio-manage startup opportunities. Rather than a binary go/no-go decision on one venture, investors can evaluate a pool of startups and allocate resources proportionally to their scores – thus spreading risk according to each startup’s evaluated merit.

In summary, the state-of-the-art in startup assessment is moving toward integrated, multi-criteria frameworks that combine the strengths of expert judgment and advanced analytics. The literature underscores that effective startup evaluation must be flexible enough to account for different contexts and criteria, yet structured enough to allow systematic comparison. Traditional evaluation criteria (market size, financial projections, etc.) are now complemented by deeper analysis of qualitative factors (such as team competence, innovation capacity and adaptability to change) in what is increasingly a data-rich decision environment. However, despite notable progress, few existing methods fully capture the synergy between human expertise and cutting-edge analytical tools. This gap sets the stage for an approach that fuses expert comparative judgment (as exemplified by Thurstone’s LCJ) with the data-driven ethos of Quality 4.0 – an approach that this study seeks to develop and validate.

This section aims to define the multicriteria problem of startup evaluation, which is addressed and solved in the following sections. It is divided into two subsections corresponding, respectively, to (i) the definition of the evaluation criteria and (ii) the construction of the assessment matrix by a team of experts.

Building on insights from recent literature and professional practice – and consistent with the twin paradigms of Quality 4.0 and Industry 4.0 emphasized by Khourshed (2023), Khourshed and Gohar (2023) and Khourshed et al. (2023) – five key criteria for evaluation have been identified (cf. RQ1):

1. Team characteristics (CR₁). A strong, cohesive team builds investor confidence and positively influences the startup’s growth (Blank and Carmeli, 2021; Kotter et al., 2015). The quality of human capital and the ability to establish strong external relationships are crucial factors (Hatch and Dyer, 2004; Fernández-Olmos and Ramírez-Alesón, 2017).

2. Patents and market potential (CR₂). Patents reduce information asymmetry between the startup and external evaluators and the number of patents is positively correlated with the startup’s economic value (Jeon, 2019; Conti et al., 2013). Market potential involves the startup’s target market size, growth prospects and demand for its product or service.

3. Strategic alliances (CR₃). Strategic alliances provide access to resources and signal the quality of the startup. The ability to form such alliances is often valued by investors, especially those focused on financial returns (Mamédio et al., 2019).

4. Financial health and funding (CR₄). Financial stability, including cash position and investments in research and development, positively influences startup valuation, while high debt levels may have a negative effect (Sievers et al., 2013).

5. Innovation and technology (CR₅). This criterion evaluates the startup’s innovation level and technological advancements, including unique selling propositions, intellectual property and market disruption potential (Noh et al., 2018).

The proposed methodology – which will be described in detail in Section 5 – aims to provide a simple, user-friendly tool for startup evaluators while employing an agile model based on minimal assumptions and straightforward comparisons. This approach relies on statistical assumptions about expert preferences (Franceschini et al., 2022) and is easy to implement, even by non-specialists. The Thurstone’s LCJ – recalled in Section 4 – constitutes a well-established basic model for achieving the intended goal, offering an indirect measurement framework: expert evaluations are statistically transformed into global-scale values, a procedure generally referred to as scaling. Alternative scaling paradigms include the Rasch model (Rasch, 1966; Bond and Fox, 2007) and conjoint analysis (Luce and Tukey, 1964). However, LCJ is preferred here because it is conceptually straightforward, easier to communicate outside the psychometric field where it originated and is based on comparatively fewer restrictive assumptions (Pollitt, 2012).

This subsection presents a general formulation of the startup-assessment problem, which can be concisely captured by the assessment matrix in Figure 2, whose columns list the individual startups (S_j) and whose rows represent the evaluation criteria (CR_i). Each entry in the assessment matrix (⁠$rij$⁠) reflects the performance level of the jth startup (S_j) with respect to the ith criterion (CR_i). The evaluations are carried out collectively by a team of startup experts, with adequate documentary and/or direct knowledge of the startups under consideration. To guarantee assessments that are sufficiently comprehensive yet not overly cumbersome, the team should combine complementary competences while remaining relatively lean – typically no more than six to eight members (Franceschini and Rossetto, 1995).

The importance of each criterion varies depending on the evaluative context or strategic objectives; the decision regarding the weights can be made collectively by the expert team itself or established a priori by decision-makers (Franceschini et al., 2022). The performance of startups for each individual criterion is assessed on a four-level ordinal scale: “high,” “medium,” “low” or “null.” Nevertheless, more “powerful” measurement scales (interval or ratio) may be adopted, provided that the resulting values are later converted to an ordinal scale (Stevens, 1946; Franceschini and Maisano, 2019; Franceschini et al., 2022).

The next section shows how LCJ can aggregate these multiple-criteria evaluations into a comprehensive positioning of the startups under study.

Thurstone’s LCJ posits that it is easier for individual respondents to compare two alternatives side by side, rather than evaluating each one in isolation. This principle forms the foundation for various research methodologies that rely on direct comparisons between pairs of alternatives (Thurstone, 1927; Franceschini and Maisano, 2015; Franceschini et al., 2022). In this application, the classical LCJ is adapted to a real-world context in which a panel of experts (i.e. respondents) performs multiple-criteria evaluations of startups (i.e. alternatives) and these evaluations must be merged into an overall assessment (cf. Section 5).

The general problem addressed by Thurstone’s LCJ can be summarized as follows: a group of experts provide their assessments on one or more specific criteria of certain alternatives. Each criterion is defined as a specific attribute (i.e. a property or feature of interest) of the alternatives that elicits a response from the group of respondents. The individual assessments are combined according to the procedure summarized in the following steps:

• Collection of individual assessments: Respondents provide their evaluations based on specific criteria for the alternatives in question.

• Aggregation of assessments: The assessments related to the individual criteria are then aggregated to obtain a global measure for each alternative. This step often involves creating a proportion matrix that reflects the frequency with which one alternative is preferred over another (Thurstone, 1927; Edwards, 1957).

• Transformation to z-scores: The aggregated data is then transformed into z-scores, which are standard scores that indicate how many standard deviations an alternative is from the mean. This step normalizes the data, allowing for meaningful comparisons (Franceschini and Maisano, 2015; Franceschini et al., 2022).

• Scaling: Finally, the z-scores are used to determine the average values of the attributes of the alternatives, placing them on a “preference” scale that quantifies the positioning of the alternatives (Edwards, 1957; Kelly et al., 2022).

This methodology captures the complexity of respondent evaluations while facilitating the interpretation of the data through advanced statistical models. A more detailed presentation of the Thurstone’ LCJ model can be found in Edwards (1957) and in Franceschini et al. (2022).

The ability of the LCJ model to convert multiple-criteria evaluations into aggregated, quantifiable data makes it a powerful tool for analyzing and comparing various alternatives. It also offers a structured approach that enables detailed and statistically rigorous analyses, which can be especially valuable in fields like startup assessment and scouting.

This section outlines the application of Thurstone’s LCJ to the comparison of startups, focusing on its implementation in a real-world case study within a corporate environment. The assessment concerns eight startups of interest (S₁, […], S₈), which are the alternatives of the problem. These startups were drawn from the Piedmont’s aerospace–digital-manufacturing cluster, one of the Europe’s major hubs for Quality 4.0 technologies. Focusing on this innovation-intensive region provides a coherent economic context, while offering sectoral variety (e.g., additive manufacturing, artificial intelligence [AI]-enabled inspection, satellite Internet of Things, etc.). The sample therefore balances regional representativeness (all startups belong to the same policy ecosystem targeted by the funding partner) and technological diversity (capturing different early-stage challenges). The expert panel – which is better described in Section 5.1 – was composed to reflect this scope, combining venture-capital analysts, corporate R&D managers and a university technology-transfer officer, each with direct exposure to the cluster. This purposive sampling improves both contextual relevance and external validity.

The specific attribute considered is each startup’s performance level – interpreted as its development and success potential – for every criterion. The benchmarking analysis adopts the five criteria described in Section 2 (CR₁, […], CR₅). The weights collectively decided by the expert team correspond to evidence from large samples in entrepreneurship research. Innovation and technology (CR₅) received the highest relative importance (33%), echoing longitudinal studies that link patent-based technological advantage to superior growth and valuation (Conti et al., 2013; Noh et al., 2018). Team characteristics (CR₁) ranked second (22%), in line with venture-capital findings that management quality is the strongest predictor of funding success and survival (Manigart et al., 1997; Blank and Carmeli, 2021). The moderate weights for strategic alliances (CR₃) and financial health (CR₄) likewise mirror meta-analyses showing these factors to be supportive, though not dominant, predictors of early performance (Mamédio et al., 2019; Sievers et al., 2013). Reproducing these empirical patterns within a small-sample expert setting underscores the validity of the proposed criteria weights.

The startup evaluations are conducted by a group of experts whose experience in startup assessment is both deep and diverse. In this case, a panel of six experts with heterogeneous seniority and complementary backgrounds operated: two venture-capital analysts, one serial entrepreneur, one university technology-transfer officer and two industrial R&D managers. Prior to the large-scale data collection, a pilot test was conducted, to increase the reliability and clarity of the assessment criteria prior to their formal application ^[1].

The evaluation process begins with the construction of the assessment matrix shown in Figure 2. Each jth startup (columns of the table in Figure 2) is evaluated against each ith criterion (rows of the table in Figure 2) using an ordinal scale represented by the following symbols: “△” for “low,” “○” for “medium,” “●” for “high,” and “∅” for “null” degree of performance; Table 1 illustrates the four levels of this scale. Figure 3 exemplifies the assessment matrix that includes the evaluations assigned by the expert group. As an example, with reference to the team characteristics criterion (CR₁, cf. Section 2), the expert group decided to assign the highest value on the scale, “●,” to startup $S7$⁠. The intermediate value, “○,” was given to startups $S2$ and $S5$⁠, while a null evaluation, “∅,” was assigned to the remaining startups.

Content of the assessment matrix can be visualized graphically in the form of performance profiles, as exemplified in Figure 4.

The assessment matrix serves as the starting point for initiating the comparison process of startups. The evaluation criteria (CR_i) and their respective weights, along with the specific assessments of each startup by the expert group, form the foundation on which Thurstone’s LCJ methodology is built. The technique involves the following structured steps (a, b, c and d):

• Transformation of the assessment matrix into rankings

In the LCJ framework, the dummy alternatives are postulated to follow a normal distribution with an unknown mean and the same variance as the regular alternatives (Franceschini and Maisano, 2015; Franceschini et al., 2022). These dummy alternatives will be used to “anchor” the scaling derived from the LCJ process, as explained later:

• Transformation of rankings into paired-comparison relationships

• Thurstone’s LCJ application

In summary, for each (jkth) comparison between pairs of startups, there will be a number of pairwise comparisons equal to the number of criteria present in the assessment matrix. Figure 5(c) illustrates the construction of these relationships from the assessment matrix. For clarity, for each $CRi$ and each jkth paired comparison, it is useful to define three binary coefficients as follows:

It is evident that these coefficients are mutually exclusive, and they maintain a complementary relationship, as shown by the condition: $ci,jk>+ci,jk∼ +ci,jk< = 1$⁠.

A general coefficient that expresses the degree of preference of $Sj$ over $Sk$ from the perspective of the ith criterion can be defined as:

For symmetry, it follows that: $ci,jk=1-ci,kj$⁠. The computation of each single binary coefficient and the corresponding global values (⁠$ci,jk$⁠) for all the paired-comparison relationships is shown in Figure 6. As last step, the coefficients $ci,jk$ are aggregated in the following weighted sum, which determines the $pjk$ values (Maisano et al., 2024).

In adapting the LCJ (cf. Section 4), the $pjk$ proportions are calculated by aggregating prior comparisons through a weighted sum. Conceptually, this value represents the weighted fraction of criteria for which the startup $Sj$ has a greater influence than the startup $Sk$⁠. For example, for the paired comparison (⁠$S1, S3$⁠) (at the top of Figure 6), the corresponding value for $p13$ would result in: $p13 = 0.5·22%+0·20%+1·14%+0·11%+0·33 = 0.250$⁠.

The $pjk$ values can be aggregated into a P matrix of proportions, which – consistently with the LCJ – includes elements that are symmetrical with respect to the main diagonal and complementary to each other with respect to the unit ^[2]. Table 2(a) contains the P matrix that results from the paired-comparison relationships in Figure 5(c). Starting from now, the traditional LCJ (cf. Section 4) is applied, determining the Z matrix (see Table 2(b)) and, subsequently, the interval scaling (x) of the various startup alternatives (see Table 2(c)).

For benchmarking, the same expert data were re-processed with a conventional Weighted Borda’s Count (WBC) procedure (Franceschini et al., 2022). The LCJ ranking showed a high concordance with the WBC results (Pearson’s R² = 0.87), with both methods identifying startups S₄, S₁ and S₈ as the first, second and third most promising candidates, respectively:

• Scale anchoring

where:

x_∅ and x_● are the scale values of S_∅ and S_●, respectively, resulting from the LCJ;

$xk$ is the scale value of a generic $Sk$⁠, resulting from the LCJ; and

$yk$ is the relevant transformed scale value in the conventional range [0, 10].

The new scale (⁠$yk$⁠) has a conventional unit and an absolute zero-point (indicating the attribute’s absence); it is therefore classifiable as a ratio scale (Stevens, 1946; Franceschini et al., 2022). Although Thurstone’s LCJ computational processes may seem complex, they are actually straightforward and can be fully automated using standard spreadsheet software, such as MS Excel^® (Franceschini and Maisano, 2019). Figure 7 provides a graphical summary of the final (ratio) scaling, illustrating the startups’ overall performance.

This paper introduced a useful method that enables diverse stakeholders – venture capitalists, private-equity investors, accelerators and incubators, economic-development agencies, public funding bodies, universities, research institutes and crowdfunding platforms – to assess and position start-ups in line with Quality 4.0 principles. Those principles guided the selection of the key evaluation dimensions: team characteristics, patents and market potential, strategic alliances, financial health and funding, and innovation and technology (see RQ1). The methodology distinguishes itself from other support tools through its simplicity and the limited amount of information required from experts. The core of this methodology lies in the assessment matrix, where, for each analysis criterion, the team of experts is asked to determine the intensity of the relationship linking each alternative (startup) to the criterion in question, on a four-level ordinal scale. Based on the startups’ evaluations for each criterion, a ranking of the startups is established. From this ranking, pairwise comparisons are derived, leading to a comprehensive positioning of the evaluated startups. The steps following the creation of the assessment matrix can be fully automated on any software platform or spreadsheet (e.g. MS Excel^®). The methodology also allows for scenario analysis by varying the configuration of weights assigned to individual criteria, as well as sensitivity analyses by adjusting some of the weights and/or the intensity ratings of specific alternatives (startups).

One of the strengths of the proposed adaptation of Thurstone’s LCJ is that it asks experts to express their assessments with simple, practitioner-friendly response modes – chiefly ordinal scales or straightforward preference rankings (Franceschini et al., 2022). This is especially valuable when evaluating early-stage startups, for which detailed quantitative data are often unavailable. Once these basic assessments have been gathered, the probabilistic structure of LCJ transforms them into an interval scaling and – after scale anchoring – a 0–10 ratio scaling that helps to identify the most promising and incentive-worthy startup alternatives (cf. RQ2). This scaling may give the whole ecosystem a shared yard-stick for judging young ventures. For professional investors, it may work as a first-pass filter: only the companies that land in the upper band of the scale deserve time-consuming due-diligence, while the same scores, read in reverse, warn against premature enthusiasm. Policy makers can turn the numbers into transparent thresholds when distributing grants or tax breaks and, by tracking average scores across sectors, monitor the “health” of their start-up landscape almost in real time. Managers of accelerators and incubators can overlay each start-up’s radar plot on an internal skills audit, design coaching that tackles the most obvious gaps and repeat the assessment mid-program to check whether support is actually moving the needle. Universities and research centers gain a fair gatekeeping tool for choosing which spin-outs to back, while the history of scores accumulated across cohorts becomes a natural data set for studying what drives success. Even crowdfunding platforms can translate the underlying ratios into intuitive bronze/silver/gold badges, so that casual investors can gauge quality at a glance without having to wade through technical reports.

Because the assessments are produced collectively, consensually and collaboratively by the expert panel, the approach helps reduce the subjectivity that inevitably arises when decisions rely on sparse or unstructured information. Individual opinions are systematically fused into a group consensus, ensuring that the final evaluation embodies the collective wisdom of the panel, minimizes personal bias, and enhances the robustness of the assessment. This makes it easier to interpret the results and clearly identify which startups stand out.

The method is also flexible, versatile enough to accommodate emerging or less-structured criteria that may be difficult to quantify using traditional methods. Statistically speaking, the method is robust, since it provides a solid framework for analyzing preferences and choices, enabling deeper insights into the factors driving decisions. A relevant aspect is that it helps identify key factors that influence how a startup is perceived. These insights can be invaluable for refining the scouting process and making better decisions in the future. In short, the methodology not only makes the process more systematic but also helps ensure that decisions are well-informed and based on shared data.

While Thurstone’s LCJ offers many advantages for evaluating startups, there are some limitations to consider. First, because the empirical test was confined to eight startups from a single regional aerospace–digital-manufacturing cluster, the findings cannot be automatically generalized to other geographies or industries. Replication studies in service-oriented, bio-tech or emerging-market ecosystems are therefore required before broader claims can be made. Second, the five Quality 4.0-aligned criteria were optimized for technology-intensive ventures; sectors with different risk profiles (e.g. med-tech or social enterprises) may call for context-specific adaptations, such as adding regulatory readiness or social-impact indicators. Third, operational scalability remains a challenge: the complexity of the LCJ model grows quadratically with the number of startups, which could be onerous for accelerators screening hundreds of applicants (Franceschini et al., 2022). Potential remedies include hierarchical clustering to reduce the comparison set, algorithmic sampling of representative pairs and crowd-sourced or AI-assisted judgment platforms to distribute the evaluation workload.

Regarding the future, there is room to explore how cultural and contextual factors influence expert judgments. Understanding these influences could lead to more tailored and effective evaluation processes, making Thurstone’s method more adaptable to different regions or industries. In future research we will evaluate replacing LCJ with more advanced models that can handle incomplete or partial assessments – for instance, situations in which the expert panel lacks reliable information to score every startup on every criterion. A promising candidate is the ZM_II method proposed by Franceschini and Maisano (2019) for analogous multiple-criteria problems. We also plan to integrate AI-based techniques capable of assisting the expert team – for example, by mining historical data sets on comparable startups and surfacing statistically significant patterns (Morande et al., 2023). Such AI support could further mitigate the inherent subjectivity of expert judgment by supplying evidence-based, supplementary insights that enrich and stabilize the collective evaluation.

Research paradigm and epistemological stance: The study is grounded in a pragmatic approach, viewing a startup’s “quality” as a latent construct that can be analyzed through observable indicators and expert perceptions. A mixed-methods design is therefore adopted, integrating qualitative expert judgments with quantitative scaling via Thurstone’s LCJ. This approach aligns with design-science research principles and with the evidence-based ethos of Quality 4.0, which emphasizes continuous improvement through data-driven insights.

This research has been conducted within the framework of the “MUR DM 352/Leonardo-Theoretical and practical methods and tools for the management of a complex open innovation network” collaboration between Politecnico di Torino and Leonardo S.p.A.

The research protocol was approved under Politecnico di Torino’s Research Ethics Guidelines. No personally identifiable data were collected, ensuring minimal risk and full confidentiality for both experts and companies.