Threshold combinatorial multicriteria acceptability analysis for group decisions with subjective interpretations of objective measurements

Many decisions are based partly or wholly on objective measurements that describe the performance of the alternatives with respect to the criteria. At the same time, several common decision methods use subjective judgements to express the level of satisfaction provided by each alternative's objective performance. The decision-makers must then interpret the objective measurements as subjective satisfaction levels, because the relationship between the measurements and the value they provide may be non-linear (Salo and Hämäläinen, 1997). We will refer to these as OMSJ (objective measurement, subjective judgement) decisions. Noori et al. (2018) describe an OMSJ group decision in which a site for a dam is to be chosen. Objective measurements were provided to the decision-makers for criteria such as Cost, Reservoir Safety and Water Quality. The decision-makers then submitted subjective evaluations of this objective data using linguistic terms such as “medium” or “good”.

In multicriteria decision-making, judgements are usually made for criteria/alternative (c, a) combinations. When the subjective judgements are based on objective measurements, the potential for inconsistencies between them arises. With the new approach, these inconsistencies can be prevented. Decision-maker judgements are threshold values that separate the measurements into different satisfaction levels. Each threshold applies to all alternatives simultaneously. This approach can also reduce the number of judgements needed, because only one threshold judgement is needed for each criterion.

In group decisions, judgements and preferences vary between decision-makers. These discrepancies are caused by unshared mental models: individual group members hold different information or interpretations about a given issue. For example, in a hiring committee, one member, who is only aware of the general company policy, might judge that two years experience are sufficient, whereas another member, who is familiar with the particular challenges of the position to be filled, would set the “sufficient” threshold at four years. Hidden Profile studies, which were introduced by Stasser and Titus (1985), have shown that decision-makers’ ability to reach a correct, unanimous decision depends on how effectively they share their mental models. A strong consensus strengthens the decision-makers’ commitment to a decision and thus improves the chances of success for its implementation (de Vreede et al., 2013; Bragaw et al., 2024).

Combinatorial multicriteria acceptability analysis (CMAA) (Goers and Horton, 2022a) is a decision analysis and consensus-building framework that can be used in conjunction with any standard decision method. It is intended for cooperative and rational groups. In this context, a cooperative group is one whose members have a common objective, and “rational” means that each member will deliver the same evaluation, if given the same information. CMAA generates combinations of decision-maker judgements and preferences and employs the user-provided decision method to determine the preferred alternative for each combination. The resulting acceptability index is a measure of the amount of support for each alternative in the space of combinations. CMAA can be used both as a decision analysis tool and as a consensus-building guide. Its key advantages are the level of detail provided by the analysis and the small number of steps with which consensus can often be achieved.

This work contributes to multicriteria group decision-making in two ways. A new decision model for OMSJ decisions is proposed that is based on threshold judgements, and a variant of the CMAA group decision method for the threshold-based model is presented. Compared to methods that map objective measurements to subjective judgements individually, the new model eliminates the risk of inconsistencies between judgements. Compared to the standard CMAA method, the new approach generates a much smaller search space and a lighter cognitive load for the decision-makers. We call this variation of the CMAA framework “Threshold CMAA” or “CMAA-T”.

The remainder of the paper is organized as follows: In the next section, some background on OMSJ group decisions is provided. Section 3 describes the basic CMAA method, and Section 4 presents the new Threshold CMAA algorithm. Section 5 reviews a previously published case study from forest management and illustrates how it can be treated with CMAA-T. Section 6 shows the results of computational experiments that illustrate the performance of the CMAA-T method for various configurations of group decisions and for four previously published group decisions from different fields. The summary and conclusions are presented in Section 7.

Decisions are often based on objective measurements: for each criterion/alternative combination (c, a), a performance value has been researched before the decision is taken. Typical categories of objective data are financial (purchase price, running costs), technical (power consumption, mean time before failure [MTBF]) or environmental (CO₂ emissions saved, noise level).

Two approaches are available for dealing with objective measurements. In the first, the data is treated (perhaps implicitly) as a measure of value or utility, and the objective values are used as-is. Non-numerical values are first converted into numerical values. In this case, decision-makers are only needed to provide criteria preferences. This approach requires scaling of the criteria weights to ensure commensurability: What saving in running costs will offset an increase in noise level of 1 dB? Determining such weights can require a large amount of data and generate a high cognitive load (Velasquez and Hester, 2013; Riabacke et al., 2012; Corrente et al., 2024).

The second, much more common approach is the OMSJ model, which uses subjective estimates of the impact of each objective measurement. With this approach, it is not the objective measurements per se that are needed, but their impact on the decision in the opinion of the decision-makers. For example, in a hiring decision using Analytical Hierarchy Process (AHP) (Saaty, 2003), four years professional experience could be judged to be “significantly more attractive” than two years experience, and in an engineering decision with a linguistic/fuzzy model (Nikouei and Amiri, 2022; Ye, 2010), a power consumption of 35 kW for a machine might be “very poor” for one decision-maker, but “fair” for another. Such discrepancies reveal unshared mental models about the issue, which might be beneficial for the group to clarify. Also, a decision-maker might consider purchase prices of €10,000 and €15,000 for two machines to both correspond to the satisfaction level “very good”, if the other competing machines are all much more expensive. In this manner, measurement differences that the decision-makers consider to be inconsequential are eliminated.

In this section, we illustrate OMSJ-type decisions using three infrastructure examples from the literature.

Noori et al. (2018) describe a group decision in which a site for a dam is to be chosen. The decision consisted of 4 alternatives, 18 criteria and 4 decision-makers. Objective measurements were provided to the decision-makers for several criteria, such as Reservoir Safety and Water Quality. The decision-makers then submitted subjective (c, a) judgements such as “good” or “medium” for these measurements. For one particular dam site location, the four decision-makers submitted the values (“good”, “good”“, “medium” and “bad”) for each of these criteria. This illustrates how different decision-makers can assign different satisfaction levels to the same objective value.

Mastrocinque et al. (2020) evaluate the renewable energy sector in seven European countries. Eleven of the 14 criteria were objective measurements, which the decision-makers converted to subjective pairwise judgements. Table 1 shows the measured data for the criterion R&D governmental support for each country and the subjective pairwise comparisons relative to Germany. The meanings of the values 6, 7, 8 and 9 range from “strongly more preferred” to “extremely more preferred”. Thus, although the objective value for Belgium is more than three times larger than for Greece, Germany has the same subjective relationship to each.

Caceolu et al. (2022) consider the problem of selecting a site for offshore wind farms in northwest Turkey. Objective measurements for each location were provided for criteria such as offshore wind speed at 100 m and water depth. The decision method was AHP, so the decision-makers had to translate the objective measurements to subjective pairwise comparisons. For example, the average offshore wind speed of between 8.95 and 9.50 m/s for the location Bozcaada and between 7.10 and 7.95 m/s for the location Karabiga were translated into a pairwise comparison of 7, meaning that, in the opinion of the decision-makers, Bozcaada performs “very much more strongly” than Karabiga.

In OMSJ decisions, (c, a) judgements carry the risk of intra-decision-maker and inter-decision-maker inconsistencies. An intra-decision-maker inconsistency occurs when the subjective satisfaction levels g₁ and g₂ proposed by a decision-maker for alternatives a₁ and a₂ are not compatible with the respective objective measurements M₁ and M₂. Table 2 shows consistent (C) and inconsistent (I) combinations for a beneficial criterion. The relations ≺, ≡ and ≻ compare satisfaction levels, and <, = and $>$ compare objective performances. It is not inconsistent for a decision-maker to assign the same satisfaction level to different objective measurements, as in the purchase price example in the previous paragraph. However, it is inconsistent for a decision-maker to assign different satisfaction levels to two alternatives, if they both have the same objective performance.

In a group decision, an inter-decision-maker inconsistency occurs, when judgements g₁ and g₂ by decision-makers DM₁ and DM₂ with respect to alternatives a₁ and a₂ contradict each other. Table 3 shows the various consistent (C) and inconsistent (I) combinations. For example, it is inconsistent for one decision-maker to judge a ranking position of 1 to be “good” and a ranking position 5 to be “fair”, while another decision-maker judges ranking position 1 to be “fair” and position 5 to be “good”. (If the ranking measures a quality that is unambiguously beneficial or detrimental, then one of the decision-makers is also committing an intra-decision-maker inconsistency.) Inter-decision-maker inconsistencies of the type shown in Table 3 are present at several locations in the data presented by Noori et al. (2018).

Both types of inconsistency can be prevented, if subjective judgements are made for threshold values instead of for (c, a) pairs. A threshold is a value that separates two satisfaction levels. For example, in a hiring decision, a committee member might judge that the threshold that separates “insufficient” professional experience from “satisfactory” should be two years. This threshold is then valid for all applicants.

In addition to preventing inconsistencies, threshold judgements generate a lower cognitive load than (c, a) judgements, since only one judgement is needed for each criterion instead of one for each alternative. The larger the number of alternatives in the decision, the greater this advantage becomes.

CMAA is not a standalone decision method; it is a framework that can be used in conjunction with a variety of decision methods. Goers and Horton (2022a) illustrate its use together with Simple Additive Weighting, Weighted Product Method (WPM), Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and the lexicographic method. CMAA allows standard (single-user) decision methods to be applied in group decisions, providing detailed analysis and a consensus-building process. CMAA was designed for cooperative, rational decision groups and assumes subjective criteria preferences and/or performance judgements.

CMAA is based on the concept of acceptability, a measure of the amount of support for each alternative that is present in the input. Acceptability was introduced with the stochastic multicriteria acceptability analysis (SMAA) method (Lahdelma and Salminen, 2006; Menou et al., 2022). SMAA was developed for decision situations in which preferences are missing or incomplete, and both preferences and criteria measurements are imprecise or uncertain. In the group context, differing decision-maker preferences are fused into a single probability distribution function.

By contrast, CMAA was designed specifically for decision analysis and consensus-building in group decisions. Since fusing decision-maker inputs into a single mathematical entity would cause their individual identities to be lost, they are retained separately. All user judgements are collected in an aggregated judgement matrix, and all preferences are collected in an aggregated preference vector. An aggregate in CMAA is the set of its constituent values, rather than their average or union. Where judgements or preferences differ on a particular issue, we have a discrepancy. Discrepancies are caused by differing mental models held by the decision-makers, and cooperative, rational groups can usually resolve them by sharing their mental models in a discussion.

In order to compute the acceptabilities for each alternative, the CMAA analysis algorithm generates all possible combinations of judgements and preferences, called instances. If the instance space is very large, Monte Carlo simulation can be used instead of complete enumeration. Each instance is passed to a decision algorithm, which returns the most-preferred alternative(s) for that instance. The acceptability of an alternative is the proportion of instances for which it was most preferred; in other words, the acceptability measures the amount of support within the instance space for each alternative. The current judgement and preference acceptabilities measure the contribution of each individual judgement or preference to an alternative's current acceptability, and the potential judgement and preference acceptabilities are the acceptabilities that would be achieved when each judgement or preference discrepancy is resolved.

CMAA consensus-building is a deterministic search in the instance space. The search is guided by the information entropy of the normalized potential judgement and preference acceptabilities. The entropy of a probability vector measures the degree of separation of its individual elements (Shannon, 1948). CMAA determines the judgement or preference resolution that will result in the greatest reduction in the entropy of the acceptabilities at each step. We will refer to this as the “EO” (entropy-optimizing) resolution. The decision-makers consider the discrepancy containing the EO resolution in a clarification conference – a discussion in which decision-makers share their mental models of the issue and resolve the discrepancy to one of its constituent inputs or to a compromise value. Consensus is achieved when the acceptabilities (and therefore also the entropy) no longer change. Usually, the entropy will become 0, meaning that a single alternative has attained an acceptability of 1 and is thus uniquely preferred by the group. Occasionally, however, the process may converge to a state in which multiple acceptability indices – and therefore also the entropy – are greater than 0. This is caused by instances that return multiple preferred alternatives. Since the decision-makers will not always agree on the EO resolution at each step, the number of clarifications needed in practice will be higher than in the EO case.

The advantage of the combinatorial approach is that the effect of each individual input can be computed – both on the current state of the decision and also on potential future states that will be reached when the decision-makers resolve a discrepancy. When these resolutions succeed, consensus can often be achieved with a small number of iteration steps, and the majority of the outstanding discrepancies can be ignored. Görs (2024) showed that CMAA can achieve consensus significantly faster and more reliably than a conventional consensus-building approach. Furthermore, no pressure is exerted on the decision-makers to modify their inputs to conform to a computed trajectory, as is the case with many consensus-building schemes (Zhang et al., 2018; Dong and Xu, 2016; Guo et al., 2023; Yang et al., 2024; Li et al., 2023; Zhang and Li, 2023).

CMAA distinguishes three types of discrepancy (Goers and Horton, 2022b), which are helpful for understanding the consensus-building mechanism. Active discrepancies are those for which different resolutions lead to different acceptabilities and which may become the subject of a clarification conference. Inactive discrepancies are those whose resolution can no longer affect the acceptabilities and can therefore safely be ignored. In general, resolving an active discrepancy converts one or more discrepancies from active to inactive status. Pivot discrepancies are active discrepancies containing a resolution that would lead to an acceptability of 0 for one or more alternatives, thereby eliminating them from the decision.

In the following, a group decision problem consists of m alternatives a_i, n criteria c_j and d decision-makers DM₁ … DM_d. Criteria preferences are represented by μ, and judgements by λ.

We denote the matrix of objective performance measurements by M_n×m and the corresponding matrix of subjective satisfaction levels by G_n×m. The number of satisfaction levels available is Z + 1. A (threshold) judgement λ(j) for criterion c_j is a vector λ(j) = [t_j,1, …, t_j,Z] with the units of c_j. Each t_j,z defines the boundary between satisfaction levels z and z + 1 for criterion c_j. We adopt the convention that satisfaction level z + 1 is more valuable than level z. In practice, the levels will have semantics such as “Very Poor” or “Poor”, rather than just numerical identifiers. For beneficial criteria, [t_j,z ≼ t_j,z+1] will hold, while for cost criteria, we will have [t_j,z+1 ≼ t_j,z]. The CMAA algorithm does not impose a limit on Z + 1; in practice, the number will be determined by the decision model chosen. A typical value is Z + 1 = 5 for the commonly used five-point Likert scale.

For a beneficial criterion c_j, the satisfaction matrix G is defined as follows:

For non-beneficial criteria, the relations are reversed accordingly.

Essentially, the m × n matrix of objective measurements has been replaced by a matrix of satisfaction levels, where the mapping from measurements to satisfaction levels is defined by the threshold values that separate one level from the next. Section 5 contains an example of an OMSJ decision with all the variables.

The key difference between CMAA and CMAA-T lies in the structure and treatment of the judgements. When used in conjunction with a standard decision method, each judgement in CMAA is a performance estimate for a (c, a) pair, of which there are n ⋅ m in total. In the case of CMAA-T, each decision-maker submits only one judgement for each criterion. However, each individual judgement contains a larger amount of information: it consists of a set of threshold values that map the objective measurements to satisfaction levels.

In standard CMAA, all judgements are collected in the aggregated judgement matrix A. Since, in the case of CMAA-T, A has dimensions n × 1, we will refer to it as the aggregated judgement vector. Each element A_j is the set of individual decision-maker judgements λ_k(j) for criterion c_j:

where the index k is local to each criterion. Whenever two decision-makers submit different threshold judgements λ_k1(j) ≠ λ_k2(j), we have a judgement discrepancy. If all decision-makers submit the same judgement for criterion c_j, the cardinality of A_j is 1; if they all submit different judgements, the cardinality is d.

A judgement instance A* is obtained by selecting one judgement for each criterion from A. A judgement instance can be used to generate a satisfaction matrix G according to Eqn. (1). An example of a judgement instance is shown in Table 6. The number of judgement instances generated by A is denoted by ‖A‖.

Preferences in CMAA-T are treated in the same way as in standard CMAA. The aggregated preference vector P contains all submitted preferences, whereby each element P_j of the vector is the set of preferences submitted for the corresponding criterion:

where the index k is local to each criterion. When two or more preferences for a particular criterion differ, we have a preference discrepancy. A preference instance P* is obtained by selecting one preference for each criterion from P. The number of preference instances generated by P is denoted by ‖P‖.

The combinatorial decision problem is denoted by [P; A], and each instance I = [P*; A*] of the problem is a combination of a preference instance and a judgement instance. The total number of instances K is given by

An instance [P*; A*] can be converted to a format that is solvable by a standard (single-user) decision method by generating the satisfaction matrix G according to Eqn. (1). The assembly of the satisfaction matrix G to obtain a solvable decision problem [P*; G] is illustrated by Tables 4, 6 and 7.

The CMAA algorithm requires three sets of counter variables B, Q and S (Goers and Horton, 2022a). The counter $Bi1$ is incremented each time the associated decision method determines that a_i is the preferred alternative of an instance. When all instances have been processed, the (rank 1) acceptability for alternative a_i is given by

The counter variable Q_i must be redefined in CMAA-T to conform to threshold judgements. It is the number of instances in which alternative a_i is preferred when threshold judgement λ_k(j) was valid in that instance:

and the current judgement acceptability q_i is the corresponding fraction:

The potential judgement acceptabilities $qˆ(λk(j))$ for alternative a_i are those that are obtained when the discrepancy at criterion c_j is resolved to λ_k(j):

where $Kˆ$ is the number of instances remaining after the discrepancy has been resolved.

All other aspects of the CMAA-T algorithm correspond to the standard CMAA method. This includes the counter variable S for the preference acceptabilities, the computation of the judgement and preference entropies and the consensus-building iteration (Goers and Horton, 2022a).

The CMAA-T initial analysis algorithm is shown in Algorithm 1. A decision algorithm for solving each instance is provided. The inputs are the objective measurements M, the threshold judgements λ_k(j) and the criteria preferences μ_k(j).

First, the aggregated judgement and preference vectors A and P are generated from the decision-maker inputs λ and μ. Then, the algorithm loops through every instance I generated from combinations of instances of A and P. If Monte Carlo simulation is being used instead of complete enumeration, line 2 is replaced by.

2 for K randomly sampled instances I = [P*; A*] do.

We have found that K = 10, 000 usually provides sufficient accuracy.

The first task within the loop is to construct the satisfaction matrix G from the objective measurements M and the threshold judgements of the judgement instance A*. The satisfaction matrix G and the preference instance P* are passed to the decision algorithm, which determines the winning alternative(s), and the counters B, Q and S are updated accordingly. After completion of the loop, the acceptabilities, judgement acceptabilities and preference acceptabilities are computed from the counters.

The consensus-building algorithm (Algorithm 2 in Goers and Horton (2022a)) remains unchanged. However, the facilitation instructions for the clarification conferences have to be modified for threshold judgements. In the standard case, the clarification refers to a discrepancy in a performance estimation for a (c, a) pair. In the threshold case, the instruction to the decision-maker group is of the following form:

We illustrate the behaviour of the CMAA-T algorithm using data from a group decision problem from forest management (Laukkanen et al., 2002; Kangas and Kangas, 2003; Kangas et al., 2006). An area of forest in Finland was owned by three stakeholders, each of whom had different priorities. The decision consisted of 20 different proposals P1, …, P20 for managing the forest, and 5 criteria for which preferences had been agreed. In descending order of priority, the criteria were c₁: the present value of income from timber production, measured in thousands of Finnish Marks (TFIM), c₂: biodiversity, given as an expert ranking and as cardinal values, c₃: the estimated value of timber production after 20 years (TFIM), c₄: scenic beauty, also given as an expert ranking and as cardinal values, and c₅: the blueberry yield, with units kilograms per hectare (kg/ha) (Kangas and Kangas, 2003; Laukkanen et al., 2002; Kangas et al., 2006).

The objective measurements for the 20 alternatives with respect to the 5 criteria are shown in Table 4. We have abbreviated and rounded some values for clarity. These modifications do not impact the decision result. Note that alternative P2 has the strongest performance in four of the five criteria. The resulting decision problem was solved using SMAA-2 by Kangas et al. (2006), using Multicriteria Approval, AHP and PROMETHEE by Laukkanen et al. (2002) and using SMAA-O and Multicriteria Approval by Kangas and Kangas (2003). In the case of SMAA, normal probability distributions were superimposed over the measurements in order to model uncertainty.

In order to make this decision problem amenable to treatment with CMAA-T, we modelled preferences using five linguistic levels “Very Low”, “Low”, “Medium”, “High” and “Very High”. We used threshold judgements of type [t₁, t₂] to determine three satisfaction levels “Poor”, “Medium” and “Good” for the objective measurements.

Table 5 shows an example of an aggregated judgement vector A. Each element of the vector is a set of judgements {λ₁, λ₂, λ₃} by the three decision-makers. Note that the threshold judgement values were generated by us to represent decision-makers with different satisfaction levels and were not part of the original problem description. Since the preferences were unanimous, the preference vector P contains no discrepancies and simply reflects the ordinal values given in the case study: (c₁ = VH, c₂ = H, c₃ = M, c₄ = L, c₅ = VL). The number of judgement instances is ‖A‖ = 3⁵ = 243, and the number of preference instances is ‖P‖ = 1⁵ = 1. The overall number of instances is therefore K = 243 ⋅ 1 = 243, according to Eqn. (2).

Table 6 shows an example of a judgement instance A*, obtained by randomly selecting a judgement for each criterion from the aggregated vector A. From this instance, a decision matrix can be generated by applying the thresholds to the measurements to determine a satisfaction level P, M or G for each (c_j, a_i) pair. Table 7 shows a portion of the satisfaction matrix that is generated by the problem instance in Table 6.

The decision method used was fuzzy simple additive weighting (FSAW). Triangular fuzzy numbers were used, because they are widely used to model linguistic information (Yeh and Chang, 2009Yeh and Chang, 2009). Table 8 shows the fuzzy representations of the satisfaction levels and preferences that were used. The resulting fuzzy decision matrix and preference vector were passed to the FSAW algorithm to determine the preferred alternative. Defuzzification was performed with simple averaging: (l + m + u)/3, where l, m and u refer to the lower, middle and upper parameters of the triangular number, respectively.

Table 9 shows the acceptabilities computed by Algorithm 1 using all 243 instances that are generated by the example judgements of Table 6. Note that $∑bi1>1$⁠, because several instances produced multiple winning alternatives. This is due to the fact that the diversity of values is low: the 20 different objective measurements for each criterion are mapped to only 3 linguistic values “Poor”, “Medium” and “Good”, which increases the chances that multiple alternatives achieve the same overall score in the FSAW method.

For this set of judgements, alternative P2 has the most support by a wide margin. Several alternatives have no support, meaning that there is no combination of discrepancy resolutions that will make them the preferred alternative. The entropy of these acceptabilities after normalization to a probability vector is 2.760. The maximum possible entropy for this problem is 4.32, which would be the case if all acceptabilities had the same value. A comparison of these values with the acceptabilities obtained by Kangas et al. (2006) or the ranking positions of Laukkanen et al. (2002) is not meaningful, because they are based on a single set of arbitrary threshold judgements. A reliable comparison can be found in Section 6.4.

Table 10 shows the current and potential initial judgement acceptabilities for alternative P1. Analogous data (not shown) exists for the other 19 alternatives. The discrepancies at criterion “Biodiversity” and “Future value” are pivots, since this alternative receives no support from λ₁ for the former and receives no support from λ₂ or λ₃ for the latter criterion.

It is easy to see that the λ₂ and λ₃ resolutions will eliminate alternative P1. Both resolutions result in a satisfaction level of “Medium” for P1 for the criterion “Future value”. Since P2 has a satisfaction level of “Good” for this criterion and all other levels are equal, P2 will always outrank P1.

The situation for the λ₁ resolution for “Biodiversity” is more subtle; it is not immediately obvious why it should lead to the elimination of P1. Because thresholds apply to all alternatives simultaneously, certain combinations of satisfaction levels that are present in the aggregated judgements cannot occur in any problem instance. The λ₁ resolution itself leads to a satisfaction level of “Good” for P1 and “Medium” for P12, while the choices λ₂ and λ₃ for the criterion “Future value” both result in satisfaction levels “Medium” for P1 and “Poor” for P12. This is sufficient for the fuzzy computations in the FSAW method to produce higher scores for P12 than for P1 in every case.

Resolving the discrepancies in any of these inputs would eliminate P1 from the decision, irrespective of the outcome of any other discrepancy resolutions. Otherwise, the sensitivity to the judgements is low, especially for the blueberry yield, where all three judgements contribute equal support. The potential acceptabilities show that every resolution would reduce the acceptability of P1 (from 0.198) to varying degrees with the exception of λ₁ for the criterion “Future value”, which would increase it to 0.259. Across all 20 alternatives, a total of 42 judgement pivots are present.

Table 11 shows the initial judgement entropies computed from the potential judgement acceptabilities of the 20 alternatives. (There are no preference entropies in this decision, because there were no preference discrepancies.) The smallest value is 1.463, which represents a substantial improvement over the initial value of 2.76. It would be achieved if the criterion “Future value” were resolved to judgement λ₃. Therefore, in the consensus-building process, the first clarification conference should attempt to resolve the threshold proposals for this criterion. Resolution to λ₂ would also improve the separation of the acceptabilities, but to a lesser degree. If, however, the decision-makers agreed on λ₁, the separation would deteriorate from 2.76 to 2.931.

Table 12 shows the consensus path that is obtained by selecting the EO resolution at each step. At step 1, the discrepancy to be resolved was determined by the smallest value in Table 11, as previously described, and the resulting entropy had the predicted value of 1.463. In step 2, the EO resolution was λ₂ for the “Biodiversity” criterion, which resulted in consensus.

The preferred alternative at consensus is P2. This is perhaps unsurprising, since it achieved the highest performance in four out of the five criteria (see Table 4). The judgement discrepancies for the criteria “Future value” and “Biodiversity” have been resolved to λ₃ and λ₂, respectively. ‖A‖ has been reduced to 3³ = 27, and the remaining three discrepancies are inactive: even though they are unresolved, they nevertheless have no effect on the outcome of the decision and can be ignored. If CMAA-T had been used in the original case study, the decision-makers would only have needed to agree on two threshold issues in order to achieve consensus on the plan for their forest.

A comparison of Tables 4 and 5 demonstrates the reduction in cognitive load that the threshold model can offer. Using any conventional (i.e. non-threshold) decision model, each decision-maker would have to provide performance evaluations for each entry in Table 4. By contrast, the threshold method only requires input corresponding to one λ column of Table 5 – only 10 numerical values instead of 100.

This consensus path was generated using EO resolutions, which is likely to require a lower-than-average number of steps. In Section 6.3, we explore the effect of the choice of resolution on the number of steps needed in the consensus path.

In this section, we study the effect of various parameters on the performance of the threshold method, and we apply it to data from four previously published group decision problems.

We performed a Monte Carlo simulation experiment to study the effect of the number of thresholds z in each judgement and the number of decision-makers d on the length of the consensus path. The objective measurements, preferences and fuzzy parameters from the forest planning problem of the previous section were used.

The simulation was used with 1,000 sets of random starting judgements. The parameter z ranged from 1 to 4. The larger the value of z, the more diversity among the judgements is present: z = 1 corresponds to a pass/fail judgement, and z = 4 corresponds to the typical five-point Likert scale. The number of decision-makers d ranged from two to ten. The larger the number of decision-makers, the greater the number of different judgements λ submitted for each criterion is likely to become. The EO resolution was chosen at each step of the consensus path.

Figure 1 shows the results of the experiment. With the exception of d = 2, the mean path lengths (solid lines, left vertical axis) are essentially independent of d. The mean path lengths become slightly shorter with increasing z. We conclude that a large number of decision-makers or satisfaction levels will not restrict the efficiency of the method.

Also shown in Figure 1 (dashed lines, right vertical axis) is the effect of the number of thresholds z on the multiplicity of most-preferred alternatives. For z = 1, a small number of decision-makers can result in multiple winners at the conclusion of the consensus process. Otherwise, the probability of multiple most-preferred alternatives is very low.

In order to study the effect of n and m on the consensus path length, we conducted Monte Carlo simulations for various problem sizes.

We considered decisions with dimensions 4 ≤ n ≤ 16 and 4 ≤ m ≤ 20. For each (m, n) combination, 1,000 decision problems were generated using random measurements, random [t₁, t₂] judgements and (VL, L, M, H, VH) preferences. 10,000 Monte Carlo samples were used to compute each set of acceptabilities, and entropy-optimal resolutions were chosen at each consensus step. Judgements and preferences were fuzzified according to Table 8. The computation time on a notebook computer was less than one second.

Figure 2 shows the mean path lengths computed by the simulation. The dependency on both m and n is very weak: the path lengths grow slowly with m and appear to converge with increasing n. This suggests that the threshold algorithm will be efficient even for large decision problems.

In the previous sections, EO resolutions were chosen at each step of the consensus-building process. This is unlikely to happen in practice, since there is no reason to assume that the decision-makers will agree on the judgement that minimizes entropy at each step. We therefore studied the consensus path lengths when random resolutions are selected using four previously published case studies.

The independent variable was 0 ≤ P(EO) $≤1$⁠, the probability of choosing the EO resolution at each step. For each value of P(EO), 1,000 decision problems were generated by randomly selecting judgements and preferences from the aggregated vectors. Monte Carlo simulation with 10,000 independent replications was used to determine the acceptabilities at each consensus step for the wind turbine selection problem, and full enumeration of the instances was used for the Forest planning, Timber harvesting and Composite drilling problems.

The decision method used was FSAW with triangular fuzzy numbers as shown in Table 8, except where the case studies specified crisp preferences. The judgements were synthesized randomly for each decision-maker.

The results of the simulation are shown in Table 13. The column “j + p” indicates the number of judgement and preference discrepancies in each problem; the sum of the two values is therefore the maximum possible length of the consensus path. The mean number of consensus steps over the 1,000 random sample inputs varies from 1.4 in the best case in the Drilling problem to 7.8 in the worst case for the Timber harvesting problem. All four decisions result in close to linear relationships. If CMAA-T had been used to establish consensus in each of the original decisions, in most cases, only a handful of clarification steps would have been needed.

For the mean path length experiment, the objective measurements of Table 4 were used. Since preferences had been agreed a priori for this problem by the three decision-makers, only judgement discrepancies needed to be considered. Table 13 (“Forest planning”) shows the mean path lengths. The two steps to consensus that were needed in Table 12 were slightly better than the expected EO value of 2.3. In the average worst case, 4.1 resolution steps would have been needed.

Laukkanen et al. (2005) present a problem in which a group of three shareholders had to decide on a method for harvesting timber in an area of forest in Finland. Thirty different alternatives A1 … A30 were under consideration, and five criteria were applied. Criteria measurements and decision-maker preferences were provided. The decision method used was a modified Multicriteria Approval method.

We mapped the ordinal preferences to (VL, L, M, H, VH) fuzzy values. The result of the simulation is shown in Table 13 (“Timber harvesting”). The mean path length in the entropy-optimal case P(EO) = 1 was 3.6. In the worst case, P(EO) = 0, the mean path length was 7.8.

Bagočius et al. (2014) describe the problem of selecting a wind turbine for an offshore wind farm off the coast of Lithuania. Five criteria were applied: nominal power of the wind turbine, max power generated in the area, the amount of energy per year generated in the area, investments and CO₂ emissions. Four different turbines were under consideration: Nordex, Vestas, GE and REpower, and their objective performance with respect to each of the criteria was given. Ten decision-makers were involved; their criteria preferences were given as crisp numbers. The arithmetic mean of the resulting criteria weights was used in the computation. In the original study, the problem was solved using the Weighted Aggregated Sum Product Assessment (WASPAS) method (Zavadskas et al., 2012).

The result of the simulation is shown in Table 13 (“Wind turbine”). The mean path length in the entropy-optimal case P(EO) = 1 was 3.5, and in the worst case P(EO) = 0, the mean path length was 7.4.

Osmond et al. (2021) describe the problem of selecting settings for drilling holes in a composite fibre material. The eight alternatives A1 … A8 were different combinations of settings of the drill, and the three criteria were concerned with the quality of the drilled hole. Three decision-makers participated in the decision. Objective measurements were given for each (c, a) pair, as well as the crisp criteria preferences of each decision-maker. Preferences were not aggregated; instead, the decision problem was solved for each set of individual decision-maker preferences separately. Furthermore, the decision problem was solved using three different methods: TOPSIS, Weighted Sum Method (WSM) and WPM.

The result of the CMAA-T simulation is shown in Table 13 (“Textile drilling”). The mean path length in the entropy-optimal case P(EO) = 1 was 1.4, and in the worst case P(EO) = 0, it was 2.7.

In a second experiment, we compared the result of each of the four published decisions from the previous section with the result of the initial CMAA-T acceptability analysis. Using the same problem specifications, we ranked the alternatives based on the mean initial acceptabilities using 1,000 sets of random threshold judgements and preferences, where appropriate, in each case.

For all 20 alternatives of the forest planning problem, Kangas et al. (2006) provided rank acceptabilities obtained using the SMAA-2 method, and Laukkanen et al. (2002) gave ranking positions using PROMETHEE. Figure 3 shows scatterplots that compare the rankings obtained by CMAA-T and PROMETHEE and by CMAA-T and SMAA-2. The correlation between each pair of rankings is very strong (Spearman coefficient ρ = 0.941 and ρ = 0.944, respectively).

Table 14 shows the results given in each publication and the corresponding result obtained using CMAA-T. For the composite drilling problem, only the winning alternative A6 was given (Osmond et al., 2021), which was identical for the WSM, WPM and TOPSIS methods. CMAA-T produced the same preferred alternative. In the case of the wind turbine selection, a complete ranking of all four alternatives obtained using the WASPAS method was given (Bagočius et al., 2014). For this problem, CMAA-T produced the same most- and least-preferred alternatives A_D and A_A, but reversed the order of the second and third-ranked alternatives. For the timber felling problem, the top-two alternatives obtained using a modified Multicriteria Approval method were named as A8 and A28 by Laukkanen et al. (2005). CMAA-T reproduced the most preferred alternative, but not the second-ranked choice.

The CMAA-T method is very different in its approach and structure from other decision methods. In particular, it reduces the diversity of the judgements for each criterion from m, the number of alternatives, to z + 1, the number of satisfaction levels. For the timber felling problem, this was a reduction from 30 different values down to only 3. Furthermore, in order to perform the comparisons, we had to synthesize threshold judgements, which were not part of the original case studies, and then average out their effects using Monte Carlo simulation.

Despite these stark differences, for each of the four case studies cited, the initial CMAA-T acceptability analysis produced the same most-preferred alternative as the original source. This increases both confidence in the results obtained in each case study and also in the viability of the threshold-based acceptability approach.

Furthermore, the new approach has the added advantage of providing a consensus-building process. In the wind turbine selection, the experts' criteria preferences varied by factors of up to 4.5; in such cases, the use of averaged values may conceal valuable information that is held by a minority of decision-makers. The CMAA consensus-building process would have prompted the decision-makers to share their mental models and thereby potentially strengthened their commitment to their decision result.

CMAA is a framework for decision-making by cooperative, rational groups. It is used in conjunction with a standard decision method to provide detailed analysis and a consensus process. Our objective in this paper was to develop a variant of CMAA for a class of decision problems in which objective measurements are provided, but the decision matrix consists of subjective judgements.

We proposed a new decision model for this type of decision in which the subjective judgements are thresholds that separate satisfaction levels, rather than performance estimates for individual criterion/alternative pairs. This leads to a coarsening of the granularity of the decision matrix, since the number of satisfaction levels will in general be lower than the number of distinct measurements.

The CMAA framework can be adapted to the threshold-based approach quite easily, generating a much smaller search space than standard CMAA. This results in a faster algorithm and a lower cognitive load for the decision-makers.

The initial CMAA-T analysis reproduced the most-preferred alternative from four previously published case studies that employed a variety of decision methods. Simulation experiments showed that the number of steps needed to reach consensus is only weakly dependent on the size of the decision problem, the number of decision-makers and the number of satisfaction levels. We conclude that CMAA-T can be an efficient and robust framework for group decisions for its particular class of decision problems.

Many decisions contain both objective measurements and subjective judgements. In the future, a hybrid CMAA could be developed that treats the objective criteria with threshold judgements and the subjective criteria with the usual judgements for criterion/alternative pairs.

Objective measurements may have an associated uncertainty due to imperfect measurements or because the measured quantities are intrinsically stochastic in nature. These would be represented by intervals around each measurement or by probability distribution functions. The CMAA-T may need modification in order to handle these situations.

Finally, a threshold-based sensitivity analysis could be developed. This analysis would compute the stability of the decision when the placement of the thresholds is varied. The sensitivity would be considered to be high, if shifting a threshold from one measurement to its neighbour resulted in a change of the preferred alternative or of the overall ranking.