Investigation of preference-based speech quality assessment by integrating adaptive pair selection and pseudo-labeling

Speech quality is commonly evaluated using subjective listening tests based on direct assessment (DA) scores, in which a group of listeners rate the perceived quality of speech samples. The final quality score of a speech sample is typically obtained by aggregating these scores into a mean opinion score (MOS). For speech processing systems, performance is further evaluated by averaging the MOS across all generated samples. However, ensuring the robustness of these subjective metrics requires a substantial number of listeners, making DA-based evaluation inherently resource-intensive and time-consuming.

To reduce the cost and time required for subjective evaluation, speech quality assessment (SQA) aims to automatically evaluate or rank the quality of speech samples or speech processing systems. In recent years, driven by the broader success of artificial intelligence and deep learning with state-of-the-art methods across various domains (Naoumi et al., 2026; Delamou et al., 2023), data-driven approaches have gained increasing attention. In these approaches, machine learning models are trained to predict perceived speech quality directly from speech signals, using human-annotated scores (such as DA scores or MOS) as ground truth (Mittag et al., 2021; Huang et al., 2022; Lo et al., 2019). Thanks to the availability of large-scale public datasets from various domains, such as voice conversion (Toda et al., 2016; Lorenzo-Trueba et al., 2018; Yi et al., 2020; Huang et al., 2022; Cooper et al., 2023; Huang et al., 2024), simulated and real noisy speech (Mittag et al., 2021), noisy and denoised speech (Hu and Loizou, 2007), text-to-speech systems (Black and Tokuda, 2005; King and Karaiskos, 2013; Perrotin et al., 2023; Maniati et al., 2022), and synthesized singing voice (Cooper et al., 2023; Huang et al., 2023; Tang et al., 2024), the research on SQA using DA scores for training and testing has grown rapidly.

However, despite the increasing use of DA scores in SQA research, previous research has pointed out that DA scores are often subject to various biases (Zielinski et al., 2007), which hinder an accurate evaluation. Furthermore, since DA scores are ordinal data, calculating an average score from them is considered meaningless (Marcus-Roberts and Roberts, 1987). Therefore, the results obtained using DA scores for SQA are possibly inaccurate and lack mathematical significance.

In contrast to DA scores, preference scores can also be used for SQA. Similar to DA scores, a group of listeners is gathered to conduct listening tests to compare the quality of two speech samples. Preference scores are generally considered easier and faster for human raters to evaluate than DA scores (Thurstone, 1927), which contributes to their less noisy nature (Shah et al., 2014; Teodorescu et al., 2016). By preference scores, utterance scores or system scores can be computed and ranked using voting methods or mathematical modeling (Thurstone, 1927; Bradley and Terry, 1952). The scores or rankings obtained this way are more accurate and mathematically meaningful than those based on DA scores. However, unlike DA scores, large-scale publicly available datasets of real preference scores are novel and scarce. While some studies have used DA-score-derived preference scores (Valentini-Botinhao et al., 2022; Hu et al., 2023; Hu et al., 2024), research using real preference scores remains limited (Hu et al., 2025).

The primary bottleneck causing the scarcity of real preference score datasets is the quadratic cost of pairwise comparisons: fully evaluating $n$ systems requires comparing $O(n2)$ pairs, whereas DA scores only scale linearly [$O(n)$]. To mitigate this, Yasuda and Toda (2024) proposed a dynamic preference test that successfully addresses the cost issue by reducing the number of required pairs through online adaptation. However, while these methods achieve efficiency, the reliability and downstream utility of the scores obtained from such optimization remain insufficiently verified. To address this gap, we assess the effectiveness of pair selection strategies from two critical evaluation perspectives: (a) True score derivation: The validity of scores derived from such sparsely sampled subsets remains insufficiently verified. Yasuda and Toda (2024) evaluated their method primarily by comparing the ranking consistency with MOS. However, MOS is a fundamentally different metric with its own biases and cannot serve as the absolute ground truth for preference tests. Crucially, prior work did not verify whether the scores from the sparse subset actually match the true scores calculated from the full preference dataset, leaving the accuracy of the sampling strategy unconfirmed. (b) Model training performance: The utility of using such actively sampled scores for downstream SQA model training remains unexplored. Yasuda and Toda (2024) focused solely on ranking analysis and did not investigate the feasibility of using the resulting sparse dataset for model training. This is a critical gap because pair selection inherently drastically reduces the number of labeled utterance pairs compared to the full combinatorial dataset. Consequently, the model is exposed to significantly fewer comparisons during training. It remains to be determined whether this reduced supervision is sufficient for model convergence, and crucially, how we can effectively utilize the vast majority of “unselected” (unlabeled) utterance pairs without conducting additional listening tests for model training.

Guided by these evaluation perspectives, we systematically compare three distinct pair selection strategies under a fixed annotation budget. First, as a baseline, we examine a pair balanced strategy that distributes the budget uniformly across all system pairs. Second, we investigate the applicability of the adaptive preference-testing framework by Yasuda and Toda (2024), referred to here as a standard Merge-Rank (MR). While this framework was originally designed to output a stochastic ranking based on transitivity, we propose a novel extension to this approach: specifically, we aggregate the raw comparison outcomes collected during its execution to derive precise system-level scores via voting methods or mathematical modeling. Third, we propose a novel strategy termed MR Bias Equalization. Unlike the standard MR which focuses on sorting convergence, this method actively guides the pair selection process to equalize the estimated uncertainty – specifically the MR-based bias error – across all system pairs during data collection. We evaluate these approaches by simulating subset sampling from a full preference dataset and measuring the correlation between the derived scores and the ground truth, as well as their effectiveness in training SQA models.

Regarding model training performance, while pair selection drastically improves efficiency, it inherently reduces the number of labeled utterance pairs, leaving the majority of potential comparisons undefined. To compensate for this data scarcity and enable the model to use the vast pool of unselected utterances, we propose a pseudo-labeling strategy. Pseudo-labeling is a highly effective semi-supervised approach widely recognized for mitigating limited data issues across diverse domains (Pham et al., 2021; Njima et al., 2022). Inspired by its success, we adapt this concept to the unique context of preference tests. Specifically, we use the preference scores collected from the sparse subset to estimate system-level quality, assigning them as “pseudo-scores” to all utterances. Crucially, we use these pseudo-scores as a numerical standard to conduct pairwise comparisons between these previously unselected utterances. This process generates synthetic preference labels, effectively transforming the vast amount of unlabeled data into a rich supervision signal for model training.

The structure of this paper is as follows: in Section 2, we delve into related research on SQA and pseudo-labeling methods; in Section 3, we introduce our proposed pair selection methods and preference-based SQA framework used for experiments. In Section 4, we present our simulations and experiments. Finally, in Section 5, we discuss the limitations of our study and conclude our findings.

Previous speech quality assessment models, such as MOSNet (Lo et al., 2019), LDNet (Huang et al., 2022), and MBNet (Leng et al., 2021), have primarily focused on training and testing with individual speech samples and their corresponding DA scores. Recently, many model architectures have incorporated the process of learning preferences into their training. This is because incorporating references during training allows the model to avoid learning the distribution of hypothetical perfect speech samples assumed by raters, thereby improving the model’s performance (Manocha et al., 2021). In addition, since humans are generally better at determining preferences than assigning absolute scores, imitating the human preference decision process has been shown to improve model performance (Manocha et al., 2022). However, most current approaches only use this technique for pre-training and do not incorporate preference scores as a method for final evaluation.

DPAM (Manocha et al., 2020) and CDPAM (Manocha et al., 2021) leverage their collected Just Noticeable Differences dataset to perform metric and contrastive learning using neural networks. By introducing noise at various levels to speech samples, they generate paired reference samples. The model then learns to extract similar or dissimilar utterance-level embeddings depending on whether human-annotated perceptible differences exist between them.

For NORESQA (Manocha et al., 2021), nonmatching utterance pairs are created during training by adding noise separately to two speech samples. The noise levels are then compared to derive preference scores. NORESQA trains on these nonmatching utterance pairs and their associated preference scores. In the testing phase, one or more clean, nonmatching reference samples are used as perfect quality anchors to predict preference scores. The quality score is then calculated based on the preference score between the test utterance and the clean reference.

SQAPP (Manocha et al., 2022) uses a two-stage training process. In the first stage, like NORESQA, the model is trained on nonmatching utterance pairs and preference scores. In the second stage, it trains directly on individual speech samples and their associated DA scores. During evaluation, the model predicts DA scores for the speech samples directly.

PrefNet (Valentini-Botinhao et al., 2022) compares multiple speech samples’ DA scores during training to generate preference scores for training. Although this model relies entirely on preference scores for both training and testing, it focuses only on the accuracy of preference prediction during testing, rather than on utterance- or system-level scores or rankings.

Compared with methods that may use references for training or testing in specific cases, we use E2EPref (Hu et al., 2025), which incorporates four additional components, such as Pair Generation, Preference Function, Threshold Selection, and Preference Aggregation, into the typical DA-score-based SQA framework, allowing it to predict system-level quality scores and rankings in an end-to-end manner.

The pseudo-labeling approach is a semi-supervised method designed to address situations where labeled data is scarce. This approach has been applied to various tasks, such as image classification (Pham et al., 2021; Xie et al., 2020), automatic speech recognition (Synnaeve et al., 2020; Kahn et al., 2020), indoor localization (Njima et al., 2022), and speech emotion recognition (Venkateswaran et al., 2021). These tasks typically use a pre-trained model, a teacher model, or a model in the training process to predict pseudo labels for unlabeled inputs, which are then used to train the model.

In our approach, we assume that the quality of the speech samples generated by the system remains constant. We use a preference aggregation method to combine the collected preference scores into system scores. These system scores are considered as quality ratings for all speech samples produced by the system. We then compare these scores to generate derived preference scores for training.

In this study, we focus exclusively on the evaluation of preference-based pair selection methods. Our primary objective is to investigate the effectiveness of different pair selection strategies applied within a crowdsourcing environment. Specifically, we address the following core question:

To address this, we outline an online pair selection framework intended for real-world crowdsourcing scenarios. As illustrated in the Application Stage (top half) of Figure 1, the system operates as an iterative cycle. In each step, a selection algorithm determines a specific system pair (e.g. System A vs System B) based on a sampling strategy. From this pair, an utterance pair is randomly sampled and presented to listeners for subjective evaluation. This cycle continues until a termination condition is met. However, a critical limitation of this real-world deployment is the absence of ground truth: simply collecting sparse scores does not allow us to quantify how much the predicted quality deviates from the true rankings.

Therefore, to overcome this limitation and enable evaluation against a valid ground truth, we adopt a simulation-based experimental procedure, as depicted in the Evaluation Stage (bottom half) of Figure 1. Instead of deploying the system to live listeners, we simulate the pair selection process using a previously collected complete preference dataset. Specifically, we use the selection algorithm to iteratively select utterance pairs and retrieve their corresponding preference scores from the full dataset, forming a sub-dataset. We then evaluate this sub-dataset from two perspectives:

• Ground-truth derivation: We define the ground truth as system scores aggregated from the full dataset using preference aggregation methods. We then compute the correlation between this ground truth and system scores derived from the subset selected by each pair selection method. A high correlation indicates that the method successfully extracts representative pairs suitable for efficient listening tests.

• SQA model training performance: We also investigate how well a model can be trained when only a limited number of comparisons are available. In this context, we further evaluate the effectiveness of our proposed pseudo-labeling strategy, which leverages the vast pool of unselected utterance pairs to augment the training supervision. A high correlation indicates that the method successfully extracts representative pairs suitable for effective SQA model training.

It is important to note that this objective fundamentally differs from reward-maximizing strategies, such as standard multi-armed bandit algorithms. While such methods use sampling to support decision-making (e.g. identifying the best systems), our framework instead uses it to obtain reliable pairwise preference estimates for downstream aggregation and model training.

In this paper, we investigate four pair selection methods: one trivial baseline and three based on the MR method (Falahatgar et al., 2017). Section 3.2 introduces the MR method, explaining how it reduces the number of required preference comparisons while preserving the accuracy of system-level rankings, along with its mathematical foundations. Then, in Section 3.3, we describe all investigated pair selection methods in detail. Section 3.4 introduces a pseudo-labeling approach to further enhance SQA model training efficiency. Finally, to evaluate by SQA model training performance, we adopt the E2EPref framework (Hu et al., 2025), described in Section 3.5.

When we apply MR method on SQA task for scoring and ranking systems, it reduces comparisons on two distinct levels:

1. System-pair selection level: At the system-pair selection level, MR reduces the number of required comparisons by leveraging the strong stochastic transitivity assumption. Under strong stochastic transitivity, if the algorithm has already established that system i is preferred to system j, and system j is preferred to system k, each with an error rate bounded by $δ$⁠, then the relative order between i and i can be inferred without performing a direct comparison while preserving the same $δ$ guarantee. This transitive inference allows MR to prune a large number of system-pair comparisons, particularly for pairs with substantial quality differences, and therefore improves comparison efficiency. The complete MR procedure is provided in  Appendix 1.

2. Individual comparison level: At the individual comparison level, MR determines the winner between two systems through repeated stochastic comparisons. To reduce the sampling cost while maintaining correctness, MR incorporates an adaptive early-stopping rule consisting of two conditions. The first condition evaluates an estimation bias term (illustrated in Figure 2). After $r$ repeated comparisons between systems $i$ and  $j$⁠, let $pi,jr=w/r$⁠, and define the error bias as:

where the confidence term $c$ is:

MR halts the comparison early if $ϵ^<ϵ$⁠.

The second stopping condition enforces a maximum sampling budget, independent of the estimation bias, to guarantee that the comparison does not continue indefinitely. The budget is given by:

and MR halts the comparison when $r=m$⁠. Together, these two stopping rules ensure that each pairwise decision has an error probability bounded by $δ$⁠. The full probabilistic guarantees and derivations are provided in  Appendix 2.

Original MR methods only output ranking results. In this paper, we not only examine the rankings produced by the original MR algorithm (i.e. the output of Algorithm 1 in  Appendix 1), but also propose a new way to use MR by aggregating system-level scores directly from the collected pairwise comparison outcomes (i.e. all preference scores collected from line 9 in Algorithm 3 in  Appendix 2).

We denote the original MR method as MR (S), which serves as our baseline and produces rankings through system-level transitivity. Since MR (S) only yields system rankings, we evaluate only its ranking correlation in our experiments. In contrast, our proposed method, denoted as MR (U), extends MR by aggregating the raw comparison outcomes using the preference aggregation methods adopted in the E2EPref framework introduced in Section 3.5. Importantly, because MR avoids comparing system pairs with large score differences, its role in MR (U) is to guide the collection of preference scores specifically for systems with similar estimated scores, making the aggregated scoring both efficient and informative.

We investigate four pair selection methods: one trivial method and three based on the MR framework. Despite their differences, all methods follow the same iterative process. In each step, we use the pair selection algorithm to choose an informative system pair. From this chosen system pair, we simply randomly sample an available utterance pair and add it to our sub-dataset. This cycle repeats until the stopping criterion is met.

This method represents a trivial but informative baseline that assumes all system pairs require equal evaluation effort. Given the total number of preference scores to be collected, we gather an equal number of comparison results for each system pair. As a result, this method guarantees full coverage of all system pairs and serves as a reference for evaluating the benefits of adaptive pair selection strategies.

This approach applies the MR method with the system array initialized in sorted order based on the MOS, representing a scenario where a coarse prior ranking is available. The goal is to examine how incorporating such prior information influences the selection of system pairs and the resulting preference data. Under this pair selection strategy, we denote the system rankings obtained directly from MR as MR Sorted (S), and the system scores obtained by aggregating the collected preference scores as MR Sorted (U).

This method applies the MR framework with the system array randomly initialized, without relying on any prior ranking information. It serves as an unbiased reference for evaluating the effectiveness of adaptive pair selection when no prior knowledge is available. Under this pair selection strategy, we denote the system rankings obtained directly from MR as MR Random (S), and the system scores obtained by aggregating the collected preference scores as MR Random (U).

In addition to MR Sorted and MR Random, we also propose a MR-based pair selection method that focuses on equalizing the MR bias error across all system pairs. MR provides an explicit estimate of uncertainty for each system pair through its bias error, which reflects the reliability of the corresponding preference comparisons under noisy listening conditions. In this study, where preference data are further used for system-level aggregation and pseudo-labeling, maintaining comparable uncertainty across system pairs is important to avoid over-representing highly confident pairs while leaving others under-explored.

Based on this observation, by adopting this approach, we aim to gather comparison results from every system pair, using the bias error $ϵ^$ introduced in the MR framework as a criterion to determine how many listening tests each pair requires to reach a similar confidence level. Specifically, during each iteration, we prioritize the system pair with the highest bias error and collect additional listening test data for it. To prevent excessive testing on pairs whose bias error decreases more slowly, we impose a maximum budget constraint that caps the number of tests allowed for any single system pair, ensuring a balanced and efficient evaluation process.

To compensate for the inherent reduction in labeled utterance pairs caused by pair selection, this approach enables the model to use the vast pool of unselected utterance pairs. Instead of relying solely on the sparse human-annotated subset, we propose a mechanism to extract supervision signals from the available but uncompared data.

Specifically, we first aggregate the collected preference scores using the preference aggregation methods in the E2EPref framework (introduced in Section 3.5) to derive system-level quality scores. We then assign these derived system scores to every utterance produced by the respective system, designating them as utterance-level pseudo DA scores. This assignment is applied globally to form a “pseudo-labeled dataset” that encompasses all available utterances – including both the vast number of unselected utterances and those that were already involved in real preference comparisons. By establishing these pseudo DA scores as a universal numerical standard within this dataset, we enable the formation of training pairs from any combination of utterances, which are subsequently used in the model training process described in Section 3.5.

In our experimental workflow, the proposed online pair selection method is first used to select a representative sub-dataset. This selected sub-dataset is then used to train and test the downstream model, E2EPref (Hu et al., 2025). E2EPref serves as the backbone SQA framework and processes the input data through four components: Pair Generation, Preference Function, Threshold Selection, and Preference Aggregation, as illustrated in Figure 3.

It is crucial to distinguish our upstream pair selection method from the internal “Pair Generation” component of E2EPref. During training, this component adapts its behavior to support two experimental configurations: (1) training solely with the real preference dataset, and (2) training with a mixture of the real preference dataset and the pseudo-labeled dataset. For the real preference dataset (used in both configurations), the internal generation step acts as a deterministic retrieval mechanism. Since the comparison pairs are explicitly defined by the upstream selection process, the component simply fetches the existing utterance pair and assigns its annotated real preference score as the ground truth (⁠$sreal$⁠). For the pseudo-labeled dataset (used in the second configuration), the component operates in a dynamic construction mode. In this case, two (utterance, pseudo DA score) tuples are randomly sampled to form a training pair. The ground truth is then derived from the comparison of their pseudo DA scores (⁠$y1,y2$⁠). When training with the mixed configuration, pairs are sampled with equal probability from the two sources. Regardless of the source, the model’s predicted preference is defined as:

where $x1$ and $x2$ represent the scores predicted by the model for the two utterance inputs. The ground truth is formally defined as:

Here, $sgn(·)$ represents the sign function. The overall training loss is given by: $L = MSE(prefgt−prefdv)$⁠.

During the inference phase, given a set of test systems to be evaluated, our objective is to derive their final system-level scores and rankings. To achieve this, we strictly follow the protocols defined in Hu et al. (2025). The process initiates with Pair Generation, where we construct a specific set of comparison pairs from these test systems to be processed by the subsequent modules. Specifically, we use the Balanced System Pair Selection method for inference. This strategy ensures that every system pair is assigned the same number of test utterance pairs for comparison. Subsequently, the model evaluates the DA scores of the utterances in each generated pair and applies the same Preference Function used during training to produce a continuous preference score for each pair. Regarding Threshold Selection, we apply no filtering threshold; the sign of the predicted score is directly used to determine the preference direction (win or loss) for the subsequent aggregation step. Finally, Preference Aggregation is performed to derive a single scalar score for each system from the collected pairwise outcomes. We use four standard methods: winning count (WC), which sums the total wins; differential count (DC), which calculates wins minus losses; and two generalized linear model (GLM) methods, namely, the Bradley–Terry–Luce (BTL) model and Thurstone’s Case V Model (THUR). As described in Hu et al. (2025), these GLM methods estimate latent system scores such that the distance between two scores correlates with the probability of one outperforming the other. This is achieved via a link function $g(μ)$ mapping the score difference $μ$ to a probability space $[0,1]$⁠. Specifically, the BTL model uses the logit link $g(μ)=ln(μ1−μ)$⁠, while the THUR model uses the probit link $g(μ)=Φ(μ)$⁠, where $Φ$ is the standard normal CDF.

As introduced in Section 3, we evaluate the pair selection methods and the pseudo-labeling approach from two perspectives: ground-truth derivation and SQA model training performance. For ground-truth derivation, we use correlation metrics to assess pair selection methods by comparing system-level quality scores aggregated from extracted preference scores with those derived from the full set of preference scores. For SQA model training performance, we use correlation metrics to assess the proposed method by comparing system-level quality scores aggregated from predicted and ground-truth preference scores of test dataset.

In Section 4.1, we introduce the correlation metrics used for evaluation. The details of the dataset used in the experiment are provided in Section 4.2. We then present the experimental setup and results for the ground-truth derivation and the SQA model training performance in Sections 4.3 and 4.4, respectively.

To compare system-level quality scores aggregated from extracted preference scores with those derived from the full set of preference scores, we use two widely used correlation metrics: linear correlation coefficient (LCC) and Spearman’s rank correlation coefficient (SRCC).

The LCC measures the strength and direction of the linear relationship between the predicted system-level scores and the ground-truth system-level scores. It is defined as:

where $Xi$ and $Yi$ represent the predicted and ground-truth system-level scores for the $i$th system, respectively, $X¯$ and $Y¯$ are their respective means across all systems, and $N$ is the total number of systems being evaluated.

LCC ranges from $−1$ to $1$⁠, where values close to $1$ indicate a strong positive correlation, values close to $−1$ indicate a strong negative correlation, and values near $0$ suggest no linear relationship. A higher LCC value indicates better alignment between the predicted and ground-truth system-level quality scores.

The SRCC is a rank-based metric that assesses the monotonic relationship between the predicted and ground-truth system-level scores. Unlike LCC, SRCC considers only the order of the values rather than their absolute differences, making it robust to nonlinear relationships. It is computed as:

where $Di$ is the rank difference between the predicted and ground-truth system-level scores for the $i$th system, and $N$ is the total number of systems.

SRCC ranges from $−1$ to $1$⁠, with values near $1$ indicating a strong positive monotonic relationship and values near $−1$ indicating an inverse monotonic relationship. A higher SRCC suggests that the predicted rankings of system-level speech quality closely match the ground-truth rankings.

We used the Voice Conversion Challenge 2020 (VCC2020), which was the third edition of the Voice Conversion Challenge. It comprises two tasks: the intra-lingual semi-parallel VC task and the cross-lingual VC task. In this paper, we focus on the intra-lingual semi-parallel task for our experiments. The available datasets for this task include the VCC2020-DA dataset, which contains DA scores (Yi et al., 2020), and the VCC2020-Pref dataset, which includes the preference scores collected later (Hu et al., 2025).

In VCC2020, there are 32 voice conversion systems, each producing 80 utterances given the same set of input utterances and target speakers. For the VCC2020-DA dataset, each listener provided between 32 and 320 scores, totaling 604 participants. Each utterance was evaluated 12 times, resulting in a total of 30,720 scores (32 systems × 80 utterances × 12 evaluations). The VCC2020-Pref dataset contains pairwise comparison results for all systems. Each utterance in every system pair was compared twice, leading to 158,720 preference scores (32 systems × 31 system pairs × 80 utterances × 2 comparisons). Notably, due to the matching utterance design used in the listening test, each comparison involved two utterances generated by different voice conversion models but sharing the same input utterance and target speaker. Across 1,548 participants, each listener evaluated between 40 and 10,040 utterance pairs.

In our experiments, we used two criteria proposed in Hu et al. (2025) to split systems into training and testing sets based on the properties of system pairs in each set. The first, termed DACon, involves selecting system pairs for the test set with statistically significant differences in DA scores. This ensures a subset of systems with unambiguous rankings, allowing evaluation based solely on DA scores. The second criterion, PrefCon, aims to select a subset of systems that maximizes the number of conflicting preference directions of its internal system pairs between DA and preference scores. It prioritizes cases where preference scores capture subtle distinctions that DA scores possibly miss. To implement this on the VCC2020-Pref dataset, we first precalculated the pairwise preference directions based on DA scores and preference scores. Then, we brute-forced all $(3210)$ combinations of selecting 10 systems out of 32. For each combination, we evaluated its $(102)$ internal system pairs to count the conflicting directions, ultimately selecting the combination with the maximum conflicts. Thanks to the precalculated directions, the overall time complexity remains highly manageable. These criteria provide complementary perspectives for ranking evaluations.

Using these two criteria, ten systems were selected from the complete dataset for the test set ^[1], leaving the remaining 22 systems as the training set. Based on the two criteria (DACon and PrefCon), two different dataset splits of the VCC2020-DA dataset were created: one called VCC2020-DACon-DA, and the other called VCC2020-PrefCon-DA. Similarly, the VCC2020-Pref dataset was split according to the same two criteria, resulting in VCC2020-DACon-Pref and VCC2020-PrefCon-Pref. Notably, when splitting the VCC2020-Pref dataset, preference scores were used selectively: only scores between training systems were included in the training set, while scores between test systems were assigned to the test set.

For the test set under the DACon criterion, the system MOS obtained from all preference aggregation methods for VCC2020-DACon-Pref showed perfect correlation (SRCC = 1) with the average system MOS obtained from VCC2020-DACon-DA. In contrast, for the test set under the PrefCon criterion, 25 out of 45 system pairs among the ten selected test systems exhibited differing tendencies in MOS compared to preference scores. The SRCC between the system scores obtained from the BTL method for VCC2020-PrefCon-Pref and the system MOS from VCC2020-PrefCon-DA was approximately 0.733. It is worth noting that only the VCC2020-Pref dataset was used in the experiments; the VCC2020-DA dataset was used to calculate the MOS for the initialization of the MR Sorted method.

In this subsection, we evaluate the effectiveness of various pair selection methods through a simulation-based approach. Specifically, we extract a subset from the VCC2020-Pref training set using pair selection methods, as illustrated in Figure 1. Before the evaluation, we first investigate the relationship between the adjustment of $ϵ$ and the number of collected scores. Our goal is to identify combinations of MR methods and their corresponding $ϵ$ values that yield a comparable number of collected scores, ensuring a fair comparison between different MR methods. Next, we examine the number of collected scores and the final bias error achieved for each system pair using each proposed selection method. Finally, we evaluate the proposed pair selection methods using the system-level quality scores obtained from the full training set as ground truth, and assess the LCC and SRCC between the subset-derived scores and the ground truth.

We simulate the listening test process for the four pair selection methods using the training set of both VCC2020-PrefCon-Pref and VCC2020-DACon-Pref. The collected preference scores are then aggregated using four preference aggregation methods introduced in Section 3.5. The aggregated system scores or rankings are compared against the ground truth computed from the complete dataset using LCC and SRCC. We plot the LCC and SRCC versus the number of scores. This allows us to evaluate which method can more accurately estimate system-level scores or rankings with the same number of preference scores.

For the Pair Balanced and MR Bias Equalization methods, we consistently collect data in multiples of the total number of system combinations $C222=231$⁠. Initially, we simulate a single multiple. Starting from $n=1000$⁠, we increase the dataset size in steps of 1,000 up to 20,000, collecting $floor(n/231)×231$ pairs per simulation. In addition, for the calculation of bias error in MR Bias Equalization method, we use $ϵ=0.0877$ and $δ=0.05$ from Yasuda and Toda (2024). Specifically, we set the confidence level to $δ=0.05$ to ensure a standard 95% statistical confidence. The tolerance bias was set to $ϵ=0.0877$⁠, which is mathematically derived from Hoeffding’s inequality (⁠$m=12ϵ2log2δ$⁠) by bounding the maximum number of evaluations per system pair to m = 240. Structurally, this bound is explicitly tailored to our dataset capacity: it ensures that the algorithm samples at most 75% (240 out of 320) of the available evaluation scores per system pair in the worst-case scenario, preventing data exhaustion while preserving sufficient sampling space to avoid repetitive or deterministic outcomes. For MR methods, we fixed $δ=0.05$ and varied $ϵ$ from 0.05 to 0.5 in steps of 0.01. A higher value of $ϵ$ indicates greater tolerance for bias error, which reduces the difficulty in determining the winner in a system pair. Each $ϵ$ value is simulated 100 times, collecting 100 subsets per simulation, resulting in 10,000 subsets in total. Specifically for the MR Random method, the system array initialization is entirely reshuffled for each of the 100 simulations, rather than being held static. Notably, the number of scores collected in each subset, as well as the SRCC and LCC values compared to the ground truth, may vary.

We first analyzed the relationship between $ϵ$ and the number of collected scores on both VCC2020-DACon-Pref and VCC2020-PrefCon-Pref. In this analysis, we calculated the average number of scores for each $ϵ$ and plotted them in Figure 4(a) and (b),. It is important to clarify that we are currently analyzing the raw collected dataset. Since the distinction between MR (S) and MR (U) lies only in the final scoring derivation and not the data collection itself, we present our analysis solely in terms of MR Sorted versus MR Random.

Figure 4(a) shows the changes in the number of collected scores caused by adjusting $ϵ$ in the MR Sorted and MR Random methods on VCC2020-DACon-Pref. We found that as $ϵ$ increases, the number of collected scores starts to decrease exponentially. We believe this is due to the design of the maximum budget $m$⁠: as $ϵ$ increases, the number of collected scores is limited by a relationship that is inversely proportional to the square of $ϵ$ as mentioned in Section 3.2. Moreover, we observe that the number of collected scores by the MR Random method is consistently greater than that by the MR Sorted method. This suggests that, by the MR method, using MOS for array initialization can significantly reduce the number of scores needed to be collected.

Similarly, we also analyzed the relationship between $ϵ$ and the number of collected scores in VCC2020-PrefCon-Pref. As shown in Figure 4(b), the changes in the number of collected scores for the MR Sorted and MR Random methods when adjusting $ϵ$ were similar to those observed in VCC2020-DACon-Pref. The number of collected scores also decreases exponentially for both MR methods. Under the same $ϵ$⁠, the MR Sorted method consistently collected fewer scores than the MR Random method. This further supports the observation that, within the MR method, using MOS for array initialization can significantly reduce the number of scores needed to be collected.

We analyze the number of scores collected and the resulting bias error for each system pair by each pair selection method. For each method, we collect 5,000 scores. Specifically, for MR Random and MR Sorted, we identified the $ϵ$ that allows us to collect approximately 5,000 scores on both VCC2020-Pref splits separately through the analysis in the previous discussion, and used these $ϵ$ to extract preference scores. We plot the relationship between the scores collected by each method and the bias in Figures 5 and 6. In these figures, the x-axis represents the systems sorted by MOS, from left to right, and the y-axis represents the system order from top to bottom after the MOS sorting.

First, for both datasets, we observed that the Pair Balanced and MR Bias Equalization methods successfully achieved their objectives: the former extracted the same number of scores for each system pair, while the latter ensured that the bias for each system pair decreased equivalently. For the MR Sorted method, each iteration divided the systems into high-score and low-score groups based on the MOS scores. This approach effectively used the lowest-scoring system from the high-score group to compare with all systems in the low-score group, reducing comparisons between pairs with larger score differences. For example, comparisons in the lower-left and upper-right regions of the plot were entirely avoided in this method. In contrast, MR Random lacks this advantage, as the systems were initially randomly sorted. However, this method still effectively reduced the number of comparisons between system pairs that were farther from the main diagonal. We plotted the average number of scores collected per system pair versus the offset from the main diagonal on the y-axis and x-axis in Figure 7. From this, we can see that the majority of comparisons occurred between system pairs that were closer to the main diagonal.

In this section, we evaluate the effectiveness of the pair selection methods by measuring the correlation (LCC and SRCC) between the system scores derived from the collected subset and the ground-truth scores aggregated from the full dataset. To ensure a systematic evaluation, we apply a consistent aggregation protocol: the same aggregation function is applied to both the partial collected scores and the full dataset to derive their respective system scores. This process is repeated for each aggregation method, with results presented in Figures 8 and 9. Unlike the previous data collection analysis, here we explicitly distinguish between the MR (U) and MR (S) variants due to their distinct output formats. The MR (U) variants produce continuous scores via the aforementioned aggregation protocol. In contrast, the MR (S) variants generate system rankings and are therefore independent of post hoc aggregation functions. Consequently, while we include MR (S) in the figures for comparison, we evaluate their performance solely using the SRCC metric, as they provide ordinal rankings rather than interval quality scores.

We first examined two mathematical model aggregation methods, THUR and BTL. On the VCC2020-DACon-Pref split, we observed that the Pair Balanced method and the MR Bias Equalization method exhibited similar performance in both SRCC and LCC. The MR Sorted (U) method achieved comparable LCC performance to these two methods when the number of collected scores was low, but its performance slightly dropped when the number of collected scores increased. In terms of SRCC, MR Sorted (U) achieved the best performance overall. MR Random (U) showed the opposite trend compared to MR Sorted (U) in LCC. It performed worse than the other three methods when the number of collected scores was low but achieved similar performance to the Pair Balanced and MR Bias Equalization methods when the number of collected scores was high. In terms of SRCC, its performance was also comparable to the Pair Balanced and MR Bias Equalization methods. On the other hand, MR Random (S) and MR Sorted (S), which are based on the original MR output, consistently showed the worst performance among all methods.

On the VCC2020-PrefCon-Pref split, similar trends were observed, except that the MR Bias Equalization method achieved the best performance across all numbers of collected scores. Similarly, MR Sorted (U) performed relatively well with a small number of collected scores but degraded when the number of collected scores increased. MR Random (U) again showed poor performance with few collected scores but improved as the number of collected scores increased. Meanwhile, MR Random (S) and MR Sorted (S) consistently showed the worst results.

In summary, when using BTL and THUR aggregation methods, we conclude that MR Bias Equalization yields the best performance. It effectively balances the bias across system pairs, leading to more reliable overall system score predictions. Although the MR Sorted (U) method benefits from utilizing MOS information and performs well with a small number of collected scores, it struggles when the number of collected scores increases. We believe this is due to the reduction in the number of system pairs compared, making further performance improvements more difficult. As for MR Random (U), although it is weaker with fewer collected scores, it can achieve performance similar to that of the Pair Balanced method and the MR Bias Balanced method when the number of collected scores is high. In contrast, MR Sorted (S) and MR Random (S) consistently underperform across all scenarios and fail to achieve results comparable to methods that aggregate preference scores.

For the DC and BC counting methods, which depend on the number of system comparisons, imbalanced comparison counts can affect the final aggregated scores. From the LCC and SRCC results, it is evident that the Pair Balanced method performed the best. Although the MR Bias Equalization method balanced system comparisons in some cases, it still underperformed compared to Pair Balanced. The MR methods, due to imbalances in system comparison counts, performed worse than both Pair Balanced and MR Bias Equalization. Among the MR methods, MR Random (U) and MR Random (S) outperformed MR Sorted (U) and MR Sorted (S), respectively. This is because the random initialization of the system list in MR Random methods helps balance the frequency of comparisons between higher- and lower-quality systems, which are compared less frequently in MR Sorted (U) and MR Sorted (S).

In conclusion, the combination of MR Bias Equalization and GLM aggregation methods emerges as the most robust configuration, demonstrating consistent effectiveness across varying numbers of collected scores and dataset splits. When compared with Pair Balanced, MR Bias Equalization shows clear advantages, especially when the number of preference scores is small. On VCC2020-DACon-Pref, the two methods achieve similar performance under low pair counts. However, on VCC2020-PrefCon-Pref, Pair Balanced is consistently limited to around 0.75 LCC and SRCC regardless of the aggregation method, whereas MR Bias Equalization combined with GLM methods consistently achieves LCC and SRCC above 0.8. This indicates that MR Bias Equalization preserves system-level ranking quality more reliably when data are sparse. MR Random (S) and MR Sorted (S) perform significantly worse than all other methods and are not recommended. If an MR method must be used, our proposed MR (U), which aggregates all collected preference scores, is a better choice than MR (S), although it remains less robust than MR Bias Equalization.

In this section, we analyze the effectiveness of pair selection methods and the pseudo-labeling approach based on model performance. For each method, we collect approximately 5,000 scores for model training. In addition, we also evaluate models trained with and without the proposed pseudo-labeling method. For the MR methods, we use only our proposed MR (U) method to compute pseudo system scores. We do not use MR (S), as the simulation results in Section 4.3.2 show that MR (S) produces poor ranking accuracy and is therefore unsuitable for generating reliable pseudo labels. When applying MR (U), we pair it with the DC aggregation method, which provides stable system-level scores for the pseudo-labeling process. As a baseline, we also report models trained on the full dataset. We evaluate both this baseline and the pair selection methods with and without the pseudo-labeling approach. When implementing the pseudo-labeling approach, specific aggregation methods are used to derive the requisite system-level quality scores: we use the DC aggregation method for the Full Dataset and the Pair Balanced method, while using the BTL aggregation method for the three MR variants [MR Sorted (U), MR Random (U), and MR Bias Equalization]. To ensure the reliability of the results, we repeated each experimental configuration 20 times independently. The final results, including the mean and statistical comparisons based on $p$-values derived from these 20 trials, are summarized and presented in Tables 1 and 2 .

In the following analysis, we systematically evaluate the pseudo-labeling approach and the pair selection methods considered in this work. We first conduct a detailed ablation study to assess the contributions of these components. We then broaden the analysis by comparing these configurations with a baseline trained on the full dataset. Finally, we summarize the key findings to provide practical insights.

To assess the effect of pseudo-labeling, we conducted an ablation study by comparing models trained with and without pseudo-labeling across all pair selection methods. On the VCC2020-DACon-Pref split, the use of pseudo-labeling consistently led to statistically significant improvements regardless of the pair selection method. Specifically, we observed absolute increases of 0.07–0.09 in SRCC and 0.13–0.20 in LCC. On the VCC2020-PrefCon-Pref split, the impact was more varied: while changes in SRCC were marginal (below 0.03) and not statistically significant, LCC still showed a significant improvement of approximately 0.04 across all methods except for MR Sorted (U). These quantitative improvements highlight the core advantage of our pseudo-labeling strategy: it acts as an effective data augmentation mechanism. By leveraging the vast pool of unselected utterances, the model captures broader acoustic diversity and avoids overfitting to the sparse human-annotated subset, thereby achieving better generalization with fewer human labels.

To further examine the impact of potential noise introduced by pseudo-labeling, we analyzed the consistency between pseudo-label-induced pairwise decisions and human preferences. The results showed that pseudo labels achieve accuracy close to human consistency, indicating that the noise introduced by system-level assignment is limited. Detailed results are provided in  Appendix 3.

To analyze the impact of different pair selection strategies, we compared the performance of MR Sorted, MR Random, Pair Balanced, and MR Bias Equalization. No statistically significant differences were observed across methods on either dataset split, regardless of whether pseudo-labeling was applied. On the VCC2020-DACon-Pref split without pseudo-labeling, the maximum performance gap between the best and worst models was small (0.02 in SRCC and 0.04 in LCC). With pseudo-labeling, these differences further decreased to 0.007 in SRCC and 0.009 in LCC. Similarly, on the VCC2020-PrefCon-Pref split, the maximum differences without pseudo-labeling remained small (0.011 in SRCC and 0.019 in LCC). Although the introduction of pseudo-labeling on this split appeared to increase the maximum gaps (up to 0.05 in SRCC and 0.024 in LCC), the relatively high variance in the results rendered these differences statistically insignificant.

While no statistically significant differences were found overall, the average performance across conditions revealed some noteworthy patterns. On the VCC2020-DACon-Pref split, without the pseudo-labeling approach, the MR Bias Equalization method yielded the best results, whereas with the pseudo-labeling approach, Pair Balanced performed best. Conversely, on the VCC2020-PrefCon-Pref split, MR Sorted (U) was the top performer without the pseudo-labeling approach, while MR Bias Equalization and MR Random (U) achieved the highest performance when the pseudo-labeling approach was used.

Although Section 4.3.2 discusses the relative effectiveness of each method in producing pseudo-derived system scores, we found no clear or consistent alignment between those earlier findings and the performance outcomes observed here, except in the case of MR Bias Equalization. While it did not significantly outperform other methods in any single setting, it consistently ranked among the top, frequently placing first or second across both dataset splits, both with and without the pseudo-labeling method. This recurring pattern suggests that MR Bias Equalization may offer greater robustness across different training scenarios. Therefore, despite the overall lack of statistically significant differences, we consider MR Bias Equalization a generally reliable and effective choice for pair selection in practice.

Finally, we compared the performance of models trained using pair selection methods combined with the pseudo-labeling approach against those trained on the full dataset. Since the pseudo-labeling approach generally improves performance, we focused on configurations that incorporate it. The model trained on the full dataset without the pseudo-labeling approach achieved the highest SRCC across both dataset splits when compared to models using pair selection methods. However, it consistently underperformed in terms of LCC relative to those models. In contrast, the model trained on the full dataset with the pseudo-labeling approach showed more competitive results. On the VCC2020-DACon-Pref split, models using pair selection methods achieved SRCC values comparable to, or even slightly better than, those of the full dataset model with the pseudo-labeling approach, although their LCC values remained slightly lower. On the VCC2020-PrefCon-Pref split, the SRCC of the full dataset model with the pseudo-labeling approach dropped significantly, allowing all pair selection methods to surpass it. In terms of LCC, some pair selection methods – particularly those using DC and WC aggregation – matched or slightly exceeded the performance of the full dataset model.

From this, we observed that although the pseudo-labeling approach benefited pair selection methods, its effectiveness was less consistent when applied to the full dataset. One possible explanation is that pseudo-labels convert inherently uncertain human preferences into deterministic pairwise comparisons. In human annotations, system pairs with similar quality often exhibit low agreement, reflecting ambiguity in relative preference, whereas pseudo-labels enforce a fixed ordering for such pairs.

Specifically, this deterministic supervision may encourage the model to learn more globally consistent ordering across systems, which in turn can lead to more stable and well-separated score assignments. This may strengthen the linear relationship between predicted and ground-truth scores, thereby benefiting LCC.

However, for ambiguous pairs, enforcing a fixed ordering may conflict with the underlying human preference distribution. Since SRCC is sensitive to local ranking errors near such decision boundaries, this mismatch could contribute to the observed degradation in SRCC. Further investigation is required to validate this hypothesis, which we consider an important direction for future work.

In summary, we conclude two key points: (1) Robustness of pair selection: First, regarding the acquisition of preference scores, MR Bias Equalization proved to be the most robust strategy. Even with a limited budget of 5,000 scores, it consistently yielded stable system-level scores across varying data distributions. While it performed comparably to the strong Pair Balanced baseline on the standard VCC2020-DACon-Pref split, it demonstrated a clear advantage on the more challenging VCC2020-PrefCon-Pref split, outperforming Pair Balanced across all metrics. This confirms MR Bias Equalization as a reliable choice for selecting high-quality subsets of data. (2) Effectiveness of pseudo-labeling: Second, to effectively use these selected subsets, the pseudo-labeling approach is essential. This method directly compensates for the inherent scarcity of explicit preference scores resulting from the selection process. By deriving system-level scores to annotate individual utterances, pseudo-labeling generates dense supervision signals that enable the model to generalize effectively. Consequently, models trained with this pseudo-labeled data achieved performance comparable to those trained on the full dataset, demonstrating that high-performance learning is achievable with significantly reduced annotation costs.

While the investigated strategies demonstrate robust performance on the Voice Conversion task using the VCC2020 dataset, their generalizability to other domains warrants further discussion.

First, the pseudo-labeling strategy assumes negligible intra-system variance by assigning a global system-level score to all its utterances. This assumption holds well for the Voice Conversion dataset used in our study (VCC2020), as it consists of clean speech recordings and strictly intra-lingual (English-to-English) conversions, which naturally constrain the intra-system variance. However, in domains with high intra-system variance, such as Text-to-Speech with complex prosodic variations or Speech Enhancement under varying signal-to-noise ratios, this method could introduce significant label noise. Second, the MR Bias Equalization algorithm inherently prioritizes the system pair with the highest bias error. While this mechanism is highly effective under normal conditions, a limitation arises in high-variance domains like TTS or SE. In such domains, certain system pairs may be inherently difficult to evaluate due to specific text prompts or extreme noise conditions. When encountering such a pair, the algorithm will repeatedly allocate listening tests to it until it hits the predefined maximum budget cap. If a dataset contains a large number of these highly variable pairs, they will act as a bottleneck. The algorithm will exhaust a significant portion of its total evaluation budget simply reaching the cap for these specific pairs, which consequently hinders and delays the overall reduction of bias across the rest of the systems.

In this study, we conducted a comprehensive investigation into online pair selection strategies for preference-based SQA. We systematically evaluated three pair selection strategies under a fixed preference budget: (i) the Pair Balanced strategy (a uniform baseline); (ii) the Merge-Rank, where we investigated both its original stochastic ranking output [MR (S)] and our proposed score aggregation method [MR (U)]; and (iii) the proposed MR Bias Equalization, which actively guides pair selection to equalize uncertainty across system pairs. Our findings can be summarized from two key perspectives: true score derivation and model training performance.

First, regarding true score derivation, we demonstrated that MR Bias Equalization consistently achieves the most robust performance. Under the same annotation budget, it exhibits greater stability compared to other methods, including the competitive Pair Balanced baseline. Furthermore, our analysis confirms that the proposed MR (U) significantly improves stability relative to the original MR (S). Conversely, the original MR (S) exhibits consistently poor correlation with ground truth, indicating that direct reliance on the raw stochastic rankings is insufficient for accurate system evaluation.

Second, concerning model training performance, we highlight the critical role of the pseudo-labeling approach. This method effectively compensates for the inherent scarcity of explicit preference scores caused by pair selection. By deriving reliable system-level scores to provide dense supervision, it enables models to generalize well even with limited data. However, its success depends strictly on the reliability of the source scores. Consequently, MR (S) variants were excluded from pseudo-label generation. Our experiments showed that in the absence of pseudo-labeling, MR Sorted (U) achieved the strongest performance. However, when pseudo-labels were introduced, MR Bias Equalization and MR Random (U) demonstrated superior stability across different dataset splits, whereas other methods exhibited larger performance fluctuations.

In conclusion, due to its consistent robustness in both accurate score derivation and downstream model training, we identify the combination of MR Bias Equalization (for selection) and GLM (for aggregation) as the most reliable strategy for preference-based SQA.

Algorithm 1. Merge-Rank

Algorithm 2. Merge

Algorithm 3. COMPARE

In this section, we detail the Merge-Rank (MR) procedure. MR adapts the classic Merge Sort algorithm to the stochastic nature of system evaluation. The procedure operates on two levels: system-pair selection (via the sorting logic) and individual pairwise comparison (via a sampling subroutine).

Similar to Merge Sort, MR uses a divide-and-conquer strategy. The algorithm recursively divides the set of systems into halves until individual systems remain, and then merges them back in sorted order. This approach avoids the exhaustive $O(N2)$ comparisons required by a round-robin tournament. The pseudocode for the main procedure and the merge logic is outlined in Algorithms 1 and 2.

The efficiency of MR at the system-pair selection level relies on the assumption of strong stochastic transitivity. In the context of ranking, strong stochastic transitivity assumes that the winning probability propagates through intermediate comparisons. Specifically, for systems $i,j,k$⁠, the probability satisfies:

This property implies that if the algorithm determines $i≻j$ and $j≻k$ (each with an error probability bounded by $δ$⁠), the order $i≻k$ can be inferred with the same error bound $δ$ without performing a direct comparison between $i$ and $k$⁠. This transitive inference allows MR to prune a significant number of comparisons, particularly when systems exhibit distinct quality differences.

In this section, we provide the probabilistic derivations for the adaptive stopping rules used in the COMPARE function described in Algorithm 3. The goal is to ensure that the error rate of determining the winner between any two systems $i$ and $j$ is bounded by $δ$⁠. The function uses two stopping mechanisms: a bias-based criterion and a maximum budget constraint.

Let $r$ denote the number of comparisons performed so far, and let $pi,jr$ be the empirical probability that system $i$ wins over system $j$⁠. We define the confidence term $c$ and the estimated bias $ϵ^$ as:

The algorithm terminates when $ϵ^<ϵ$⁠. Substituting equation (8), we obtain:

which is equivalent to:

Suppose the empirical winner is system $i$⁠, namely:

Combining this with the stopping condition yields:

An error occurs only if the true preference is actually in favor of system $j$⁠, that is:

Therefore:

which implies:

To apply Hoeffding’s inequality, let $Xk∈{0,1}$ denote the outcome of the $k$-th comparison, where $Xk=1$ if system $i$ wins over system $j$⁠, and $Xk=0$ otherwise. Then:

with $E[Xk]=pi,j$⁠. Since $Xk$ are independent Bernoulli random variables, Hoeffding’s inequality gives:

Setting $t=c−ϵ$⁠, we obtain:

Following the derivation provided in our methodology, we upper bound this term by $exp(−2r(c2))$⁠:

Since the stopping condition could be met at any step $r$⁠, we apply Boole’s inequality (union bound) over all possible values of $r$⁠:

Using the Basel problem identity $∑r=1∞1r2=π26$⁠, we obtain:

Thus, when the algorithm halts due to the bias-based criterion, the error rate is less than $δ/2$⁠.

To prevent infinite loops, the algorithm also enforces a maximum budget $m$⁠, defined as:

If the comparison reaches $r=m$⁠, the algorithm halts. By Chernoff’s bound, the error probability at this limit is:

Substituting $m$⁠:

Thus, when meeting the budget constraint, the error rate is bounded by $δ/2$⁠.

Combining both mechanisms, whether the COMPARE function halts due to the bias threshold (⁠$ϵ^<ϵ$⁠) or the budget constraint (⁠$r=m$⁠), the probability of correctly identifying the winner is maintained such that the error rate is consistently less than $δ/2$⁠.

We conduct the analysis on the training sets of VCC2020-DACon and VCC2020-PrefCon. Pseudo labels are constructed following the same procedure as in the model training experiments described in Section 4.4.1. Specifically, we consider the same set of pair selection methods. For each method, we sample approximately 5,000 preference scores to derive system-level quality scores.

The aggregation strategies are consistent with those used in Section 4.4.1. In particular, we apply the DC aggregation method for the pair balanced method and the full dataset setting, while using the BTL aggregation method for the MR variants. These aggregated system-level scores are then assigned to all utterances of each system to construct pseudo labels.

To evaluate pseudo-label quality, we compare pseudo-label-induced pairwise decisions with ground-truth human preference labels. For each preference instance, the two utterances correspond to two systems, each associated with a pseudo-label (i.e. system-level score). By comparing these scores, we derive a pairwise preference decision in the pseudo-label space, which is then compared with the human annotation. The accuracy is computed as the proportion of agreements between the two.

To ensure a fair comparison, we aim to estimate human consistency under the same decision framework. Ideally, this would require identifying a globally optimal system-level ranking that maximizes agreement with all human preference labels. However, this is computationally intractable due to the factorial number of possible rankings (e.g. $22!$ for 22 systems).

Instead, we approximate this upper bound by computing system pair-level human consistency. Specifically, for each system pair, we aggregate all human preference annotations involving that pair and treat the majority vote as the ground-truth label. We then measure how often individual human annotations agree with this consensus.

This formulation aligns the evaluation of pseudo labels and human consistency in the same pairwise decision space, while providing a tractable and meaningful approximation of the upper bound imposed by human agreement.

As shown in Table A1, pseudo-label-based decisions derived from different pair selection methods achieve accuracy close to human consistency, even when using only 5,000 preference scores (averaged over 100 simulations). Increasing the data size to the full dataset yields only marginal improvements, indicating that reliable pseudo labels can already be obtained from relatively small subsets.

Across all settings, the gap between pseudo-label accuracy and human consistency remains small. Notably, human consistency itself is relatively modest (0.6288 on VCC2020-DACon and 0.6455 on VCC2020-PrefCon), reflecting inherent variability in human preference annotations. This suggests that most of the residual disagreement is due to intrinsic human inconsistency rather than errors introduced by pseudo labeling.

Overall, these results indicate that human agreement imposes a practical upper bound on performance, which the proposed pseudo-labeling strategy already closely approaches, supporting its reliability for training.