An integrated framework for ASR post-processing with error correction, rescoring, and text-speech matching

Automatic speech recognition (ASR) technology transforms human speech into text, enabling hands-free, intuitive operation of various devices (Hinton et al., 2012; Jelinek, 1976; Rabiner and Juang, 1993). Its integration into systems allows users to control computers and other devices via voice commands, significantly enhancing usability and accessibility. ASR is pivotal in applications such as virtual assistants, smart home devices and interactive entertainment. Recent advancements in deep learning have enabled modern speech recognition models, trained on large and diverse data sets, to achieve near-human-level accuracy (Ao et al., 2022; Hannun et al., 2014; Radford et al., 2023). However, deploying these models on resource-constrained devices poses significant challenges due to their large parameter sizes. Additionally, cloud-based speech recognition services are not universally viable, as they face issues related to domain mismatch, privacy concerns and operational costs. Addressing these challenges by improving recognition accuracy while maintaining model efficiency remains a key research focus.

Techniques such as N-best reranking (Cai et al., 2023; Chiu and Chen, 2021b; Fohr and Illina, 2021; Salazar et al., 2020; Shin et al., 2019; Shivakumar et al., 2023; Udagawa et al., 2022; Xu et al., 2022) and error correction (Du et al., 2022; Dutta et al., 2022; Errattahi et al., 2018; Leng et al., 2023; Leng et al., 2021b; Leng et al., 2021a; Ma et al., 2023a; Mani et al., 2020; Zhao et al., 2021; Zhu et al., 2021) serve as effective post-processing modules to improve speech recognition performance without modifying the ASR model architecture, training methodology or inference pipeline (Li et al., 2023; Li et al., 2019; Wang et al., 2022b). This decoupling from the ASR system is a critical advantage, enabling these methods to be flexibly applied across different ASR systems.

N-best reranking approaches reevaluate the N-best hypotheses generated during ASR decoding. Instead of selecting the top-one hypothesis as the final result, these methods try to identify the hypothesis with the lowest error rate from N-best candidates as the output. In contrast, error correction models directly revise the ASR output by applying substitutions, insertions or deletions. Unlike N-best reranking methods, error correction models are not constrained by the set of N-best candidates and can leverage a broader vocabulary, making them particularly effective for addressing errors involving rare or out-of-vocabulary words. While error correction models offer greater flexibility and the potential for higher accuracy, N-best reranking methods are inherently more stable due to their limited search space.

Given the distinct strengths of N-best reranking methods and error correction models, their complementary nature suggests that combining these approaches could yield superior results. Building on this idea, several studies have demonstrated the effectiveness of such combinations on various benchmarks (Kang et al., 2024; Kuo and Chen, 2022; Leng et al., 2021b). However, a notable challenge arises: error correction models often overcorrect, producing hypotheses with strong linguistic coherence that can mislead the scoring during reranking. In other words, hypotheses modified by error correction models may be grammatically sound and semantically coherent yet deviate from the original speech content.

To address this challenge and advance the field, we propose a novel ASR post-processing framework in this study. The framework integrates a text correction module, a text rescoring module and a text–speech matching module in a pipeline. First, the error correction module generates new hypotheses by refining the N-best candidates produced by ASR. Next, the text rescoring module evaluates the quality of the wording and semantic coherence using text–level cues. Finally, to prevent hypotheses from drifting away from the original speech content, the text–speech matching module compares the hypotheses against the speech signal, ensuring alignment and enhancing overall accuracy. Furthermore, to implement the text–speech matching module, we design and present three distinct mechanisms, discussing their strengths and limitations in our experiments. The proposed framework integrates correction, rescoring and text–speech matching, which we denote as CREAM for short. In our experiments, CREAM is evaluated on the HypR benchmark (Wang et al., 2024) using the AISHELL-1 (Bu et al., 2017), TEDLIUM-2 (Rousseau et al., 2014) and LibriSpeech (Panayotov et al., 2015) data sets to validate its feasibility across domains and languages.

To enhance recognition performance, numerous methods have been proposed to revise ASR hypotheses, which can be broadly categorized into two major approaches: N-best reranking and error correction.

The primary objective of N-best reranking is to assign a reliable score to each hypothesis, ensuring that the score strongly negatively correlates with the error rate, where hypotheses with higher scores should correspond to lower error rates. By ranking hypotheses based on these scores, the optimal candidate from the N-best list is selected as the final output. Depending on the scoring granularity or computation approach, these methods can be classified into token-level, sentence-level and comparison-based approaches (Wang et al., 2024).

Token-level methods use advanced language models to compute scores at the token level, evaluating the probability of each token within a hypothesis. Two key paradigms are causal language modeling (CLM) (Jelinek, 1998; Jurafsky and Martin, 2000) and masked language modeling (MLM) (Kenton and Toutanova, 2019; Xu et al., 2022). For a hypothesis $X={x1,x2,…,xL}$⁠, CLM calculates the score by decomposing X into a sequence of conditional probabilities using the chain rule:

where $x<i$ represents the preceding tokens. While mathematically sound, CLM considers only unidirectional context, ignoring future tokens. In contrast, MLM leverages bidirectional context, computing scores based on the entire hypothesis:

where $x\i$ denotes the hypothesis with $xi$ masked.

Sentence-level methods represent entire hypotheses as vectors and convert these representations into scores for ranking (Chiu and Chen, 2021a; Xu et al., 2022). During training, ground truths such as error rates, rankings or token-level scores are used to guide the model parameter updates through supervised learning. Compared to token-level methods, sentence-level approaches treat each hypothesis as a single unit, reducing computational overhead during inference. Moreover, advanced sentence embedding techniques can distill lexical, semantic and latent information into compact vector representations (Conneau et al., 2017; Kenton and Toutanova, 2019; Logeswaran and Lee, 2018), making sentence-level methods both practical and efficient for generating reliable scores.

Comparison-based methods reformulate N-best reranking as a pairwise comparison task (Fohr and Illina, 2021; Illina and Fohr, 2021; Ogawa et al., 2018). These methods evaluate hypotheses by performing one-on-one comparisons, identifying the candidate with the most “wins” as the best option. This approach simplifies the reranking process by focusing on relative correctness, avoiding the complexities of absolute scoring. However, comparison-based methods often come with higher computational costs due to the large number of pairwise evaluations.

Error correction models improve ASR performance by directly detecting and correcting recognition errors. These models typically operate in a pipeline combining error detection and correction modules (Leng et al., 2023; Leng et al., 2021b; Leng et al., 2021a; Wang et al., 2022a). With advances in deep learning, research has increasingly focused on end-to-end sequence-to-sequence frameworks that streamline the process and minimize error propagation. Recently, modern approaches often leverage powerful pre-trained language models, such as BART and bidirectional encoder representations from transformers (BERT), as foundational components to enhance performance (Dutta et al., 2022; Hrinchuk et al., 2020; Shunmuga Priya et al., 2022; Zhao et al., 2021). To further improve robustness, some methods incorporate additional information from the ASRN-best list, expanding the capabilities of error correction models (Ma et al., 2023a; Zhu et al., 2021). Non-autoregressive decoding techniques have also been explored to accelerate inference while maintaining accuracy (Leng et al., 2023; Leng et al., 2021b; Leng et al., 2021a).

Key differences between N-best reranking and error correction models can be summarized as follows. First, error correction models generate new outputs that may not belong to the original N-best set, potentially surpassing the performance of N-best reranking methods confined to the predefined hypotheses. Second, pipeline or end-to-end error correction models are inherently more complex to train, whereas N-best reranking methods usually rely on simpler, well-established training techniques. Third, error correction models depend solely on textual clues without direct reference to the input speech, increasing the risk of introducing unexpected errors. In contrast, N-best reranking methods are more conservative and stable.

Beyond N-best reranking and error correction, recent research has explored hybrid and alternative strategies to further enhance ASR performance. Leveraging word lattices as input expands the search space for candidate hypotheses, improving recognition outcomes (Pandey et al., 2022; Tsai et al., 2022). However, encoding lattice structures is computationally intensive and typically requires large supervised data sets (Dai et al., 2022; Ma et al., 2020).

The rapid development of large language models (LLMs) has introduced zero-shot and one-shot strategies for revising ASR hypotheses, achieving remarkable performance (Brown et al., 2020; Chen et al., 2024; Ma et al., 2023b; Ouyang et al., 2022; Touvron et al., 2023a; Touvron et al., 2023b; Workshop et al., 2022; Yang et al., 2023). For N-best reranking, LLMs have been used to select the best hypothesis directly from the list using in-context learning, often with demonstrations in the prompt to improve task understanding (Chen et al., 2024; Ma et al., 2023b; Wang et al., 2024).

LLMs have also been applied to error correction tasks, using advanced techniques such as multi-task learning and prompt engineering to achieve state-of-the-art results (Chen et al., 2023; Ma et al., 2023b; Min and Wang, 2023; Nie et al., 2022; Pu et al., 2023). Additionally, cross-modal fusion technologies that integrate LLMs with pre-trained speech models offer promising directions for ASR post-processing (Radhakrishnan et al., 2023; Wang et al., 2024). However, the large parameter sizes and high computational demands of these methods remain significant considerations.

Recent ASR post-processing research has shown a growing interest in pipeline-based architectures, where multiple specialized modules are cascaded to incrementally refine recognition outputs. As summarized in Table 1, these frameworks generally consist of two or more stages, such as uncertainty estimation, text correction, rescoring or acoustic consistency checking. Their modular design allows each component to address specific error types and facilitates interpretability during inference.

Early studies like Cross-Modal ASR Post-Processing (Du et al., 2022) introduced a dual-stage pipeline combining confidence estimation with cross-modal error correction, explicitly integrating both textual and acoustic cues. Subsequent works extended this paradigm toward multi-stage language-model-driven correction. For instance, Multi-stage LLM Correction (Pu et al., 2023) performs uncertainty estimation on N-best hypotheses followed by rule-guided LLM refinement, whereas ClozeGER (Hu et al., 2024) and multi-modal and multi-granularity generative error correction (MMGER) (Mu et al., 2024) exploit multimodal and multigranular modeling to bridge textual and acoustic modalities. SpeechLLM-based approach (Prakash et al., 2025) further combines multiple ASR outputs with generative reasoning to form pseudo-labeled data for self-improvement.

Other frameworks focus primarily on text–level enhancement. Tag and Correct (Ziętkiewicz, 2022) uses a tagging–correction pipeline to achieve high-precision localized edits and the Pinyin Regularization method (Tang et al., 2024) introduces phonetic normalization before LLM-based refinement to handle Chinese homophone ambiguities. Meanwhile, prompting-based designs such as LLM Error Correction with Task-Activating Prompting (Yang et al., 2023) explore the capability of LLMs to act as both rerankers and correctors under zero-shot or few-shot conditions.

From a comparative standpoint, most existing pipelines rely on N-best hypotheses and external language models for reranking and text–level correction, while only a subset – such as Cross-Modal ASR Post-Processing, ClozeGER and MMGER – explicitly leverage speech information. This trend highlights the field’s transition from single-model correction toward more modular, multimodal and LLM-augmented post-processing paradigms.

Building upon previous research, we propose a novel post-processing framework for ASR that seamlessly integrates error correction, rescoring and acoustic information to create a more stable and accurate system. More formally, this framework consists of three interconnected modules: the text correction module, the text rescoring module and the text–speech matching module, collectively referred to as CREAM. An overview of the framework is illustrated in Figure 1. These modules operate sequentially: the text correction module first refines the original ASRN-best hypothesis list by generating enhanced hypotheses. Subsequently, the text rescoring module evaluates the linguistic coherence and semantic quality of these hypotheses. Finally, the text–speech matching module ensures that the hypotheses align with the original speech signal. Together, these modules collaboratively identify the most accurate hypothesis, leveraging both language- and acoustic-based criteria to determine the final output.

In the first stage, the text correction module rewrites each hypothesis to generate a new set of candidate hypotheses. Its objective is to replace recognition errors with correct predictions, ideally aligning the new hypotheses with the ground truth. Formally, given the original ASR-generated hypotheses $YASR={Y1ASR,…,YnASR,…,YNASR}$⁠, the text correction module generates a new set of refined hypotheses $YTC={Y1TC,…,YnTC,…,YNTC}$⁠:

where $TC(·)$ denotes the text correction model. To maximize the capacity of the candidates and avoid missing plausible hypotheses, the original and refined sets are merged for input into subsequent modules:

We adopt BART and FastCorrect models for implementing the text correction module (Leng et al., 2021a; Zhao et al., 2021). BART refines hypotheses autoregressively, while FastCorrect applies non-autoregressive corrections. Implementation details are discussed in Section 4.2.

After the text correction module, a new set of hypotheses $Y$ is obtained and is expected to have hypotheses that are close to the ground truth. In the following step, the text rescoring module is introduced to evaluate each hypothesis based on text–level regularity and semantic coherence:

where $TR(·)$ represents the text rescoring model and $Yi∈Y$⁠.

To balance performance and efficiency, we use BERT as the underlying model to compute perplexity scores for each hypothesis (Kenton and Toutanova, 2019; Udagawa et al., 2022; Xu et al., 2022). By leveraging its bidirectional contextual understanding, BERT enables efficient hypothesis rescoring. Implementation details are provided in Section 4.3.

Although the hypotheses generated by the text correction module are typically fluent and semantically coherent, they may deviate from the original speech content due to the lack of acoustic information in the correction process. Such deviations are challenging to detect using the text rescoring module, as it also relies solely on text–level information. To address this, we propose the text–speech matching module to calculate the degree of alignment between each hypothesis and the corresponding speech utterance. We introduce three implementations of the module.

Using an ASR model is a straightforward approach. For a hypothesis $Yi$ and its corresponding speech utterance $X$⁠, the text–speech matching score $SASRTS(Yi)$ is defined as the ASR decoding score. For example, in a connectionist temporal classification (CTC)-attention ASR model, the text–speech matching score is computed as (Watanabe et al., 2017):

where $α$ is a weighting factor and $sctc(Yi|X)$ and $satt(Yi|X)$ represent the CTC and attention scores, respectively. The CTC score is defined by taking the log of the summation of probabilities for all alignments $Φ$ between $Yi$ and $X$⁠:

The attention score is computed by summing the log probability of every token $wi$ conditioned on its previous tokens:

where L denotes the number of tokens in hypothesis $Yi$⁠.

Inspired by recent advancements in multi-modal research (Chen et al., 2022c; Tsimpoukelli et al., 2021), we use the masked audio-text encoder (MATE) model as a judge to calculate the text–speech matching score (Cai et al., 2023). MATE integrates three key components: a pre-trained masked language model (BERT), a pre-trained speech encoder [HuBERT (Hsu et al., 2021) or WavLM (Chen et al., 2022b)] and a convolutional neural network (CNN)-based multi-modal alignment module. For a given speech utterance $X$ and hypothesis $Yi$⁠, MATE uses WavLM to convert $X$ into a sequence of feature vectors and the embedding layer of BERT to map $Yi$ into a series of token embeddings. To reconcile the sequence lengths of the two modalities, the multi-modal alignment module downsamples the speech feature vectors and projects them into the same embedding space as the BERT token embeddings. These aligned features from both modalities are concatenated and processed by BERT, enabling the fusion of information from both speech and text. MATE is trained using the masked language model objective and contrastive loss. The former encourages the model to predict masked tokens in the text sequence, leveraging contextual and cross-modal information. The latter promotes greater similarity between paired acoustic and lexical representations while discouraging similarity between unpaired representations. This helps MATE learn robust matching relationships between text and speech. Detailed descriptions of MATE architecture and training processes can be found in Cai et al. (2023).

Using MATE, the text–speech matching score for a speech-hypothesis pair (⁠$X,Yi={w1,…,wL}$⁠) is computed in a MLM manner:

where $Yi,\l$ denotes the hypothesis $Yi$ with the token $wl$ replaced by the special mask token. The underlying principle of this scoring mechanism is straightforward: hypotheses that are both textually fluent and acoustically consistent with the speech signal will receive higher scores, while those that are incoherent or misaligned with the acoustic input will score lower. Notably, MATE not only models the relationships between text and speech but also incorporates a strong language modeling perspective, enhancing its ability to evaluate semantic and lexical coherence alongside acoustic alignment.

While using an ASR model to calculate the text–speech matching degree is intuitive and effective, this approach is computationally expensive and requires a large number of parameters. To address these challenges, we propose a lightweight model trained to distill knowledge from an ASR system and efficiently evaluate the matching degree. We refer to this model as MIMICA because it is designed to mimic an ASR.

To reduce the computational overhead while considering acoustic-level information, MIMICA leverages phoneme sequences derived from the N-best list as acoustic references. During training, for a given speech utterance $X$ and its N-best hypotheses $YASR={Y1ASR,…,YNASR}$⁠, each hypothesis $YnASR$ is converted into a phoneme sequence $HnASR$ using a grapheme-to-phoneme (G2P) model. These phoneme sequences are concatenated to form a single acoustic reference:

For a target hypothesis to be scored, its phoneme sequence is concatenated with the acoustic reference as the input to MIMICA. Following the BERT-style architecture, a [CLS] token is prepended for the score prediction and a [SEP] token is used to separate the hypotheses. Segment embeddings are used to differentiate the target hypothesis (assigned 0) and the acoustic reference (assigned 1). MIMICA is implemented as a multi-layer Transformer model followed by a simple linear layer for score prediction. After multi-round interactions among the phoneme units of the target hypothesis and the acoustic reference, the final hidden state of the [CLS] token is passed through the linear layer to come up with the text–speech matching score:

where $hn,[CLS]ASR$ is the final hidden state of the [CLS] token for the target hypothesis $YnASR∈YASR$ and W represents the weight matrix of the linear layer. The training objective of MIMICA is to minimize the mean squared error between the decoding score $SASR(YnASR)$ calculated by ASR as defined in equation (6) and the predicted score $SMIMICATS(YnASR)$⁠:

The model architecture is depicted in Figure 2.

During inference, MIMICA computes the text–speech matching score for each hypothesis $Yi∈Y$ by concatenating its phoneme sequence $Hi$ with the corresponding acoustic reference $H1:N$⁠. The inference process uses only phoneme sequences, eliminating the need for acoustic features paired with a full ASR system. In summary, MIMICA provides a lightweight and efficient solution by leveraging phoneme sequences from N-best hypotheses as acoustic references to calculate text–speech matching scores. Compared to using an ASR model directly, MIMICA significantly reduces computational complexity while maintaining robust performance.

The CREAM framework integrates three modules – text correction, text rescoring and text–speech matching – to achieve robust ASR post-processing. The process begins with the text correction module, which expands the pool of candidate hypotheses by refining the original N-best list. Next, each hypothesis is evaluated based on text–level token regularity and semantic coherence by the text rescoring module, as well as the alignment between text and speech by the text–speech matching module. The final score for each hypothesis is computed by combining the rescoring and matching scores using a weighted sum. The hypothesis with the highest combined score is selected as the final output:

where $STR(Yi)$ represents the text rescoring score, $STS(Yi)$ represents the text–speech matching score and $β$ is a tunable hyperparameter that adjusts the relative influence of the two scores. This framework ensures that the selected hypothesis balances both linguistic quality and acoustic alignment, resulting in a more accurate and reliable ASR output.

It is worth noting that CREAM also follows a pipeline-style design, similar in spirit to many recent ASR post-processing frameworks summarized in Table 1. However, unlike prior systems that typically focus on one or two isolated components, CREAM integrates three complementary modules into a unified and ASR-agnostic architecture. Specifically, it performs text correction, text rescoring and text–speech matching sequentially, while maintaining modular independence that allows each component to be optimized or replaced without affecting the overall framework. This unified yet flexible pipeline enables CREAM to fully exploit textual, contextual and acoustic information, thereby achieving a more balanced tradeoff between correction accuracy, acoustic consistency and computational efficiency.

In this study, we adopted the publicly available speech recognition hypothesis revising benchmark, HypR ^[1], to evaluate the performance of our proposed CREAM framework. HypR provides speech recognition results for several publicly available speech corpora, with the speech recognition model for each data set using a Transformer-based CTC/attention architecture built with ESPnet (Watanabe et al., 2018). To evaluate our framework under varying ASR quality, we leverage recognition results from HypR with and without language models to simulate low-error and high-error scenarios, respectively. Our experiments were conducted on three public speech corpora included in the benchmark: the Chinese corpus AISHELL-1 (Bu et al., 2017) and the English corpora TEDLIUM-2 (Rousseau et al., 2014) and LibriSpeech (Panayotov et al., 2015). For each speech utterance, we extracted the top 10 hypotheses to train, tune and test all models compared in the experiments. For each corpus, we used the corresponding training sets to train the models, selected the best model checkpoints and hyperparameters based on the development set performance and finally evaluated the results on the test sets. Table 2 summarizes the statistics of the data sets used in our study.

We adopted BART and FastCorrect models to evaluate the versatility of our proposed framework, as these two models use different modeling philosophies. BART is a classical end-to-end sequence-to-sequence autoregressive model, while FastCorrect is a non-autoregressive model when processing generation. For BART, we fine-tuned the text correction models using pre-trained Chinese ^[2] and English ^[3] checkpoint models individually. However, since only the Chinese ^[4] checkpoint model is available for FastCorrect, experiments with this model were conducted exclusively on the AISHELL-1 data set. The vocabulary sizes of BART are 51,271 for Chinese and 50,265 for English, whereas FastCorrect has a vocabulary size of 32,652 for Chinese. The fine-tuning configurations are summarized in Table 3.

BERT has demonstrated its effectiveness in various tasks, including serving as a language model for speech recognition (Shin et al., 2019; Udagawa et al., 2022; Xu et al., 2022). In this study, we use BERT to implement the text rescoring module and compute the perplexity score for each hypothesis (Xu et al., 2022). Publicly available pre-trained models for Chinese ^[5] and English ^[6] are used to initialize the model parameters. These models are then fine-tuned using the MLM loss with a $15%$ masking rate applied to each training text sequence. The vocabulary sizes of BERT are 21,128 for Chinese and 30,522 for English. The fine-tuning configurations are summarized in Table 4.

In this study, we explore three approaches for implementing the text–speech matching module: the ASR model, MATE and MIMICA. For the ASR model, we used the pre-trained ASR model provided by the HypR benchmark, keeping its parameters frozen during the experiments. The interpolation weight $α$ between the CTC and attention scores [cf. Equation (6)] was adjusted for each data set based on the settings in the HypR benchmark.

For MATE, we followed the methodology outlined in the previous study to combine BERT with a pre-trained speech encoder to construct the multi-modal matching model (Cai et al., 2023). HuBERT ^[7] and WavLM ^[8] were used as the speech encoders for Chinese and English, respectively. On top of the speech encoder, the multi-modality alignment component was built using three layers of CNN with 768 channels, strides (2, 1, 2), kernel widths (3, 1, 1) and a bottleneck adapter layer with a compression factor of 0.5. The fine-tuning hyperparameters are summarized in Table 5.

For MIMICA, hypotheses were converted to their phoneme sequences using the pypinyin ^[9] toolkit for Chinese, whereas a G2P ^[10] model was used to generate phoneme sequences for English (Lee et al., 2018). The main structure of the model is built using a stack of twelve Transformer layers. During training, MIMICA learned to predict the decoding scores calculated by the ASR model. During inference, it predicted the text–speech matching score for each hypothesis in $Y$⁠. Table 6 summarizes the hyperparameters used for fine-tuning.

In the first set of experiments, we discuss the performance of our proposed CREAM framework and compare the results with basic baselines. Each of the three data sets is tested under both high-error and low-error scenarios, depending on whether an language model (LM) is used during the ASR decoding stage. Tables 7 and 8 summarize the experimental results. Several noteworthy observations emerge from these experiments. First, Top-1 represents the ASR output without any post-processing and Oracle refers to selecting the hypothesis with the lowest error rate from the 10 best candidates for each speech utterance. Comparing the two baselines, the performance gap highlights room for improving speech recognition results and the potential for ASR post-processing research. Second, we present the performance achieved by BART, FastCorrect and BERT as references. BART and FastCorrect are error correction models, whereas BERT is an N-best reranking method. In the high-error condition, all three models effectively enhance recognition performance. However, in the low-error scenario, BART fails to outperform Top-1, while FastCorrect and BERT still demonstrate improvements in most cases. These results indicate that although ASR post-processing typically improves performance, error correction models may sometimes degrade accuracy compared to the original ASR output. This occurs because such models rely solely on textual cues when making corrections, thereby increasing the risk of introducing spurious errors due to the lack of direct reference to the input speech. Table 9 presents several correction examples produced by BART, along with the corresponding ASR output (Top-1) and the ground truth (GT) for comparison. As shown, without guidance from acoustic information, error correction models like BART may mislead the original ASR output toward commonly occurring phrases such as “办公,” “projecting,” or “god bless you.” However, these phrases drift entirely away from the actual spoken content.

Next, we turn the focus to the proposed CREAM framework, which has different settings because the text correction module and text–speech matching module can be implemented by using various strategies. Among all configurations, the text rescoring module is implemented using BERT. At first glance, regardless of the specific implementation, CREAM consistently outperforms the baselines in most cases. Upon closer examination, FastCorrect proves to be more effective than BART for the text correction module, aligning with their standalone performance. Regarding the text–speech matching module, ASR outperforms both MATE and MIMICA in most settings. This is expected, as MIMICA is designed as a lightweight alternative to ASR, predicting the matching scores without relying on acoustic features and an ASR model during inference. As a result, although MIMICA does not obtain superior results, it has very scalable usability, flexibility and generalization capabilities. MATE, on the other hand, is a multi-modality mixture model that functions as an acoustic-enhanced language model. Its advantages over ASR include simplicity during inference and the absence of the need for an ASR model for alignment. However, compared to MIMICA, MATE relies on acoustic feature vectors and its computational cost is higher due to token-wise probability predictions.

Each strategy for implementing the text–speech matching module has distinct characteristics. ASR evaluates the matching degree by calculating the alignment between acoustic features and text, providing a classical and reliable approach. MIMICA translates hypotheses into phoneme sequences and evaluates matching scores from a phoneme-level perspective, offering a lightweight and efficient solution. MATE enriches text with acoustic features to build an acoustic-enhanced language model, leveraging masked token prediction for scoring. We are thus curious about whether they could complement each other and enhance the performance further. Therefore, we combined all three methods to calculate the text–speech matching score using a simple linear weighted combination. When paired with either BART or FastCorrect, this combined strategy achieves the best results across all cases, as shown at the bottom of Tables 7 and 8. In conclusion, integrating ASR, MIMICA and MATE efficiently enhances the text–speech matching score. The combination leverages the unique strengths of each method, resulting in improved performance across all tested scenarios. These results demonstrate the potential for a comprehensive model that benefits from the diverse attributes of the individual components.

In this ablation study, we evaluate the contribution of each component within the CREAM framework and analyze the impact of the N-best list size. AISHELL-1 is selected as a representative data set and the best configuration, in which BART is used for the text correction module, BERT for the text rescoring module and ASR for the text–speech matching module, is adopted for experimental validation.

We first use AISHELL-1 as a case study to investigate the contribution of each component in the CREAM framework, with the results summarized in Table 10. Six configurations are listed and numbered from 1 to 6. Configurations 1 and 2 demonstrate the individual contributions of the text–speech matching and text rescoring modules, respectively. Configurations 3, 4 and 5 combine any two of the three components, while Configuration 6 represents the case where all three modules are jointly applied. At first glance, the results may seem inconclusive; however, several noteworthy patterns emerge upon closer examination.

When the text correction module is fixed (either enabled or disabled), we further examine the experimental results by enabling only the text rescoring module (i.e. no. 2 and 5) or only the text–speech matching module (i.e. no. 1 and 4) to compare their independent effects. The findings reveal that these two modules exhibit complementary behavior under different word error rate conditions. In the high-error scenario, the text rescoring module achieves better performance; specifically, experiment no. 2 outperforms no. 1 and no. 5 outperforms no. 4. In contrast, in the low-error scenario, the text–speech matching module proves more effective, as experiment no. 1 performs better than no. 2 and no. 4 surpasses no. 5. This is because, in high-error cases, candidate sentences often contain numerous mistakes and selecting the more fluent option can effectively reduce the error rate, making rescoring the dominant factor. In contrast, when the error rate is already low and the candidate sentences are mostly fluent, rescoring offers limited additional benefit, whereas the text–speech matching module becomes crucial by identifying and selecting the candidate most consistent with the acoustic signal.

Subsequently, when both the text rescoring and text–speech matching modules are enabled simultaneously (i.e. no. 3 and 6), the results – consistent with expectations – show that combining the two yields the best overall performance, regardless of whether the text correction module is included, since they exploit distinct yet highly complementary sources of information.

Next, we examine the role of the text correction module. In most configurations, incorporating this module improves performance. However, when it is combined with the text rescoring module but without the text–speech matching module (i.e. no. 5), its inclusion actually degrades performance, particularly under the low-error scenario. This observation suggests that the text correction module may sometimes transform recognition candidates into more fluent yet acoustically inconsistent sentences. When only textual scores are considered (as in the rescoring module), such misleading candidates may be preferentially selected, leading to a higher overall error rate.

Overall, these experiments validate the design of the CREAM framework. By appropriately organizing and integrating the modules, the framework effectively reduces ASR recognition errors through the complementary use of correction, rescoring and matching strategies.

We further analyze the impact of the N-best list size on recognition performance. For this experiment, we use the best configuration, where BART is used for the text correction module, BERT for the text rescoring module and ASR for the text–speech matching module. The experimental results are summarized in Table 11, with N-best list sizes of 5, 10, 20 and 50. From the results, we observe that CREAM achieves improved recognition accuracy as the N-best list size increases – the larger the candidate set, the lower the recognition error rate. This improvement occurs because a larger N-best list provides greater hypothesis diversity, increasing the likelihood of including a more accurate candidate. However, it should also be noted that the computational cost increases proportionally with the candidate size. Therefore, although larger N-best lists can further improve performance, the choice of candidate number should be balanced against the available computational resources and the desired level of accuracy.

In the third set of experiments, we compare the proposed CREAM framework with several baseline models, categorized into N-best reranking and error correction models. These models are implemented by the HypR project and detailed settings and configurations can be found in the HypR documentation (Wang et al., 2024). The results are shown in Table 12. Based on the results, N-best reranking methods generally outperform error correction models, with a noticeable performance gap. This discrepancy may stem from the fact that error correction models often produce outputs that are fluent, smooth and fluid but may drift from the original speech content. Within the N-best reranking methods, MLM outperforms CLM, as MLM leverages bi-directional contextual information, whereas CLM is limited to uni-directional modeling. While RescoreBERT is more time-efficient than MLM, this comes at the cost of reduced performance. Among the reranking methods, PBERT achieves the best performance due to its contrastive training objective, which effectively distinguishes the hypothesis with the lowest word error rate from the others (Chiu and Chen, 2021b). Interestingly, the LLM-Reranking method, which uses a large language model (LLM) to select the best hypothesis, still leaves room for improvement.

Regarding error correction models, BART and FastCorrect serve as representative examples. On the AISHELL-1 data set, FastCorrect demonstrates superior performance compared to BART. However, compared with Top-1 listed in Tables 7 and 8, BART also achieves consistent improvements across both Chinese and English corpora.

Despite the widespread success of LLMs in various applications, using an LLM to address speech recognition errors (i.e. the LLM-Correction model) does not yet yield competitive results on these data sets. The results suggest that the use of LLMs exhibits certain instability. To replicate successful outcomes across different data sets, further exploration is needed in areas such as better prompt design and fine-tuning strategies, which may help achieve more satisfactory and stable results.

Finally, when comparing CREAM to these baselines, the proposed framework consistently achieves the best results across all data sets and settings (i.e. with and without LM). The superior performance of CREAM can be attributed to its meticulous design, which incorporates an error correction module to expand the pool of candidates and reweights each hypothesis based on token regularity and text–speech matching perspectives. By carefully integrating these components, CREAM effectively minimizes recognition errors and demonstrates its robustness across diverse scenarios.

Although CREAM outperforms most baselines in accuracy, its main limitations lie in model size and computational latency. Table 13 reports the parameter counts and the average computation time per utterance (in seconds) for several baselines and for CREAM. Overall, CREAM is approximately 2–3× larger than the baselines, primarily because it consists of three modules. In the text correction module, BART and FastCorrect have about 140 M and 65 M parameters, respectively. The text rescoring module, implemented with BERT, contains about 102 M parameters. For the text–speech matching module, we present three variants – ASR, MATE and MIMICA – with sizes of roughly 30 M, 200 M and 102 M parameters, respectively.

Beyond parameter counts, CREAM’s computational latency also varies depending on the chosen configuration. Generally, using FastCorrect for text correction is faster and smaller than BART. Among the text–speech matching variants, MATE is both the largest and the slowest, while MIMICA runs faster than ASR but has a slightly larger model size. Because CREAM operates in a pipeline-style workflow, inference incurs cumulative latency across modules, resulting in relatively longer per-utterance processing time. Although the text rescoring and text–speech matching modules can, in principle, be executed in parallel to increase throughput, we measure latency sequentially to reflect the worst-case scenario. Nevertheless, reducing CREAM’s parameter footprint and improving inference speed remain important directions for future work.

Though ASR post-processing models generally enhance recognition performance, we are curious whether the corrections primarily involve common function words or more meaningful named entities. To investigate this, we conducted a set of experiments on the AISHELL-NER data set (Chen et al., 2022a), which is based on the AISHELL-1 corpus and includes manually annotated named entities. The test set contains 898 person names, 1,330 location names and 1,165 organization names. Table 14 summarizes the named entity recall rates of various models.

First, all models outperform the Top-1 baseline. We also observe a consistent trend where lower ASR error rates are linked to higher NER accuracy. However, this relationship is not entirely linear. For example, in the high-error scenario, FastCorrect achieves a lower speech recognition error rate than BART, yet BART attains a higher named entity recall rate. Moreover, within the CREAM framework, the choice of the text–speech matching module affects named entity recognition. When BART is used for the text correction module, pairing it with ASR as the text–speech matching module results in a lower speech recognition error rate compared to MATE or MIMICA. However, the named entity recall rate is higher when BART is paired with MATE. Similarly, when FastCorrect is used for text correction, MATE remains the superior choice for the text–speech matching module, yielding better named entity recognition results than ASR or MIMICA in both low-error and high-error scenarios. These findings implicitly suggest that a greater contribution from the language model leads to improved named entity recognition results. Therefore, MATE, which functions as an acoustic-enhanced language model, tends to outperform both ASR and MIMICA. Finally, combining ASR, MATE and MIMICA achieves the best named entity recall rates in both low-error and high-error cases, as expected. This set of experiments demonstrates that the CREAM framework effectively enhances recognition results for both critical components, such as named entities and commonly used words in speech utterances.

Table 15 presents a real example extracted from the AISHELL-1 data set. For this speech utterance, the ground truth is “盲目捐款没有益 (i 4) 处”. Three corresponding speech recognition hypotheses are labeled as (1), (2) and (3). Hypothesis (1) is the perfect result. Hypothesis (2) contains the homophone “易 (i 4)” and is phrased awkwardly. Hypothesis (3) includes the homophone “意 (i 4)” and a substitution error “义 (i 4),” yet the overall sentence remains coherent. The scores for these three hypotheses, evaluated by the text rescoring module and the text–speech matching module, are listed in the table. The text rescoring module assigns scores based solely on textual information, whereas the text–speech matching module evaluates the degree of alignment between speech and text. Due to their differing focuses, some discrepancies in scoring arise. For example, the grammar and semantics of hypothesis (3) are correct and reasonable, while those of hypothesis (2) are problematic. Consequently, the text rescoring module assigns a higher score to hypothesis (3) than to hypothesis (2). In contrast, the text–speech matching module assigns a higher score to hypothesis (2) than to hypothesis (3), as it evaluates the alignment between text pronunciation and speech. Additionally, while both “没有意义” and “没有益处” are valid expressions, “没有意义” is more commonly used in Chinese. As a result, the text rescoring module assigns a slightly higher score to hypothesis (3) than to hypothesis (1). This example demonstrates that our proposed CREAM framework, by integrating the two scoring modules to evaluate correctness from both perspectives, is not only a reasonable and feasible approach but also explains why it can achieve superior results.

In this study, we explored ASR post-processing technology and proposed a novel framework called CREAM, which comprises three key components: the text correction module, the text rescoring module and the text–speech matching module. For the text–speech matching module, we introduced three different implementation methods, each with distinct advantages, limitations and suitable use cases. We evaluated the performance of CREAM on three widely used data sets, demonstrating its effectiveness and robustness. In future work, we plan to explore potential modifications and optimizations for each module, including leveraging LLMs to enhance their implementation. We also aim to investigate the feasibility of transforming CREAM into an end-to-end framework and test its performance on more challenging data sets. Ultimately, we aspire to adapt this framework to a broader range of applications, ensuring consistent and robust improvements in speech recognition performance.