Integration of multiple acoustic information included in end-to-end ASR models via density ratio approach for Japanese language

Integrating the outputs of multiple automatic speech recognition (ASR) systems has been shown to be an effective technique for improving ASR performance. Previous studies have investigated various ASR integration methods when using hidden Markov model (HMM)-based ASR systems. For example, NIST recognizer output voting error reduction (ROVER) method integrates the output of multiple large vocabulary continuous speech recognition (LVCSR) systems using a voting module (Fiscus, 1997). Other methods use the outputs of multiple LVCSR systems in a machine-learning manner, for example, using a support vector machine (SVM) (Utsuro et al., 2004). These multiple model ASR systems have outperformed single model systems, demonstrating the effectiveness of this approach. Robust acoustic models have also been built by integrating different types of models, for example, large-scale models, models trained using noisy speech and models trained with different styles of speech (Kang and Kwon, 2016). Researchers have also recently proposed integrating a hybrid DNN–HMM ASR model and an end-to-end (E2E) ASR model at the hypothesis level to improve recognition performance (Wong et al., 2020), while other investigators have integrated ASR models with completely different model architectures, such as a hybrid ASR model and a GMM–HMM ASR model (Swietojanski et al., 2013).

In contrast, only a few studies have explored combining multiple E2E ASR models (Tuske et al., 2021). This is because it is generally believed that the huge number of training parameters of a single E2E ASR model allow a sufficient degree of freedom to process speech data from a variety of domains, including speech data containing noise and multiple styles of speech, so it is unnecessary to integrate the output of multiple E2E ASR models.

But, when inference is performed on a domain different from the training data, the accuracy of E2E ASR models is generally significantly degraded. As a results, a common countermeasure is to fine-tune a pretrained E2E ASR model with a small amount of target domain data. Fine-tuning with a small amount of domain speech is effective, but requires the use of GPU computational resources and annotation. In addition, unlike conventional HMM systems, in which acoustic, lexical and language models (LMs) can be replaced individually, E2E ASR models embeds these functions in a single network, making domain migration more difficult. To address this problem, researchers have also explored the use of E2E ASR models in combination with external LMs. As a result, the recognition performance of large-scale E2E ASR models has steadily improved over the years.

Shallow fusion (SF) is one method frequently used to integrate an ASR model and an external LM (Kannan et al., 2018; Hori et al., 2017). This allows easy integration without re-training, unlike other integration methods such as deep fusion (Gulcehre et al., 2018), cold fusion (Sriram et al., 2018) and component fusion (Shan et al., 2019). The density ratio approach (DRA) (McDermott et al., 2019; Variani et al., 2020), which is a stochastic improvement of SF, is a more recently proposed integration method which considers the implicit language information contained inside the trained ASR model. It has been reported that external LMs can approximate the language information contained in an ASR model for English (McDermott et al., 2019; Variani et al., 2020), Spanish, Italian (Saon et al., 2021) and Japanese (Mori et al., 2024; Takagi et al., 2024) and that the DRA can be used to replace the language information implicitly acquired by the ASR model with language information from a different domain.

Based on the theoretical concept of the DRA, in this paper we explore extracting only pure acoustic information from an E2E ASR model. This extraction allows us to combine the acoustic information contained within multiple existing ASR models with language information from an external target domain LM using the DRA, under the assumption that there may be situations in which it is not feasible to create a large E2E ASR model for a particular target domain. By combining acoustic information from multiple existing E2E ASR models with language information from a target domain LM, we thought it might be possible to create an ASR model that is acoustically robust for the target domain, improving speech recognition performance. Therefore, our proposed method extracts multi-domain acoustic information from multiple E2E ASR models and combines this with knowledge about the target domain obtained from an external LM. The main advantage of this method is that it leverages robust acoustic information from diverse domains without requiring large amounts of paired speech-text data. It can be applied using an LM trained on text data from the source and target domains, along with a small amount of paired data for hyperparameter tuning, thereby reducing the cost of collecting new paired data.

In our previous work (Hojo et al., 2023), we conducted experiments integrating two ASR models trained on different domains and demonstrated the effectiveness of system combination using a recurrent neural network (RNN)-based model (Watanabe et al., 2017) and Transformer-based model (Karita et al., 2019) as each ASR model. In this study, we increase the number of ASR models and compare the proposed method with conventional system integration methods by conducting additional ASR performance experiments, to confirm the robustness and generality of our approach across multiple domain combinations.

Our contributions are summarized as follows:

• This paper proposes a method for constructing an ASR model that integrates multi-domain acoustic information obtained from multiple existing ASR models with the linguistic information of a target domain from a LM, using DRA.

• Our experimental results confirm that our proposed method outperforms conventional LM integration methods, each of them uses a single ASR model, without any additional training.

• We demonstrated that integrating multiple ASR models trained with various source domains enhances the robustness of the acoustic information, leading to improved performance in the target domain.

Integrating the output results of multiple conventional GMM–HMM-based ASR systems has been shown to output more accurate hypotheses than a single system, as the error tendencies of each system complement each other. For example, ROVER (Fiscus, 1997) achieves a lower word error rate (WER) than single systems by aligning the outputs of multiple ASR systems and using majority voting to produce a final hypothesis, as shown in Figure 1. ROVER is composed of two modules; an alignment module and a voting module. First, a single composite word transition network (WTN) is created by dynamic programming of the WTNs obtained from the recognition results of multiple ASR systems, using the alignment module. Next, the best-scoring word sequence is selected from the created composite WTN using the voting module. ROVER does not require additional model training and has the advantage of easily combining different system outputs. Subsequent work has further enhanced ROVER with confidence-weighted voting and pattern-matching strategies to improve robustness (Schwenk and Gauvain, 2000; Alemzadeh et al., 2012).

In addition to voting-based methods, rescoring approaches that use SVMs, random forests and neural networks have also been proposed (Utsuro et al., 2004; Matsushita et al., 2003; Xu and Mangu, 2005; Song et al., 2019; Kang et al., 2024). Learning-based rescoring methods that incorporate bidirectional encoder representations from transformers (BERT) embeddings, perplexity features and other meta-features have reported relative WER reductions of up to 20% compared to conventional rescoring techniques (Song et al., 2019). Segment-level confidence estimates from an external quality estimation model have also been introduced into the ROVER voting process, demonstrating absolute WER reductions of up to 7.3% (Jalalvand et al., 2017). Other studies have proposed combining GMM–HMM models with DNN–HMM models or GMM–HMM models with E2E models, to integrate the output of ASR systems with different structures to take advantage of the strengths of both types of models (Wong et al., 2020; Swietojanski et al., 2013).

Beyond these output-level fusion techniques, researchers have also merged multiple acoustic models at the GMM-space level (Kang and Kwon, 2016). Unlike ROVER or rescoring, which operate on a final hypotheses or scores, this method integrates the output of acoustic models trained with different speech styles and environmental noise. The results of this research suggest that integrating models trained with acoustically diverse data is effective for achieving more robust ASR.

Although all of these integration methods have contributed to improving the robustness and accuracy of single systems, these studies do not focus on internal information, such as how the acoustic model and LM are processed within each system and integrates only information obtained from the final output, such as the final output word sequence or N-best hypotheses. Therefore, the common limitation of these integration methods is that they cannot directly take advantage of the various representations that each recognizer internally acquires and retains. Based on the above findings and on the effectiveness of output-level integration methods observed during the GMM–HMM era, this study proposes a new approach for integrating multiple ASR models that takes into account the information inside E2E ASR models.

Several LM fusion methods have been proposed (Kannan et al., 2018; Gulcehre et al., 2018; Sriram et al., 2018; Shan et al., 2019; McDermott et al., 2019). In this paper, we will only focus on techniques which do not require additional ASR model training, therefore we will only discuss the two methods which meet this condition, SF and DRA.

SF is the standard method used to integrate information from an ASR model and an external LM (Kannan et al., 2018). The output probabilities of the ASR model are added with those of the external LM in the logarithmic domain as follows:

where $P(y|x)$ is the output probability of the ASR model, that is, the probability of inferring symbol label y when acoustic feature x is given and where $Q(y)$ is the prior probability of the external LM. When using the SF method, the external LM is only used during inference. Note also that the ASR and LMs are trained independently.

The DRA is similar to SF, however it also takes the implicit language information contained within the LM into account during inference (McDermott et al., 2019). Conventional ASR models attempt to infer output symbol $y^$ as follows:

where $Psrc(y|x)$ is the probability of symbol label y obtained from the ASR model for the source domain when given input feature x. Here, “source domain” refers to the task during which the speech data used for model training was recorded, for example, dialog speech and presentation speech. Output probability $Psrc(y|x)$ from the ASR model can be expanded using Bayes’ rule as follows:

where $Psrc(x)$ is a term commonly calculated for all y and thus does not affect the relative relationship of the probability. Therefore, equation (3) can also be expressed as follows:

On the right side of equation (4), we can see that the ASR model includes acoustic information term $Psrc(x|y)$ and language information term $Psrc(y)$ and the latter of which represents the implicit language information referred to previously in this paper, that is, the statistics of language obtained from the speech data used for training the ASR model. This language information is considered to be approximately the same as the language information that would be possessed by an LM, $Qsrc(y)$⁠, trained in the same domain, that is, $Psrc(y)≈Qsrc(y)$⁠.

ASR models take advantage of language information to improve decoding when the domains of the test and training data are the same. But if these domains differ the ASR model’s language information can degrade its decoding performance. The DRA can be used to remove this language information by dividing $Psrc(y|x)$ by $Qsrc(y)$ and can replace LM information from the source domain with information from the target domain by multiplying the output probability of the target domain LM. This procedure is accomplished using subtraction and addition in the logarithmic domain as follows:

where $Qtgt(y)$ is the probability of the external LM for the target domain and where $λ$ and $μ$ are addition and subtraction weights, respectively. The DRA assumes that source domain acoustic information $Psrc(x|y)$ is roughly equal to target domain acoustic information $Ptgt(x|y)$⁠. Therefore, according to equation (5), the acoustic information is the source domain and the language information is the target domain which approximates the target domain of the ASR model. This means that by using equations (2)–(5), we can successfully replace only the ASR model’s language information, allowing the creation of an ASR model for any target domain simply by preparing text data for that domain. Furthermore, the DRA integrates the external LM and ASR model only at the time of inference, as in SF.

In this study, we assumed that only pure acoustic information remains in a trained E2E ASR model when its implicit language information has been removed and that it is possible to construct an ASR model for a target domain by combining the acoustic information from multiple source domains with language information from the target domain. By integrating the acoustic information extracted from multiple ASR models, it should be possible to enhance the robustness of the acoustic information, even when the ASR models are trained with only a small amount of data. We first extract the acoustic information from multiple E2E ASR models and integrate it at each decoding step, using the process shown in equation (6):

where N is the number of E2E ASR models used, $Psrc,i(y|x)$ is the i-th source domain ASR model and $Qsrc,i(y)$ is the i-th source domain LM. The acoustic information for each domain is extracted by subtracting the external LM’s language information for each domain from the respective E2E ASR models for those domains. The acoustic information is then integrated by summation of the extracted acoustic information. We then add integrated acoustic information $P˜src(x|y)$ and external LM information for the target domain $Qtgt(y)$⁠, as shown in equation (7) below, confirming in theory that a target domain ASR model can be created with acoustic information obtained from multiple E2E ASR models and language information from an external LM trained with a large amount of target domain text data:

Figure 2 provides an overview of our proposed method. We integrated the acoustic information from several E2E ASR models with target domain language information from an external LM during the beam search step. The novelty of our method is that the ASR output probabilities obtained after LM subtraction are treated as pure acoustic information. Based on this idea, we apply the integration of diverse acoustic information, as in (Kang and Kwon, 2016), to the E2E ASR framework.

We expect that our proposed method will be more acoustically robust due to the use of acoustic information from multiple, existing E2E ASR models, potentially improving ASR performance while reducing development costs. Because our method does not require any re-training, such as fine-tuning, of an existing ASR model, this type of ASR system can be easily developed without incurring the cost of data set construction (e.g. pairing speech data with its transcription). Note that the proposed method is not intended to be fully unsupervised target domain adaptation. Its main advantage is that it does not require retraining or parameter updates of the ASR models. In our experiments, the weights of the external LM, $(λ,μ)$⁠, are selected on a small target domain development set. Thus, a limited amount of target domain annotated data is used only for hyperparameter tuning, while no additional model training is required.

In Section 5.1, we describe the experimental conditions of our performance evaluation, including details of the data sets, ASR models and LMs used. The results of our experiments are provided in Section 5.2, where we first measures the perplexities of each LM to observe differences in the linguistic information they contain across domains. We subsequently compare the recognition accuracy of the proposed and conventional ASR methods.

We evaluated the performance of the proposed method experimentally using an ASR task and compared its recognition accuracy with results when using E2E ASR models with SF, DRA and without any external LMs. We also included ROVER in our experiment as a widely used system integration method that does not require additional ASR training. The speech recognition accuracy for each method was evaluated using character error rate (CER) for the Test-set. The weights of the LMs used with the SF, DRA and proposed methods were tuned using a grid search for Dev-set.

We used multiple data sets to train, validate and test the ASR models used in our experiments. These included the corpus of spontaneous Japanese (CSJ) (Maekawa, 2003), the Mainichi shimbun (MS) data set, the CommonVoice (CV) data set (Ardila et al., 2020), the Japanese newspaper article sentences (JNAS) data set (Itou et al., 1999) and the corpus of everyday Japanese conversation (CEJC) (Koiso et al., 2022). Four of five data sets (excluding the MS data set) are parallel data sets which include speech and transcription data. The details of each data set are as follows:

• The CSJ dataset comprises two domains; academic presentation speech (APS), which includes presentations on the subjects of engineering, humanities and sociology and simulated public speech (SPS), which consists of presentations without academic terminology.

• The CV dataset is a crowdsourced, multilingual corpus of read speech collected via the web. We used only the Japanese subset of the CV dataset in our experiments.

• The JNAS dataset was created by having speakers read excerpts from daily newspaper articles aloud. This dataset was used as the target domain in the experiments described in Section 5.2.2.

• The CEJC dataset contains natural daily conversations in various situations by speakers of a wide range of ages. Note that this dataset was only used as target domain data in the experiments described in Section 5.2.3. The training set was used only for LM training, while the development set was used as grid search data to set the LM weights. The test set was used as evaluation data.

MS has only text data and the details are as follows:

• The MS dataset is a text dataset consisting of 58 million characters from 11 years of Japanese newspaper articles (1991–2002) and was only used for training the external LMs. Because MS contains only text and JNAS is also composed of newspaper articles, the accuracy of ASR models was evaluated on the JNAS speech corpus. Table 1 shows the details of these datasets ^[1]. We expanded the training datasets by adding voice data converted to 0.9 and 1.1 times the speed of the original voice data, using speaking speed perturbation.

We prepared three versions of the E2E ASR models: E2E ASR models trained with the CSJ’s APS domain, the CSJ’s SPS domain and the CV domain. The vocabulary size of dictionary used for training the ASRs and LMs was set to 3,262, which is the total number of characters used in the CSJ corpus.

Following ESPnet’s CSJ recipe (Watanabe et al., 2018, 2021), the ASR models were trained under the following conditions. In this study, we used two types of E2E ASR models; a hybrid connectionist temporal classification (CTC)/attention architecture and joint CTC/attention Transformer provided by ESPnet.

The hybrid CTC/attention architecture is an RNN-based ASR model that combines the advantages of both CTC and attention during training and inference for E2E ASR. Figure 3(a) shows a model of the hybrid CTC/attention architecture. The acoustic features of the input speech for these ASR models were processed using mean and variance normalization (MVN) with an 80 Mel-scale filter bank. In our experiments, the bidirectional long short term memory (BLSTM)-based ASR model consisted of an encoder with Visual Geometry Group network (VGGnet) (Simonyan and Zisserman, 2015) and four BLSTM layers and an attention-based decoder with one long short term memory (LSTM) and a linear layer. During training, the model learns to maximize an objective function $L$⁠, which is a linear combination of $Latt$ and $Lctc$ in the form of a weight parameter, using the output of the CTC decoder in addition to the output of the attention decoder for the CTC loss, as an auxiliary. Thus:

where $α$ is a hyperparameter for weighting. During inference, the output of the CTC decoder is used as a supplement to the output of the attention decoder to output a symbolic label, $y^$⁠, which is formulated as shown in equation (9):

where $β$ is the weight for the CTC decoder. In this study, $α$ and $β$ were both set to 0.3. We trained the entire ASR model in the E2E fashion using the Ada-delta (Zeiler, 2012) optimizer with a batch size of 24.

We also used joint CTC/attention Transformer as a Transformer-based ASR model that combines CTC and attention as in the hybrid CTC/attention architecture. Figure 3(b) shows the model of the joint CTC/attention Transformer. In contrast, Transformer-based ASR model first performs SpecAugment (time contraction, time masking and frequency masking) (Park et al., 2019) and MVN when acoustic features are input. The model consisted of 18 Transformer encoders, 6 Transformer decoders and a CTC module-based decoder with the weight set to 0.3. The number of multi-head attention heads used by each encoder and decoder was set to 8. We then trained the entire model in the E2E fashion using Adam (Kingma and Ba, 2015).

We prepared five external LMs, which were trained with APS transcription data, SPS transcription data, CV transcription data, MS text data and CEJC transcription data, respectively. The LMs were trained following ESPnet’s recipe, in the same as Section 5.1.2. We used two types of character-based neural LMs; an RNN LM (Mikolov et al., 2010) and a Transformer-based LM (Vaswani et al., 2017). We used these LMs with their respective ASR models, that is, the RNN LM with the hybrid CTC/attention architecture model and the Transformer-based LM with the joint CTC/attention Transformer model. The RNN LM consists of an embedding layer, two LSTM layers and a linear layer. The batch size was 256 and stochastic gradient descent was used as the optimization function. The transformer LM has 16 Transformer blocks. The number of heads was set to 8, the dimensionality of the multi-head attention was set to 512 and Adam was used as the optimization function during training.

In addition to the model architectures described above, recent E2E ASR studies have also explored stronger methods using large language models (LLMs) (Hu et al., 2023; Hori et al., 2025). For example, previous work has applied LLMs to SF and showed that pretrained LMs can be used as external LMs (Hu et al., 2023). From this viewpoint, the DRA-based framework may also have the potential to be extended in a similar direction. In this paper, however, we focus on evaluating the effectiveness of multiple model integration itself and therefore used scratch-trained RNN LM and Transformer LM.

To compare the differences in the language information of each domain, we calculated the perplexities of the external RNN LMs for each domain, which are shown in Table 2. Because the MS data set has no Test-set, we evaluated the MSLM using the JNAS Test-set (see Table 1). Table 2 shows that the language information of each domain is different, in particular, the perplexity of the CV and CEJC LMs were significantly high for the target domain, JNAS. The MS and CEJC LMs were used as external LMs for adapting to the target domain.

In addition, Table 3 shows the perplexity for the Transformer LMs with each test-set. Basically, the relative relationship between the perplexities is similar to that of the RNN LMs, but the values decrease throughout the models, suggesting that the model retains more linguistic information.

Using the ASR models and LMs obtained after training in each domain, we compared the performance of the proposed method with the conventional integration methods when JNAS data was used as the target domain.

The results of our comparative experiment using the RNN-based ASR models and RNN LMs are shown in Table 4. The first block shows the results for the conventional LM integration methods when using a single ASR model. All of the models achieved the lowest CER when DRA was used, followed by SF and the without LM. The [APS+SPS] model, a relatively large model trained with both APS and SPS data sets, achieved much lower CER than the models using only one data set for training. This model represents an ideal setup, in which large when computational resources and training data are already available. Moreover, the [APS+SPS+CV] model, which adds CV training data to APS+SPS training data, achieved an even lower CER than the [APS+SPS]ASR model.

The second block shows the evaluation results for each ASR model integration method without any additional training. Because ROVER requires outputs from three or more models, its results are not included in the two-model integration experiment. These results show that the proposed integration method outperformed the single ASR model methods (w/o LM, SF and DRA). This improvement was confirmed to be statistically significant at the 5% level by matched pairs sentence-segment word error (MAPSSWE) test (Gillick and Cox, 1989) for the [APS], [SPS] and [CV] models. The CER when the [APS]+[SPS] were integrated using the proposed method was 16.6%, which was slightly higher than the CER when using the [APS+SPS]ASR model with the DRA (15.9%), however our proposed method achieved comparable performance (no statistically significant difference) without the need to train the ASR model with both APS and SPS speech data. The relative difference between the CER of the proposed integrated model and the CER of the large ASR model was only 4.2%, so we believe the proposed method is very effective given the fact that it can integrate multiple existing E2E ASR models without re-training. Furthermore, when comparing LM integration methods (SF and proposed) used to integrate two ASR model outputs, the proposed method achieved better accuracy. These results demonstrate the effectiveness of the proposed method in extracting and integrating acoustic information.

The last line in Table 4 shows the results for the integration of three ASR models, when [CV] was used as the third model. The proposed method demonstrated higher recognition accuracy than all of the models trained with a single data set when the SF or DRA were used and also significantly outperformed ROVER. This suggests that acquiring acoustic information for the CV domain, which has a speech style not found in the APS or SPS data sets, improved ASR robustness.

Table 5 shows the results obtained when integrating transformer-base LMs and ASR models. The results show higher overall ASR accuracy than when using the RNN-based ASR and RNN LM setup, as shown in Table 4. The results shown in Table 5 are also superior to those achieved when using the Transformer ASR and RNN LM setup used in our previous work (Hojo et al., 2023). Furthermore, by comparing the results in Table 4 and Table 5, we can see that the relative accuracy relationships among the methods follows a similar trend. In particular, the proposed method outperformed the conventional integration method ROVER while achieving comparable performance to the ideal, single ASR models ([APS+SPS] and [APS+SPS+CV]) without additional training. This demonstrates that the proposed method is also effective for Transformer-based models.

Table 6 compares the error tendencies between the ideal condition, [APS+SPS]DRA and our method. First, in [APS]+[SPS], the proposed method improved all error types compared with w/o LM. In particular, large improvements were observed in substitution and insertion errors, suggesting that model integration based on pure acoustic information contributes to more accurate recognition. Compared with [APS+SPS] DRA, the largest difference for the proposed method was found in substitution errors. This may be because a single model trained on multiple source domains can integrate domain knowledge more flexibly inside the model than integrating output probabilities at inference. On the other hand, the proposed method showed lower insertion errors than [APS+SPS]DRA. This suggests that removing linguistic information separately from each model may be effective in suppressing linguistically excessive outputs. The same tendency was also observed in the three domain experiment.

Table 7 shows the grid search results of the LM subtraction weights of the proposed method ([APS]+[SPS]) for the Transformer-based model presented in Table 5. As a result, the optimal weights for LM subtraction in [APS] and [SPS] were 0.5 and 0.3, respectively. Although not shown in Table 7, the optimal subtraction weight for [CV] in the three domain setting ([APS]+[SPS]+[CV]) was 0.3. These results show that the subtraction weights for the source domains tend to be low. The same tendency was also observed in the RNN-based experiments, where the subtraction weights for the source domains were in the range from 0.1 to 0.5. These results suggest that the proposed method may also be applied to a newly added domain with relatively easy tuning.

Experimental results when the target domain is the CEJC data set are shown in Table 8. We can see that the accuracy achieved by the proposed method when using a conversational target domain (CER = 35.4%), was higher than that of the RNN-based [APS+SPS] model trained with two data sets (CER = 36.2%), unlike when the JNAS data set was used as the target domain, a condition under which the accuracy of the [APS+SPS] model was higher than that of the proposed method. We assumed that when a single model is trained, the probability distribution converges to fit the source domains, but that when multiple ASR models are integrated, multiple distributions are combined to form an abstract representation and that this distribution covers the feature space more robustly than a single ASR model, improving recognition performance in domains that are different from the source domains. This suggests that the proposed method is effective even when the speech styles of the source and target domains differ.

On the other hand, increasing the number of models being integrated (the [APS]+[SPS]+[CV] model) did not results in higher accuracy than that of the single ASR model trained with data from three source domains (the[APS+SPS+CV] model). Because the CEJC contains spontaneous speech, while the CV consists of read speech, this mismatch in speech styles is considered to be a factor in the widening performance gap. However, the proposed method when integrating the output of three ASR models ([APS]+[SPS]+[CV]) outperformed the two-model integration approach ([APS]+[SPS]) and both of these models significantly outperformed ROVER.

In this study we have proposed a method of extracting pure acoustic information from multiple existing E2E ASR models using subtraction through the DRA and then integrating the extracted acoustic information with an external LM for the target domain to create an ASR model for the target domain, without the need for additional model training. The proposed method achieved better performance than most of the conventional methods using a single E2E ASR model. By integrating the outputs of two or three different, existing ASR models using our proposed method, we have successfully created an acoustically robust ASR model, using an external LM trained only with target domain text data. The integration of multiple models using the proposed method also shows that it is possible to mitigate mismatches between the source domain data used to train the ASR models and the target domain. Furthermore, the proposed method is more effective than ROVER, a conventional method for integrating multiple ASR systems.

In future work, we will verify the generality of the proposed method by increasing the number of ASR models being integrated and by conducting experiments using corpora of other languages.