An investigation of the robustness of flow- and diffusion-based speech generation models on noisy transcriptions

Text-to-speech (TTS) is a technology that generates speech corresponding to given textual input. Leveraging the powerful learning capabilities of deep neural networks (DNNs), current TTS methods can produce high-quality speech with near-human naturalness (Kim et al., 2021; Li et al., 2019; Ren et al., 2021; Ren et al., 2019; Shen et al., 2018). To expand training corpora, modern TTS systems often rely on transcriptions generated by automatic speech recognition (ASR) (Ao et al., 2022; Chen et al., 2025; Chen et al., 2023; Neekhara et al., 2024). Indeed, for high-quality speech, current ASR models can produce transcripts that match the original audio with high fidelity. However, this ideal scenario often does not hold when processing large-scale, diverse data sets, where ASR errors are unavoidable due to varying acoustic environments. While manual data selection can mitigate this issue, it is not a scalable solution, and some low-quality transcriptions will invariably remain. This raises a crucial question of model-centric robustness: how do advanced TTS architectures inherently perform when trained on imperfect data? Therefore, investigating model behavior on noisy transcriptions remains a vital and necessary research area.

Although many models have made breakthroughs in training on noisy transcription, prior research lacked systematic investigation into how different TTS architectures inherently behave under such conditions. Fong et al. (2019) investigated AR-based TTS behavior on noisy transcription, revealing attention mechanism responses, but their scope was limited to autoregressive (AR) models. Some studies (Ma et al., 2022; Yang et al., 2020; Yasuda and Toda, 2023) mentioned the robustness potential of non-AR and diffusion-based frameworks on mildly noisy transcription, but did not provide systematic behavioral analysis across varying noise conditions. To address this gap, this paper investigates the robustness and behavior of diffusion- and flow-based TTS models when trained on noisy transcriptions.

To investigate the robustness of different TTS architectures, we trained flow-based (VITS) and diffusion-based (GradTTS, DVT) models on simulated noisy transcriptions with varying noise levels, using Tacotron 2 as the baseline. We evaluated both intelligibility (via ASR-based word error rate) and naturalness (via five-point MOS ratings), comparing noisy models against clean models trained on clean transcriptions.

To understand the underlying mechanisms, we conducted diffusion process analysis and sensitivity testing. These experiments revealed that noisy models exhibit differential activation of text representations based on input characteristics, correlating with transcription quality patterns from training. To mitigate the performance degradation associated with this phenomenon, we investigate a text prompt strategy that provides contextual guidance to bias the model toward high-quality representations, demonstrating its effectiveness as a lightweight mitigation approach without requiring fine-tuning.

A shorter conference version of this study was published in Feng et al. (2024), which primarily highlighted the exceptional robustness of diffusion-based models under noisy transcription conditions. This journal version makes the following additional contributions:

• We conduct comprehensive sensitivity tests to systematically explore model robustness to text style perturbations, revealing a latent domain separation phenomenon where text representations are spontaneously organized according to transcription quality.

• We perform detailed analysis of learned text representations through classification experiments, quantitatively demonstrating the separability of clean versus noisy text feature domains and its correlation with synthesis performance degradation.

• We investigate the VITS architecture specifically, providing explanations for its lack of robustness compared to diffusion-based models through duration predictor analysis.

• We propose a text prompt strategy to mitigate performance degradation caused by latent domain separation, demonstrating its effectiveness in activating high-quality text representations without requiring model fine-tuning.

The remainder of this paper is organized as follows: Section 2 reviews flow- and diffusion-based TTS models and other factors relevant to training with noisy transcriptions. Section 3 describes the methods for simulating noisy transcription data and the associated data conditions. Section 4 outlines the models and training configurations. Section 5 evaluates noisy model performance through subjective and objective measures. Section 6 analyzes the training and inference behavior of noisy models and Section 7 concludes this paper.

When training TTS models using noisy transcriptions, two primary research directions exist: data-driven cleaning techniques and model-driven paradigm studies. Among these, data cleaning and selection techniques are the most intuitive and effective approach to handle noisy transcriptions (Braunschweiler et al., 2010; Seki et al., 2023; Stan et al., 2013). Their core goal is to cull or down-weight noisy transcriptions, thereby increasing the effective information density of the training data. With the increasing demand for end-to-end neural TTS on large-scale, multi-speaker corpora, researchers have developed automated paradigms and data pipelines that rely on ASR confidence, language model perplexity and text rule filtering (Ardila et al., 2020; Chen et al., 2021; He et al., 2025). In contrast to data-level processing, the other direction focuses on the modeling paradigm itself. Such methods aim to enable the model to learn directly from noisy data by designing innovative model architectures or training strategies that can automatically ignore or correct mismatches between the transcription and the audio. Although most current research on noisy transcription data has concentrated on data selection and cleaning, exploring new model paradigms and module behaviors that can directly handle noise remains critically important for the effective utilization of large-scale, real-world data sets.

Early research into model paradigms for handling noisy transcriptions began with statistical methods. For instance, unit-selection-based systems relied on forced alignment to flag inconsistencies, subsequently blocking mismatched units during retrieval to reduce phoneme-level noise from annotation errors (Clark et al., 2007; Matousek et al., 2012). In the subsequent HMM-SPSS (Hidden Markov Model and Statistical Parametric Speech Synthesis) era, techniques like beam pruning within Baum-Welch training suppressed bias from erroneous transcriptions by filtering low-confidence state paths during statistical inference, making HMM systems inherently more robust than their unit-selection counterparts (Yamagishi et al., 2008). The advent of attention-based AR-based models introduced new dynamics. An investigation by Fong et al. quantified the impact of corrupted transcriptions, demonstrating that while the attention mechanism can implicitly skip extra words, it remains vulnerable to substitution and deletion errors, thus motivating explicit data filtering (Fong et al., 2019). In addition, some works showed that the network structure of text encoder affects the performance on imperfect transcriptions on AR-based TTS models. A study by Yasuda et al. revealed that a complex CBHG encoder (Kim et al., 2016) outperformed simpler CNN-based encoders, particularly with ambiguous orthography or smaller model sizes (Shen et al., 2018; Wang et al., 2017; Yasuda et al., 2021). This underscores that text representations are a crucial factor when handling noisy transcriptions (Li et al., 2020; Zhu et al., 2023). Beyond refining core components, other approaches introduce specialized modules. For example, Yang et al. proposed an unsupervised VQVAE-based module within a AR-based framework that uses acoustic features from preceding frames to generate discrete, phoneme-like representations, effectively mitigating the impact of erroneous linguistic features (Yang et al., 2020).

While the aforementioned studies on AR-based models underscore the critical role of components like the text encoder in handling noisy data, the emergence of newer generative architectures –namely, diffusion and flow-based models – presents a new research frontier that may exhibit different robustness characteristics due to their fundamentally different generative mechanisms. These models operate on different principles, and their inherent robustness to imperfect transcriptions requires a dedicated investigation. Recent work has provided promising, albeit preliminary, evidence for the robustness of diffusion-based TTS. They have demonstrated strong performance in handling ambiguous orthography, attributed to the diffusion process’s resilience to slight perturbations without needing specialized modules (Yasuda and Toda, 2023). Furthermore, their capacity for unconditional generation and speech infilling highlights a potential tolerance for incomplete or incorrect textual information (Kim et al., 2022; Tae et al., 2022). However, despite this potential, a systematic investigation into their specific model behaviors, stability and performance limitations under controlled noisy conditions is notably absent. Similarly, flow-based TTS models, renowned for their high-quality generation, represent another important architecture whose behavior under noisy transcriptions remains largely unexplored. While some properties suggest a potential sensitivity to input perturbations, empirical evidence is scarce.

Therefore, this research aims to address these gaps. We conduct an investigation into the performance and behavior of flow- and diffusion-based TTS models when trained on noisy transcriptions. To provide a comparative context, we compare their performance and behavior against established AR-based architectures and analyze the text representation to better understand their roles when handling imperfect data.

AR-based models established the dominant encoder-decoder framework for modern neural TTS synthesis (Li et al., 2019; Ren et al., 2021; Ren et al., 2019; Shen et al., 2018). In this paradigm, a text encoder first transforms the input character or phoneme sequence into a high-level latent representation. An AR decoder then generates the acoustic features (e.g. mel-spectrograms) one frame at a time, guided by an attention mechanism that dynamically weighs the encoded text representations at each step. Despite their success, the performance of AR-based models is critically dependent on the stability of their core components, especially when dealing with imperfect data. The attention mechanism, while powerful, can be a point of failure; it has been shown to be vulnerable to substitution and deletion errors in the source text, even while being able to skip extraneous words (Fong et al., 2019). This sensitivity places a significant burden on the text encoder to produce robust representations. Research has confirmed that the encoder’s network structure is a crucial factor in the model’s resilience to noisy inputs (Li et al., 2020; Yasuda et al., 2021; Zhu et al., 2023). The challenges associated with these core components have also spurred the development of auxiliary modules aimed at mitigating the impact of erroneous linguistic features within the AR framework (Yang et al., 2020).

Given its foundational status and its well-documented, complex behaviors when handling imperfect data, the AR-based model serves as the essential baseline in our study, providing a crucial point of comparison for evaluating the robustness of newer generative architectures.

Flow-based generative models have been successfully adopted for TTS synthesis (Kim et al., 2020; Kim et al., 2021; Miao et al., 2020; Valle et al., 2021). These models learn an invertible mapping, defined by a conditional normalizing flow $fθ(·∣y)$⁠, which transforms an acoustic frame sequence $x$ into a standard-normal latent variable $z=fθ(x∣y)$⁠. The exact conditional log-likelihood can then be computed efficiently from the base Gaussian distribution and the Jacobian of the transform: $logpθ(x∣y)=logN(z;0,I)+log|detJθ(x,y)|$⁠. At inference time, a sample is drawn from the latent distribution, $z∼N(0,I)$⁠, and the inverse mapping $x=fθ−1(z∣y)$ is applied, allowing for high-quality mel-spectrogram generation in a single parallel pass.

A key characteristic of flow-based generators is their fully reversible and exact nature. However, this property may also introduce vulnerabilities. As noted by Pope et al. (2020), certain components like non-regularized coupling layers can potentially amplify small errors in the input. This suggests that while the architecture is stable for clean data, it may be sensitive to adversarial perturbations or other forms of input noise. Despite this theoretical sensitivity, the behavior of flow-based TTS models when trained on or conditioned with noisy transcriptions has not been systematically evaluated.

Given their high performance alongside this potential sensitivity, flow-based models serve as an informative point of contrast to diffusion-based architectures in our investigation of robustness to noisy transcriptions.

Diffusion-based architectures are a prominent class of generative models now widely used in TTS synthesis (Kim et al., 2022; Popov et al., 2021; Yang et al., 2023; Yasuda and Toda, 2023). The core principle involves a forward process that incrementally adds noise to clean speech data and a reverse process that learns to iteratively refine a latent representation, starting from pure noise $xT$ until it reaches the clean speech representation $x0$⁠. At each step t of this reverse process, the model consumes the current noisy latent $xt$ along with linguistic conditioning y to predict a less-noisy latent $xt−1$⁠. Depending on the training objective, these models can be categorized as score-based diffusion, which estimates the data distribution’s score (Song and Ermon, 2019), or as denoising diffusion probabilistic models (DDPMs), which directly predict the noise introduced during the forward process (Ho et al., 2020). During inference, this stepwise mechanism allows the system to leverage information from both the partially denoised acoustic representation and the text transcript at every timestep, yielding stable, high-quality speech through iterative denoising.

A key characteristic of diffusion models is their robustness. This iterative denoising process regularizes the data distribution, suppresses mode collapse and has been shown to be resilient to noise and adversarial perturbations (Chen et al., 2024; Zhang et al., 2023b). In the context of TTS, Yasuda and Toda (2023) demonstrated that diffusion-based systems exhibit notable tolerance to ambiguous orthography. A plausible explanation lies in the “gradual correction” mechanism inherent in the reverse diffusion chain. At each denoising step, any misguidance from an ambiguous or incorrect character perturbs the generation trajectory only slightly. Subsequent steps then gradually pull the sample back toward the high-density speech manifold, distributing the impact of conditioning noise over many iterations and preventing catastrophic drift. While the uncertainty from ambiguous orthography is milder than mismatches from noisy transcriptions, this empirically observed robustness suggests these models have untapped potential for coping with more severe errors.

Consequently, we posit that the inherent error-correction mechanism of diffusion models makes them a primary candidate for mitigating the severe degradations induced by noisy transcripts, warranting a systematic investigation into their behavior under such challenging conditions.

To investigate the impact of noisy transcriptions on TTS performance, we use a framework consisting of three stages: (a) noisy transcription simulation, (b) model training and (c) evaluation and analysis, as illustrated in Figure 1. In Stage (a), we simulate noisy transcriptions by adding environmental noise to clean speech and then transcribing with an ASR system. In Stage (b), various TTS models are trained on clean speech paired with different transcription conditions. In Stage (c), trained models are evaluated through perceptual listening tests and analytical analysis using input text from both clean and noisy test sets.

To ensure controlled experimental conditions in Stage (a), we simulate noisy transcriptions by first generating noisy speech and then applying ASR to obtain transcriptions with varying degrees of impairment. For noisy speech generation, we use environmental noise as background noise for speech mixing to simulate realistic transcription scenarios found in real-world data. Clean speech samples are mixed with environmental noise at different signal-to-noise ratio (SNR) values, defined as:

where $Pspeech$ and $Pnoise$ represent the power of the speech signal and environmental noise, respectively. By varying the SNR during mixing, we create speech samples with different noise levels, thereby simulating the uneven speech quality commonly encountered in real-world data.

We then apply ASR to obtain noisy transcriptions from the simulated noisy speech. We denote the obtained transcriptions as WER-X, where X indicates the corresponding word error rate (WER) value. The transcriptions from running ASR on the original clean data are denoted as WER-070, and the manually processed original text of LJSpeech is denoted as “Clean.” To ensure input consistency across all experiments, phonemes are used as input symbols, with text sequences converted to phoneme sequences using a phonemizer, a tool that converts text to phoneme representations. Specific details about the environmental noise database, SNR intervals, and the resulting transcription conditions are provided in Section 4.2.

In Stage (b), we train multiple TTS models on clean speech paired with the transcription conditions generated in Stage (a). The evaluated architectures include diffusion-based models (GradTTS, DVT, DVT2), a flow-based model (VITS) and an AR baseline (Tacotron 2). To independently investigate the impact of noisy transcriptions on TTS training, all training processes use high-quality speech samples from the original LJSpeech data set, ensuring that variations in synthesis quality are attributable solely to transcription quality rather than acoustic degradation. Detailed descriptions of the TTS systems and their training configurations are provided in Section 4.

In Stage (c), we thoroughly assess and investigate the performance of the trained models through two groups of experiments: (i) experimental evaluation and (ii) analytical evaluation. The experimental evaluation includes both naturalness and intelligibility tests, which were also reported in our conference version (Feng et al., 2024) to ensure consistency in the core assessment of perceptual quality.

Beyond perceptual evaluation, we further conduct a set of analytical studies to understand model behaviors under noisy conditions. While the diffusion process analysis was already presented in the conference version, this work introduces additional analyses to deepen our understanding:

• Sensitivity test, examining robustness under specified input perturbations.

• Learned text representations, revealing how the linguistic information is encoded in behavior of noisy models.

• Text prompt analysis, exploring the controllability of robustness through textual guidance.

These new methods are designed to understand the mechanisms underlying TTS robustness on noisy transcriptions. Detailed descriptions of all evaluation and analysis methods are provided in Sections 5 and 6.

For our experiments, we selected four representative TTS models: GradTTS (Popov et al., 2021), DVT (Yasuda and Toda, 2023), VITS (Kim et al., 2021) and Tacotron 2 (Shen et al., 2018).

We evaluate diffusion-based TTS models [Figure 2(a)]: GradTTS, which applies diffusion directly to acoustic features, and DVT, which incorporates diffusion within a VAE framework, along with DVT2, a DVT variant designed to investigate encoder structure impact. Grad-TTS generates acoustic features by learning a reverse diffusion process conditioned on linguistic inputs. It is trained to minimize a denoising score-matching loss (Song and Ermon, 2019), which guides the model to estimate the gradient of the data distribution and progressively reconstruct clean mel-spectrograms from noisy latent variables. Grad-TTS is selected as the diffusion-based representative in our experiments due to its state-of-the-art synthesis performance among diffusion-based TTS models (Zhang et al., 2023a). DVT consists of a VAE-based waveform model and a diffusion-based acoustic model. While similar to VITS in its overall framework, DVT incorporates acoustic model of diffusion probabilistic models (DPM) (Ho et al., 2020). In DVT, the posterior encoder encodes speech into a Gaussian latent acoustic space $zX$⁠, while a diffusion model learns the reverse process $pθ(zX∣zY)$ conditioned on latent linguistic features $zY$⁠. Alignments are obtained with the monotonic path with the least total distance within trellis defined by the square distance between $zY$ and $gϕ(zX)$⁠, $gϕ$ is a function to convert $zX$ into $zY$ to align them. At synthesis time, text is encoded into $zY$⁠, up-sampled by predicted durations, denoised into $zX$⁠, and decoded to waveform. Based on its robust performance on ambiguous orthography, it was chosen as another diffusion-based representative. Based on DVT, we developed a variant model, DVT2, to investigate the impact of text encoder complexity on model robustness, motivated by prior findings on the importance of encoder architecture in handling noisy data (Section 2). The DVT2 model adds a three-layer PreNet module to the original DVT Transformer-based text encoder structure to match GradTTS’s encoder architecture, enabling controlled comparison of encoder effects while maintaining DVT’s framework. Each layer of the PreNet module consists of a one-dimensional convolutional layer, a normalization layer, a ReLU activation and a dropout layer.

VITS [Figure 2(b)] adopts a flow-based variational framework in which both the text encoder output and the acoustic decoder output are mapped to independent Gaussian latent variables. An invertible affine coupling flow provides a deterministic, reversible transformation between waveform features and the latent variables, allowing exact log-likelihood evaluation as part of the variational lower bound optimization; its parameters are implicitly updated through the reconstruction and KL divergence terms. VITS uses adversarial training to ensure perceptual quality, and uses monotonic alignment to stabilize training. Based on its excellent performance, VITS was chosen as a representative of flow-based models in this experiment.

Tacotron 2 [Figure 2(c)] uses an RNN-based text encoder and a soft-attention-based, AR decoder. Tacotron 2 is trained to minimize the L1 loss between the predicted and ground-truth mel-spectrograms. It served as a baseline to represent attention-based AR-based models in this study.

As described in Section 3, we simulated noisy transcriptions by mixing clean speech with environmental noise and then applying ASR. This subsection provides the specific experimental parameters for the data conditions.

Single-channel audio from the DEMAND database (Thiemann et al., 2013) was used to simulate environmental noise. The audio from DEMAND was randomly paired with speech data from LJSpeech (Ito and Johnson, 2017) and mixed using different SNR values to obtain artificially mixed noisy speech. To simulate a variety of noisy transcriptions, we set five SNR intervals for the noisy speech groups: $[−10,0)$⁠, $[−15,0)$⁠, $[−20,0)$⁠, $[−20,−5)$ and $[−20,−10)$ dB. When mixing the environmental noise and speech sounds, a random SNR value was selected from the corresponding SNR intervals. This process allows each speech sample to be mixed with noisy audio at different SNRs, resulting in speech with varying noise levels in each speech group, thereby simulating possible uneven speech quality in real-world data.

A pre-trained model of Hybrid CTC/attention-based ASR^[1] from ESPnet ^[2] was used for transcribing. This model achieved high recognition performance on the original LJSpeech speech training set with character error rate (CER) and WER of 3.3% and 7.0%, respectively. We obtained five sets of noisy transcriptions with WER values of 16.2%, 24.2%, 35.8%, 43.7% and 54.8%, respectively. Finally, we incorporated the manually processed original text of LJSpeech (named as “Clean”) as an experimental condition. To ensure input consistency across all experiments, phonemes were used as input symbols, with text sequences converted to phoneme sequences using a phonemizer ^[3]. This setup results in seven transcription conditions, as summarized in Table 1.

For each transcription condition, we split the speech and transcription pairs into training, validation, and test sets comprising 12,500, 100 and 250 samples, respectively. To independently investigate the impact of noisy transcriptions on TTS training, all training processes in this study used high-quality speech samples from the original LJSpeech data set, ensuring that variations in synthesis quality are attributable solely to transcription quality rather than acoustic degradation.

Regarding acoustic feature representation and waveform synthesis, we used model-specific configurations as follows. In Tacotron 2 and GradTTS, we used mel-spectrograms as the acoustic features and synthesized the waveforms based on the mel-spectrograms using a HiFiGAN vocoder (Kong et al., 2020). For VITS, we used linear spectrograms as acoustic features to train the acoustic encoder and the HiFiGAN-based decoder within the VAE framework. Waveforms were synthesized end-to-end by the jointly trained HiFiGAN-based decoder conditioned on the latent representations. For DVT and DVT2, we used the mel-spectrogram to train the HiFiGAN-based waveform decoder separately. After training the waveform decoder, we trained the remaining parts of the model with the waveform decoder parameters frozen.

Training was conducted on an NVIDIA Tesla V100 GPU. For Tacotron 2 (implemented with ESPnet2), we used dynamic batching with batch bins of 5,120,000. For GradTTS, we used a batch size of 16 and a learning rate of $1×10−4$⁠. For VITS, DVT and DVT2, we used a batch size of 64 with learning rate decay. All models were trained until convergence, with other hyperparameters following the original implementations (Kim et al., 2021; Popov et al., 2021; Shen et al., 2018; Yasuda and Toda, 2023).

A total of 35 models were evaluated, covering seven transcription conditions and five model architectures: five clean models were trained on the clean transcription condition, and 30 noisy models were trained on different noisy transcription conditions.

We evaluated the naturalness and intelligibility of synthesized speech using a subjective assessment method via MOS testing and an objective assessment method based on ASR results, respectively.

We conducted a listening test to evaluate the subjective naturalness of the synthesized speech. The test included synthesized samples from 15 models, i.e. five TTS methods trained on three transcription conditions: clean, WER-070 and WER-358. Natural speech samples (Ground Truth, GT) were also included for comparison. Each system contained 20 samples. Listeners were asked to rate the naturalness of each audio sample using a five-point mean opinion score (MOS) scale. The correct transcript was provided to the listeners, and they were instructed to consider the impact of mispronunciations when assigning scores. English-speaking listeners were recruited via Amazon Mechanical Turk ^[4]. Each sample received ratings from 20 listeners, resulting in a total of 6,400 collected scores. We tested the statistical significance using the t-test.

Figure 3 shows the results of the MOS test. All clean models demonstrated high naturalness, with MOS scores ranging from 3.8 to 4.1. Tacotron 2 and VITS exhibited significant degradation in naturalness as the transcription was corrupted: VITS showed a degradation in MOS from $4.15±0.07$ (clean model) to $3.37±0.16$ (WER-358 noisy model), while Tacotron 2’s MOS degraded from $3.81±0.06$ to $3.38±0.17$⁠. In contrast, diffusion-based TTS models, including GradTTS, DVT and DVT2, showed only slight degradation in naturalness with increasing transcript corruption: GradTTS’s MOS decreased from $4.11±0.10$ (clean model) to $3.82±0.12$ (WER-358 noisy model), and DVT’s MOS decreased from $3.87±0.10$ to $3.56±0.17$⁠. Compared to DVT, DVT2 could mitigate the degradation more from $3.99±0.09$ to $3.77±0.15$⁠. These results confirm that from a human perception standpoint, diffusion-based models trained on noisy data achieve better naturalness.

We conducted an intelligibility evaluation to objectively assess the linguistic content accuracy of synthesized speech. This evaluation quantifies the model’s ability to learn from corrupted transcriptions and correctly synthesize, making it an important metric for evaluating noisy models. An ASR system was used to evaluate the WER and character error rate (CER) on the synthesized speech as objective intelligibility metrics (Taylor and Richmond, 2021). The same ASR model used for simulating the noisy transcriptions in Section 3 was used for this evaluation. Text samples from a clean transcription testing set were used as input during inference. 250 synthesized samples were evaluated from each model.

Table 2 shows the result of intelligibility. All clean models showed similar performance, which was close to the recognition performance on ground truth speech, indicating that all methods could render highly intelligible speech given clean transcriptions. As the training transcriptions degraded, the noisy models exhibited different performance. Tacotron 2 degraded its intelligibility greatly up to 45.3% WER as training transcriptions corrupted. VITS also showed intensive degradation with WER increasing up to 46.7%. On the other hand, GradTTS showed only a small degradation in intelligibility as training transcriptions were corrupted, with its WER increasing to just 16.5%. DVT showed similar trends to GradTTS, exhibiting a smaller degree of degradation, with WER increasing up to 13.9%. These results support that diffusion-based GradTTS and DVT noisy models have good intelligibility performance, even on severely corrupted transcriptional training conditions. Furthermore, DVT2 mitigated the intelligibility degradation slightly better compared to DVT. This suggests that the complex network structure of the text encoder may contribute to this intelligibility.

The original VITS model exhibited poor intelligibility results under noisy transcription training conditions. Some studies have shown that the stochastic duration predictor (SDP) in VITS affects the model’s stability, particularly when modeling complex conditions (Casanova et al., 2022; Cho et al., 2022). To address this potential unstable factor, we replaced the original flow-based SDP in VITS with a duration predictor composed of two convolutional layers with ReLU activation, layer normalization and dropout, followed by a projection layer. This duration predictor structure was originally proposed in Glow-TTS (Kim et al., 2020) and is the same as that used in GradTTS and DVT. The resulting variant is denoted VITS-DP, and a similar modification has been shown to improve model stability (Cho et al., 2022). We evaluated the intelligibility performance of samples synthesized by VITS-DP trained on seven transcription conditions. Inference and evaluation methods were the same as described in Section 5.2.

The last line in Table 2 presents the results of VITS-DP. For the clean model, VITS-DP and VITS show similar performance. However, under noisy transcription training conditions, the performance of VITS-DP is significantly better than VITS. In the early stages of transcription corruption (WER-070, WER-162 noisy models), VITS-DP and DVT models exhibited similar performance. Under more severe transcription corruption conditions (WER-242, WER-358, WER-437, WER-548 noisy models), VITS-DP remained superior to Tacotron 2, but performed noticeably worse than DVT. This suggests that VITS-DP offers some robustness against transcriptional noise, but the ability is attenuated compared to DVT as corruption increases.

The improved robustness of VITS-DP can be attributed to fundamental architectural differences in how duration is modeled. SDP in VITS models duration prediction probabilistically via a normalizing flow (Kim et al., 2021), learning a distribution over possible durations conditioned on text. While this probabilistic formulation effectively captures prosodic variability in clean data, it becomes vulnerable when trained on noisy transcriptions. During training with imperfect text-speech correspondences, SDP attempts to fit a distribution that encompasses all observed duration patterns, including those influenced by transcription errors. This can lead to overly broad learned distributions. The flow-based architecture, which requires invertible transformations, may further amplify this issue when accommodating such varied patterns (Cho et al., 2022).

In contrast, the deterministic duration predictor learns to predict expected durations through regression (Kim et al., 2020; Popov et al., 2021), inherently averaging over variations in the training data. This averaging property makes it more stable when some training samples contain errors, as the prediction naturally gravitates toward the predominant correct patterns rather than attempting to model the full range of observations.

Nevertheless, the persistent gap between VITS-DP and DVT under severe noise conditions (e.g., 33.6% vs 13.9% WER at WER-548) indicates that duration prediction alone does not fully determine robustness. Because both models use identical deterministic duration predictors yet differ fundamentally in their acoustic generation mechanisms – flow-based for VITS versus diffusion-based for DVT – the results suggest that the acoustic model’s generative process plays a more critical role, as analyzed in Section 6.

Drawing upon the evaluation results, this section delves into the reasons for the varying behavior and performance of the models when processing noisy transcriptions. We designed a series of analytical experiments: the “Diffusion Process Analysis” and “Sensitivity Test” explore the diffusion process behavior and mechanisms behind performance differences. Extending this analysis, we examine the “Learned Text Representations” to explain the phenomena observed in the “Sensitivity Test.” Furthermore, we propose a “Text Prompt” method to mitigate the impact of contaminated features on synthesis quality.

Diffusion-based models have performed consistently well in evaluation even on the heavily noisy transcription conditions. In this analysis, we took the DVT model as an example to observe and analyze the behavioral differences of diffusion-based models during the diffusion process under different noisy conditions. Time steps in the diffusion process are important factors for synthesis quality (Rombach et al., 2022; Xie et al., 2024). The differences in the diffusion process were revealed by calculating the likelihood ratio between the conditional and unconditional generation at each time step.

Figure 4 shows the history of the likelihood ratio between the conditional and unconditional generation for DVT during inference, illustrating the model’s dependence on transcriptions under different noise conditions and how this dependence changes over time steps. The lower the ratio, the lower the likelihood of conditional generation is, indicating that the inferred latent acoustic representation is not close to ground truth. Note that zero diffusion time corresponds to the final time step of inference. In the clean model, likelihood was low initially, but it immediately improved after several steps and gradually approached 1.0 afterward. It indicated that the clean model generated the latent acoustic representation close to ground truth within early time steps and consumed the remaining diffusion time for only minor improvements. In contrast, the noisy models maintained a low likelihood for up to half of the diffusion time steps and then improved abruptly to approach 1.0. This indicates that the noisy models take a long time to generate latent acoustic representation that is close to ground truth. This suggests that extending the diffusion time allows noisy models to approach the dependency trend of clean model, meaning that performance could benefit from a longer diffusion time.

During the evaluations, we assessed the naturalness and intelligibility of synthesized speech from models trained on noisy transcriptions. However, the mechanisms that led to the differences in performance remain unclear. To investigate this, we conducted a sensitivity test to examine the behavior of noisy models.

Because we did not use additional constraints to suppress the learning of noisy features, we have reason to believe that the model learned both correct and contaminated features from the noisy training data. The model’s superior performance during evaluation may be attributable to two factors, individually or in combination: (1) the model learned a sufficient number of correct features from the noisy data, and (2) the model preferentially activates correct features during synthesis. Given that clean transcriptions were used as input during evaluation, if noisy models possess an activation bias, it may be related to the style of the input text. Text style here refers to stylistic differences between transcriptions of noisy recordings and those of clean recordings. This is because noisy recordings can lead to ASR hallucinations (Frieske and Shi, 2024; Goldwater et al., 2010), such as recognizing results that are more inclined to output high-frequency words. The model may have learned this transcription style difference during training, while the transcription style associated with clean recordings corresponds to features with higher correctness. This could explain why some noisy models (e.g. GradTTS and DVT’s WER-548 noisy model) show remarkable intelligibility performance when presented with “Clean-style Text” from the clean transcription test set (see Section 5.2 for results).

To explore this possibility, we designed a sensitivity test using ASR transcriptions from noisy transcription condition (referred to as “Noisy-style Text”) as inference inputs. Some potential text style features of the “Noisy-style Text” are used as specific perturbations to observe changes in the accuracy of the model’s synthesis results. We randomly selected 250 ASR transcribed text that ASR on mixing speech at SNR = [−20,0) as input text for inference, none of which were used for training. The transcriptions used for testing had a WER = 37.2% as the “Noisy-style Text” set. “Clean-style Text” set is consistent with the input conditions in Section 5.2. This sensitivity test was conducted on six noisy models from each of the five TTS systems: Tacotron 2, VITS-DP, GradTTS, DVT and DVT2. We perform intelligibility evaluation on the synthesized results of models under the “Clean-style Text” and “Noisy-style Text” conditions. ASR model condition was the same as Section 5.2.

Figure 5 shows the WER results of intelligibility evaluation of samples synthesized on “Noisy-style Text” and “Clean-style Text” by the noisy models. We also included the results of the clean models for comparison. Analysis of baseline model behavior reveals that clean models exhibited comparable synthesis performance on both “Clean-style Text” and “Noisy-style Text,” with only a marginal advantage on “Noisy-style Text” attributed to its lower textual complexity. Similarly, WER-070 models demonstrated stable performance across both text styles, consistent with clean model behavior.

However, as training transcription quality deteriorated (WER-162, WER-242, WER-358, WER-437, WER-548), a clear performance degradation pattern emerged. The WER of synthesized speech on “Noisy-style Text” progressively increased beyond that of “Clean-style Text,” indicating that noisy models became increasingly sensitive to text style perturbations. This finding demonstrates that using ASR transcriptions with specific stylistic characteristics as input perturbations can effectively reveal inference instabilities in noisy models. The differential performance suggests that these models learned features of varying quality, with feature activation contingent upon the correlation between input text style and training data characteristics. This observation points to a latent domain separation phenomenon: noisy models trained without explicit quality labels spontaneously organize text representations into distinct subspaces corresponding to “Clean-style” and “Noisy-style” text. Consequently, when presented with “Noisy-style Text” during inference, the model preferentially activates lower-quality representation domains, leading to degraded synthesis performance.

The formation of these quality-dependent domains can be understood through the heterogeneity of the training data. In our experimental setup (Section 4.2), noisy transcriptions were generated by applying ASR to speech mixed with environmental noise at SNRs randomly sampled from specified intervals (e.g. [−20, 0) dB for WER-358). This creates substantial sentence-level quality variation: some utterances retain accurate transcriptions while others suffer severe degradation, resulting in a training corpus where high-quality and low-quality text-speech pairs coexist. During training, the model learns from this heterogeneous data, and text representations spontaneously organize according to quality patterns. This organization may arise naturally from the optimization process: we speculate that text-speech pairs with accurate transcriptions, where text and acoustic content are well-matched, likely to achieve lower training loss, leading the model to develop consistent representations for such inputs. Conversely, pairs with erroneous transcriptions may yield higher loss due to text-speech mismatch, resulting in the model learning distinct representations for inputs exhibiting error patterns. Over training iterations, this differential optimization pressure may cause representations to partition into quality-dependent subspaces.

Two patterns emerged across architectures. First, non-AR models exhibited larger performance gaps between “Clean-style” and “Noisy-style” text compared to Tacotron 2, suggesting more pronounced latent separation. Second, despite this gap, they significantly outperformed Tacotron 2 on “Clean-style” text, suggesting that stronger domain separation may be beneficial: while it leads to greater sensitivity to text style, it allows the model to preserve a cleaner, more robust high-quality representation domain by effectively isolating contaminated features learned from noisy data.

In a preceding sensitivity test, we observed degraded performance on “Noisy-style Text” in some noisy models and proposed that latent domain separation occurs spontaneously during model training based on the quality of transcription. The present analysis aims to further investigate how latent domain separation within noisy models is reflected in text representation, as well as the correlation between these representations and synthesized accuracy.

To observe latent text feature domain separation, we conducted classification experiments of text representations generated by the noisy model. We used the text set, the “Clean-style Text” set or the “Noisy-style Text” set, defined in Section 6.2. A higher classification accuracy was hypothesized to imply a more pronounced difference in the representations derived from these distinct text sets, indicative of more significant latent text feature domain separation by the models. We refer to the classification accuracy as the separation degree of latent domain.

We designed two classification experiments to explore the separation of text representations inferred by the noisy model on cleanly transcribed versus severely noisy ASR transcripts, and its separation of text representations inferred on ASR transcripts, respectively:

1. Classification 1: We randomly selected 4,000 phoneme representations from diverse sentences. These sentences originated from either “Clean-style text” set or “Noisy-style text” set, with each category contributing 2,000 representations. For observing clear separation, as the “Noisy-style text” category, we exclusively selected transcriptions corresponding to ASR outputs with SNRs in the range of [−20, −15] dB. The classifier’s objective was to categorize these phoneme representations based on their source sentence category, i.e., “Clean-style text” or “Noisy-style text.”

2. Classification 2: We categorized the inference input text into four style classes based on the SNR of the mixed speech from which the corresponding ASR transcriptions were generated: Class 1 (SNRs in the range [−5, 0) dB), Class 2 (SNRs in the range [−10, −5) dB), Class 3 (SNRs in the range [−15, −10) dB), and Class 4 (SNRs in the range [−20, −15) dB). Each class comprised 2,000 phoneme representations, totaling 8,000 representations. The classifier’s objective was to categorize these representations according to their source sentence’s SNR-defined style class (i.e. Class 1 to Class 4).

The phoneme representations used in both classification tasks were 192-dimensional output vectors extracted from the Text Encoder of the WER-358 noisy models from all evaluated TTS systems. For both Classification 1 and Classification 2, Linear Discriminant Analysis (LDA) was used to reduce the dimensionality of the representations and subsequently to perform supervised classification. The entire data set for each classification task was split into training (80%) and test (20%) sets using stratified sampling to ensure consistent category distribution across the splits. Classification performance was evaluated based on accuracy achieved on the test set. Classification experiments are implemented based on scikit-learn ^[5], and LDA uses the default parameter settings.

The classification results are presented in Table 3. For Classification 1, the accuracies achieved using representations from all models were significantly higher than random chance (50% for binary classification). Similarly, in Classification 2, the results indicated that the models also exhibited representations separation for noisy transcriptions of varying degradation levels. Across both classification tasks, the accuracies achieved by VITS-DP, GradTTS, DVT and DVT2 were notably superior to that of Tacotron 2. This suggested that these TTS systems exhibited more pronounced latent text feature separation when trained on noisy transcriptions compared to the direct AR-based model (Tacotron 2), a finding consistent with observations from the sensitivity tests. In addition, we observed that representations from VITS-DP, DVT and DVT2 demonstrated superior separation. This stronger separation in VAE-based architectures may be related to their training dynamics. Unlike GradTTS, which relies solely on score matching objectives, VITS-DP, DVT and DVT2 all use Variational Autoencoder (VAE) frameworks that impose dual training objectives: accurate reconstruction alongside conformance to a prior distribution through KL divergence regularization. We speculate that when trained on heterogeneous data with mixed quality, these constraints encourage structured organization of representations. Specifically, well-matched text-speech pairs may occupy coherent regions of the latent space, while mismatched pairs are pushed toward distinct regions. This structure-inducing property may facilitate the observed domain separation. Furthermore, DVT2’s marginally superior separation compared to DVT aligns with findings that more complex text encoder architectures benefit model robustness under challenging data conditions (Li et al., 2020; Yasuda et al., 2021; Zhu et al., 2023).

Notably, a model’s synthesis performance was not directly correlated with its representation separability; for instance, VITS-DP, despite exhibiting excellent separation performance, did not achieve commensurate synthesis quality in the evaluations reported in Section 5. According to the experimental results, we speculate this is because latent low- and high-quality text feature separation, although it can alleviate the impact of noisy data on high quality feature learning, cannot directly determine the quality of the features learned by the model. We hypothesize that diffusion-based modeling systems tend to be more robust in handling noisy data and complex distributions compared to flow-based systems. To further investigate this hypothetical in text representations, we designed a simple supplementary experiment. In this experiment, we used the text representations extracted from a pre-trained DVT WER-358 noisy model to train a VITS model on WER-358 transcriptions. Using text representations from the DVT model significantly improved the performance of the VITS WER-358 model. Specifically, it achieved a synthesis accuracy of 5.5% CER and 10.6% WER on “Clean-style Text”, achieving performance nearly identical to the original DVT WER-358 model (5.4% CER and 10.9% WER), with differences within 0.1% CER and 0.3% WER. This outcome corroborated the robustness and transferability of text representations learned by the diffusion-based DVT system. Furthermore, it highlighted the critical role of text representation quality for the accuracy of noisy model synthesis.

In Section 6.2, we observed that noisy models exhibit latent domain separation, where text representations are spontaneously organized into distinct subspaces. This raises a critical question: when a noisy model produces degraded synthesis quality, can we guide it to preferentially activate high-quality representation domains during inference?

Because text representations are determined solely by input text characteristics, we hypothesize that prepending text fragments known to produce high-quality synthesis (serving as “prompts”) to target sentences might guide the entire input toward the high-quality domain through contextual influence.

To test this hypothesis, we designed a controlled experiment using text prompts. Critically, we selected the same 250 samples from the “Noisy-style Text” set used in Section 6.2 as synthesis targets specifically because they consistently triggered low-quality representations in our previous tests, providing a challenging scenario to evaluate the prompt’s effectiveness. If prompts can improve synthesis quality even for these problematic inputs, it would demonstrate the controllability of domain activation.

In this experiment, text fragments that the model can correctly synthesize were selected as “text-prompt” and concatenated with the sentences intended for synthesis (termed “synthesized-target”) to form the complete inference input. Each “synthesized-target” was connected to a randomly selected “text-prompt” using one of three methods: Header Connection (H-Connect), Tail Connection (T-Connect), and Header-Tail Connection (HT-Connect). The WER-358 noisy models from all TTS systems were used for this analysis. In addition, we introduced a “Twice-length” condition, where the “synthesized-target” was repeated twice, to assess sensitivity to input length. The intelligibility of synthesized speech was evaluated using ASR, consistent with Section 5.2. For WER calculation, “text-prompt” portions were excluded from the synthesized audio prior to ASR processing, with WER computed solely on the “synthesized-target” portion.

The experimental results are presented in Figure 6. The models exhibited minimal sensitivity to input length, with “Twice-length” performance only marginally inferior to direct synthesis. For Tacotron 2, all three prompt connection methods yielded only marginal improvements over the baseline, likely due to insufficient feature separation. In contrast, VITS-DP, GradTTS, DVT and DVT2 exhibited significant performance gains under the “text-prompt” condition. VITS-DP achieved optimal performance with H-Connect, GradTTS with T-Connect and DVT/DVT2 with both T-Connect and HT-Connect.

These results confirm that text prompts can effectively guide noisy models toward high-quality representation domains, thereby validating the controllability of latent domain separation. An important implication is that noisy models face an inference challenge: without explicit quality indicators in the input text, they cannot reliably activate high-quality representations, resulting in inconsistent synthesis performance. The text prompting strategy addresses this limitation by providing contextual guidance that biases the model toward the high-quality domain. The inspiration for this approach comes from observations in prior work on contextual conditioning: for instance, techniques such as using clean acoustic context have been explored to improve synthesis quality in challenging scenarios (Chen et al., 2025). Similarly, our text prompting leverages the sequential nature of both Transformer and RNN-based text encoders, where contextual information from earlier tokens propagates to influence later representations. Clean text prompts – free from the systematic error patterns characteristic of ASR output (Frieske and Shi, 2024; Goldwater et al., 2010) – establish a contextual state that guides the encoding of subsequent text toward the high-quality representation domain. Our empirical results (Figure 6) demonstrate that this contextual influence is sufficiently robust to maintain high-quality synthesis even when the synthesis target itself contains characteristics that might otherwise trigger lower-quality representations. This observation suggests that the balance of text patterns in the input sequence may play a role in which domain the model predominantly activates, with the prompt providing sufficient high-quality context to guide the overall encoding.

While the controlled experimental setup using “Noisy-style Text” serves primarily to validate this mechanism under challenging conditions, the underlying principle has practical merit: prepending reliably synthesizable text fragments can stabilize synthesis quality in deployed noisy models without requiring retraining. This approach represents a lightweight mitigation strategy, particularly valuable for model debugging or as an interim solution during iterative development. However, identifying suitable prompts for arbitrary inputs at scale remains a challenge for future work.

This study investigated the behavior of TTS models trained on noisy transcriptions. We evaluated the synthesis performance of five TTS systems (Tacotron 2, GradTTS, VITS, DVT and DVT2) representing different architectures under noisy transcription training conditions, and analyzed the mechanisms that influence their performance. Our experiments demonstrate that diffusion-based architectures exhibit a clear robustness advantage in complex modeling scenarios involving noisy transcriptions. This advantage is reflected in the production of higher-quality text representations. We observed that models trained on noisy transcriptions spontaneously learn to separate text representations into latent domains of high and low quality. This behavior is crucial as it helps mitigate the negative impact of contaminated information during training. We further investigated a text prompt strategy, which effectively mitigates degradation during inference without requiring architectural modification or fine-tuning. This work deepens our understanding of how internal model behaviors relate to robustness under noisy conditions and provides insights into architectural choices for training TTS systems with imperfect data. Our finding that latent domain separation occurs spontaneously in the absence of explicit quality labels offers a promising direction for future research on improving training stability and generalization in low-quality or weakly supervised scenarios. Future work will extend this investigation to large-scale real-world ASR-transcribed data sets to validate the generalization.

Funding: This work was supported by JST CREST (JPMJCR22D1), JSPS KAKENHI (21H05054), and JST SPRING (JPMJSP2125).