Expanded noise modeling for scalable and adaptive zero-shot speech enhancement

Speech enhancement (SE) aims to improve the perceived quality and intelligibility of speech degraded by noise, reverberation or other environmental interference. While supervised deep learning methods (Luo and Mesgarani, 2019; Yen et al., 2023; Hu et al., 2023; Fu et al., 2021) have achieved remarkable performance, they typically depend on large-scale, domain-matched training data sets containing paired noisy and clean samples. In many real-world applications, however, collecting such well-matched labeled data is impractical. This mismatch between training and deployment domains often results in a significant performance drop. In recent years, diffusion models have shown superior performance across various domains because of their robustness and strong generative capabilities, and this advantage also extends to the field of SE (Yang et al., 2024; Sadeghi et al., 2025; Gonzalez et al., 2024; Zhao et al., 2025).

To address this limitation, zero-shot SE has emerged as a promising paradigm. Most research in this area has focused on speech synthesis, speech conversion and text-to-speech tasks, while studies dedicated specifically to SE remain relatively limited (Sadeghi et al., 2025), leaving significant room for exploration. Unlike fine-tuning on target-domain data, zero-shot methods aim to generalize directly to unseen acoustic conditions without requiring task-specific retraining. This capability is particularly advantageous for deployment in diverse and unpredictable environments, where noise and reverberation characteristics may vary significantly over time. In addition, zero-shot methods reduce the computational and data collection costs associated with retraining, enabling faster and more scalable adaptation to real-world applications.

Despite these advantages, zero-shot SE remains highly challenging. The diverse distributions of speech and noise, coupled with the absence of in-domain training data, make it difficult for models to capture the acoustic variability required for robust enhancement. Existing methods often face a trade-off between generalization and speech fidelity, and their performance is further constrained under extreme distortions or rare acoustic conditions. These challenges underscore the need for novel architectures and learning strategies. To this end, we propose expanded noise modeling for adaptive zero-shot speech enhancement (EN-AZS), which leverages broadened noise coverage together with generative modeling to maintain strong performance under previously unseen acoustic conditions.

SE is a fundamental task in speech processing, aiming to improve the clarity and quality of speech degraded by ambient noise or reverberation. Traditional approaches such as spectral subtraction (Upadhyay and Karmakar, 2015), Wiener filtering (Upadhyay and Jaiswal, 2016; Abd El-Fattah, 2014) and statistical model-based methods (Ephraim, 2002) are computationally efficient but rely on strong assumptions about noise distributions. Consequently, they often perform poorly in complex or nonstationary environments. With the advent of deep learning, supervised SE methods have achieved remarkable progress by directly mapping noisy speech to clean targets using architectures such as convolutional neural networks, recurrent neural networks and transformers. Despite their success, these models typically require large amounts of domain-matched training data, limiting their adaptability to unseen acoustic conditions.

To mitigate domain mismatch, zero-shot SE has emerged as a promising alternative. Instead of retraining or fine-tuning on in-domain data, zero-shot methods aim to generalize directly to unknown acoustic conditions. Techniques such as leveraging large-scale pre-trained representations (Baevski et al., 2020; Wang et al., 2022), incorporating adaptive front-ends (Casebeer et al., 2020; Zhang et al., 2025) and dynamically estimating noise characteristics (Lee et al., 2020) have been explored. These approaches offer greater flexibility and reduce the need for costly data collection, yet they often struggle with highly diverse or rare noise types, resulting in trade-offs between generalization and speech fidelity.

More recently, diffusion models have gained attention in SE due to their strong generative capabilities. By iteratively refining noisy inputs, diffusion-based methods can produce high-quality and perceptually natural speech. However, their iterative sampling process introduces high computational costs, restricting their deployment in real-time or resource-limited scenarios. To mitigate this, fast sampling and knowledge distillation (Lu et al., 2022; Zhou et al., 2024; Salimans and Ho, 2022) strategies have been proposed, though they typically involve compromises between speed and enhancement quality.

In summary, traditional methods offer efficiency but lack robustness, while deep learning-based SE achieves high-quality results but requires domain-specific data. Zero-shot SE reduces data dependency but is challenged by noise diversity, and diffusion-based approaches deliver natural outputs but incur heavy computational demands. These limitations underscore the need for approaches that combine efficiency, generalization and robustness, paving the way for breakthroughs in zero-shot SE.

The main contributions of this work can be summarized as follows:

1. Noise-speech energy estimation in unknown environments: We design a framework capable of accurately estimating the energy difference between noise and speech without requiring in-domain data.

2. Expanded noise modeling for zero-shot SE: We address a key limitation of existing zero-shot approaches by enlarging the range of noise categories, improving generalization across diverse acoustic conditions.

3. Optimization of the Undiff model: We extend and refine the Undiff (Iashchenko et al., 2023) framework for SE tasks, enhancing its robustness and moving closer to plug-and-play SE.

Together, these contributions establish a foundation for practical, scalable and adaptive zero-shot SE, bridging the gap between theoretical advances and real-world deployment.

Early research on zero-shot SE can be broadly categorized into several approaches. For example, Kim and Kim (2021) proposed a knowledge distillation framework using a teacher-student paradigm, where a well-trained teacher model guided the student model to estimate clean speech in a zero-shot setting. Similarly, Zezario et al. (2021) and Sivaraman and Kim (2021) addressed unknown acoustic environments by classifying them into categories and employing a pre-trained gating module to select the most appropriate expert model. The gating decisions were based on factors such as speech quality, signal-to-noise ratio (SNR), speaker characteristics and noise sources, organized within a hierarchical structure.

More recently, advances in zero-shot SE have leveraged additional information or novel training strategies, including text-to-speech systems (Fujita et al., 2024), meta-learning (Mannanova et al., 2024; Yu et al., 2021) and data augmentation (Tzinis et al., 2022), among others. Despite these developments, managing complex and diverse environments remains a major challenge, motivating the exploration of personalized approaches. Such methods simplify environmental variability and improve adaptability to real-world conditions. However, many personalized approaches share similarities with supervised fine-tuning, where small amounts of additional training data are used to boost performance. While effective, this reliance on extra training constrains their general applicability.

In this study, we adopt Sunwoo Kim’s method as our baseline. For further comparison with supervised approaches, we also evaluate MetricGAN+ (Fu et al., 2021).

Our proposed EN-AZS method is built upon the Undiff (Iashchenko et al., 2023) framework and addresses its limitations in SE tasks. Undiff is an unsupervised learning approach based on the DiffWave architecture. As shown in Figure 1, the entire system comprises two phases: training and testing. During training, EN-AZS follows the same procedure as conventional diffusion-based generative models, aiming to reconstruct the input signal. The speech model and noise models (including the extended noise models) share the same DiffWave architecture but are trained independently on different data sets. The extended training phase follows the same procedure as standard training, but multiple noise reconstruction models are trained to cover a broader range of noise types. As the system is further expanded, it enables flexible selection and combination of noise models according to specific application needs, thereby supporting plug-and-play functionality.

In the testing phase, we impose joint constraints on the outputs of multiple trained models. Specifically, the system first excludes speech inputs with minimal noise contamination and then constrains the output signals of multiple models during the reverse iteration process to achieve zero-shot SE.

As shown in Figure 1, the SE framework consists of a speech model and multiple noise models. In this study, all models share the same architecture and use the unconditional version of DiffWave (Kong et al., 2020). The same as the traditional diffusion model (Ho et al., 2020) comprises a forward (diffusion) process and a reverse process. Let x₀ ∈ ℝ^L denote a clean speech signal of length L. For each diffusion step t, we define a step-dependent variable x_t ∈ ℝ^L. Following the framework of diffusion probabilistic models, the forward diffusion process is formulated as follows:

Here, β denotes a pre-defined noise variance schedule. Given a sufficiently large number of diffusion steps T, the reverse process progressively reconstructs the latent variable x_T back to the original signal x₀, The speech model reconstructs the clean speech input, while the noise model does the same for noise. Under the Markov assumption, the marginal distribution at an arbitrary time step t can be expressed in closed form as follows:

where p_θ(·) is the distribution of the reverse process with learnable parameters θ, the term μ_θ(x_t, t) predicts the mean of x_t–1 distribution by removing the estimated Gaussian noise ϵ_θ(x_t, t) in x_t, while $βt′$ means variance. The model is trained by minimizing a variant of the variational lower bound (ELBO) given in equation (6). The network architecture of the unconditional DiffWave is illustrated in Figure 2. Here, unconditional indicates that spectral conditioning has been removed. The blue squares within the dashed border represent multiple modules that share the same architecture as the solid blue squares. These modules are connected sequentially, with the output at the bottom left of each module linked to the input at the top left of the next. The input at the top right of each blue square receives a shared diffusion-step embedding, which is propagated across all layers to maintain consistent step information. Finally, features from all layers are aggregated through skip connections and reconstructed into speech via a convolution layer. Detailed model settings are shown in Table 1. Here, B represents the batch size, L represents the input data length, and c refers to the dilation setting in the dilated convolution c changes according to the current block level, with c = 0, 1, …, 29:

According to Undiff, inverse problems involve retrieving the object x from its partial observation y and the degradation model p(y|x). The inverse process can be expressed using Bayes’ rule as:

thus:

where $∇xtlogpt(y|x^0)$ can be computed using the degradation model. Given observation operator 𝐴 and assuming a Gaussian likelihood, the final approximation becomes:

where ξ(t) is the weighting coefficient which we set to be inversely proportional to the gradient norm. Depending on the given y, the operator A can represent various downstream tasks, such as bandwidth extension, de-clipping, source separation, and more. In this experiment, we further extend the approach based on the source separation task. For operator A in source separation, they mentioned that: Consider the observation model which mixes these two recordings, i.e. y = A(x₁, x₂) = x₁ + x₂. Note that since x₁ and x₂ are independent, unconditional density function on their joint distribution can be factorized as p(x₁, x₂) = p(x₁) · p(x₂). We define y as a noisy speech signal and decompose it into x₁, x₂, x₃, …, and then apply this framework to the SE task. However, several challenges still need to be addressed, such as:

1. Undiff cannot achieve perfect source separation, and the existence of multiple sources will lead to substitution problems, which give rise to the segment substitution problem and source substitution problem mentioned in the next section.

2. In the SE task, accurate energy evaluation of speech and noise is crucial, as it strongly influences speech quality.

3. Since noise spans a wide range, it is important to avoid retraining a new noise model from scratch whenever new environmental conditions arise.

First, regarding the limitations of Undiff in SE tasks, our experiments reveal that the permutation problem consists of two aspects. The first is the segment substitution problem, where segments belonging to the speech channel are incorrectly identified as originating from the noise channel, rather than being properly aligned within the same channel. As shown in Figure 5 (Section 4.4), this misalignment causes speech segments to become scattered across different channels, leading to the loss or distortion of certain speech features.

The second is the source substitution problem. Although the primary goal of this study is SE, the task inherently involves source separation, and the output of the noise channel is not directly relevant. However, the source substitution issue remains: after separating speech and noise, the true source cannot be directly determined. Consequently, under joint constraints and after multiple iterations, the noise channel gradually produces outputs that increasingly resemble clean speech, while the speech channel becomes contaminated with noise. Both problems become more pronounced as the number of noise models increases.

To address these issue, we propose a speech quality guidance (SQG) method, as illustrated in Algorithm 1. Algorithm 1 outlines how time point t is adjusted using SQG during the reverse process. The inputs include a noisy speech signal y, a speech model at time point t (the current output of the speech model is represented as $St$⁠), and k noise model output $Ntk$ at time point t. Each model produces an output, referred to as a channel; for instance, the reconstructed speech from the speech model corresponds to the speech channel. The speech quality score is computed using SQG, and the output of the speech channel is individually adjusted before being combined with the outputs of the noise channels for joint constraint. This approach enhances the quality of the generated speech and effectively mitigates the segment substitution problem across multiple channels:

Algorithm 1. Sampling with SQG.

Next, we optimize the energy evaluation. To facilitate future research, we apply amplitude normalization to all model inputs, further reducing the influence of input energy levels on the model. We then use our proposed coefficient calculation (CC) method to accurately estimate the signal-to-noise ratio. Following the previous formulation y = A(x₁, x₂) = x₁ + x₂, we assign an independent coefficient γ to each signal to represent its energy scaling, as shown in equation (11):

where k represents the number of noise models, and the speech and noise correspond to their respective coefficients γ_α and γ_β. In our experiments, the coefficients can be obtained in two ways: one is to directly differentiate γ and obtain them through a matrix formulation. The other method is to use backpropagation and learn the respective coefficients through iteration. The complete single-step sampling process is shown in Algorithm 2:

Algorithm 2. Sampling with CC

Finally, we combine the two algorithms described above, as illustrated in Algorithm 3. Specifically, the SQG component corresponds to lines 2–3, the CC component to lines 10–17, and the update steps to lines 7–9 and 18–20. In Section 4.5, we systematically examine the different combinations of SQG and CC, as well as the influence of the number of updates.

Algorithm 3. Sampling with both SQG and CC

In terms of training, this study uses three data sets. Among them, three are custom constructed. The speech source is the DARPA-TIMIT corpus (Garofolo et al., 1993), which contains recordings from 630 speakers, each reading 10 phonetically rich sentences. The noise source is the NoiseX-92 data set (Varga and Steeneken, 1993), which includes 15 noise types. Based on the number of noise types, we construct three subsets: N6, N9 and N15. N6 includes volvo, pink, m109, f16, destroyerops and babble; N9 contains the remaining nine noise types; and N15 includes all 15 types. We mix DARPA-TIMIT with N6, N9 and N15 to train our method, and denote the data sets as TIMIT-N6, TIMIT-N9 and TIMIT-N15. All three share the same test set, created by mixing the first 25 utterances from the TIMIT test subset with 15 different types of noise clips from NoiseX-92.

For cross-validation, we further use two additional data sets: VCTK-DM and MUSAN (Snyder et al., 2015). VCTK-DM was constructed from the CSTR VCTK corpus (Veaux et al., 2013) and mixed with 10 noise types [two artificial and eight from the DEMAND database (Thiemann et al., 2013)] under four SNR conditions: 15, 10, 5 and 0 dB. The training set includes 28 speakers, each reading 10 sentences under every noise condition, while the test set includes two speakers, each reading 20 sentences under five noise types and four SNR levels (17.5, 12.5, 7.5 and 2.5 dB). The MUSAN corpus (Snyder et al., 2015) is divided into speech, noise and music, making it well-suited for our experiments. It contains about 109 h of audio. For our test set, we premixed portions of MUSAN speech with noise and music, while the remaining data was used to train speech, noise and extended models, respectively.

Except for VCTK-DM, all training sets use five SNR levels: −10, −5, 0, 5 and 10 dB. The corresponding test sets use −5, 0 and 5 dB. All audio data were resampled to 16 kHz for consistency across experiments. The detailed data set specifications are listed in Table 2.

This study provides a detailed analysis of the improvement in speech quality achieved by introducing SQG during the reverse process. As shown in Table 3, the impact of SQG on speech quality varies depending on the evaluation criteria. Several objective evaluation methods are compared in the Method column of Table 3. In references (Manocha and Kumar, 2022; Rosenbaum et al., 2023), a reference speech was introduced instead of the corresponding clean speech to enable a more accurate assessment of perceptual quality; this approach is denoted as “Unpair Ref” in Table 3. To further assess the generalization capability of the proposed method under unseen noise conditions, the model was trained on the TIMIT-N9 data set and evaluated on the TIMIT test set.

Next, using the same training and test sets, we evaluated two CC methods: a matrix-based approach and a backpropagation-based (BP-based) approach. In the BP-based method, we set a maximum of 1000 iterations. However, the BP-based optimization was slow: 1,000 iterations took on average 1.86 s in total. Consequently, a complete reverse process requiring T such runs would take approximately 1.86 × T seconds, which is prohibitively long. To mitigate this, we implemented an early stopping mechanism that terminates the process if the loss does not improve for 20 consecutive iterations. The results are presented in Table 4 (Execution Time). As shown, the BP-based method was less effective and considerably more time-consuming; therefore, we adopted the matrix-based approach for all subsequent experiments.

In Algorithm 3, both SQG and CC directly adjust the speech and noise signals. These adjustments may be accompanied by updates, though they are not always required. Since the update strategy can significantly affect the final results, we evaluate both the number and timing of updates through an ablation study, as shown in Table 5. As reported in Table 5, both SQG and CC individually lead to significant improvements in overall performance. Regarding the number of updates, the SQG+CC combination achieves optimal results with only one update. Based on the error calculations in column 6 of Algorithm 3, there is still room for improvement. Note that one update in the table corresponds to T iterations, while two updates correspond to 2T iterations. Excessively increasing the number of updates can lead to overfitting to the observed mixture y, thereby degrading speech quality. Figures 3–5 compare the generated spectra using SQG and CC. Figure 3 presents the complete sampling results of EN-AZS. The green boxes in panels (a) and (c) indicate that although EN-AZS does not completely remove noise, it preserves most speech features within the generated channels of the speech model. Figure 4 illustrates that without SQG, speech features become dispersed across both the noise and speech model channels. A comparison between Figures 3(c) and 4(b) confirms that the substitution issue of Undiff in SE tasks has been resolved. Figure 5 demonstrates that without CC, most speech features are retained in the speech model channels; however, due to the absence of an energy estimation mechanism, segments with higher speech energy are still misclassified as belonging to the noise channel.

Based on the above experimental results, we compared extended training and retraining of the noise model. The procedure is as follows: (1) For extended training, we first trained the TIMIT speech model, then trained multiple noise models (N9 and N6) as the basis for our method. (2) For retraining, we used the TIMIT speech model but combined the noises of N9 and N6 into a single noise set (N15) and retrained it. The results are shown in Table 6. The table shows that the difference between extended training and retraining is minimal, while better results can be obtained by increasing the number of generated samples.

Next, we compare our proposed method with two baseline approaches: the unsupervised method PKD (Kim and Kim, 2021) and the supervised method MetricGAN+ (Fu, 2021). PKD adopts a knowledge distillation-based teacher–student framework, where the teacher model is Conv-TasNet (Luo and Mesgarani, 2019) and the student model is a unidirectional gated recurrent unit (GRU) network designed for spectral mask estimation. MetricGAN+ uses a generative adversarial network architecture, in which a bidirectional LSTM serves as the generator for spectral mask estimation, while Quality-Net (Fu et al., 2018) acts as the discriminator to evaluate the speech quality of the generated speech and guide the training process. For the experimental settings, our proposed method follows the configurations of Diffwave, and the baselines are kept as consistent as possible with their original implementations.

In the following experiments, we first compare the performance of the proposed EN-AZS method with these baselines on two in-domain data sets. The supervised method is fine-tuned during the in-domain testing phase, whereas the unsupervised PKD method skips the fine-tuning (personalization) step and is evaluated directly. As shown in Table 7, TIMIT-N15 differs substantially from VCTK-DM in terms of noise type, speech characteristics and signal-to-noise ratio. Since the speech in the VCTK-DM test set is relatively clean, the performance of MetricGAN+ (without or with insufficient fine-tuning) is even worse than the noisy baseline. For unsupervised methods, PKD performance is notably affected by data set characteristics, whereas the proposed EN-AZS demonstrates much greater robustness. Table 8 shows a similar trend.

To further validate the robustness and scalability of EN-AZS, we trained it on the larger MUSAN corpus and evaluated it on the two previous test sets (cross-domain). For the baselines, we manually mixed noise and music samples from MUSAN with speech to create paired noisy–clean data sets. For fine-tuning, approximately 2 GB of data from the target domain (VCTK-DM or TIMIT-N15 training sets) were used to fine-tune the baseline models. For the extended training of EN-AZS, the same amount of data was used, and the corresponding results are presented in Tables 9 and 10. As shown in the tables, while fine-tuning provided modest improvements for the baseline models, these gains were limited by the small size of the fine-tuning data set and the model architecture. Nevertheless, the extended training remained effective under these constraints.

In this work, we propose EN-AZS, a zero-shot SE framework with expanded noise modeling, aimed at improving robustness under unseen acoustic conditions. By extending the noise model and optimizing the Undiff-based architecture, EN-AZS achieves strong generalization across multiple data sets without requiring task-specific fine-tuning. Extensive experiments on TIMIT-N6/N9/N15, VCTK-DM, and MUSAN demonstrate that EN-AZS consistently outperforms both supervised and unsupervised baselines, particularly under mismatched noise types, speech characteristics and SNR levels. Ablation studies further show that the combination of SQG and CC mechanisms, together with carefully controlled update strategies, is critical for maximizing speech quality while avoiding overfitting to observed mixtures. Overall, EN-AZS offers a practical and scalable solution for zero-shot SE, effectively bridging the gap between theoretical generative modeling and real-world deployment.

In future work, we plan to incorporate additional acoustic factors, including reverberation and cross-lingual speech, refine the error calculation methodology in the reverse process, and investigate adaptive strategies to further improve personalization and robustness in highly diverse or challenging noise conditions.