Multi-step prediction and control of hierarchical emotion distribution in text-to-speech synthesis

Text-to-speech (TTS) synthesis focuses on generating human-like speech from text input (Tan et al., 2021). Advancements in deep learning have significantly improved the naturalness and quality of synthesized speech. However, current TTS systems still struggle with conveying emotional expressiveness and precisely controlling emotional nuances, limiting their ability to deliver humanlike expressive speech (Triantafyllopoulos et al., 2023). To address these limitations, Emotional TTS aims to bridge this gap by enhancing speech expressiveness, enabling more engaging and empathetic dialogue systems with emotional intelligence (Triantafyllopoulos and Schuller, 2024).

Emotional TTS faces challenges stemming from the hierarchical structure of human emotions (KUN, 2022). Since speech emotion is characterized by distinct prosodic patterns at the phoneme, word, and utterance levels (Hirschberg, 2006; KUN, 2022; Triantafyllopoulos et al., 2023), these patterns naturally form a hierarchy, as established in previous studies (El Ayadi et al., 2011; Schuller, 2018). The prior literature indicates that modifying only global prosodic attributes does not capture the full complexity of emotional speech (Latorre and Akamine, 2008; Xu, 2011; Zhou et al., 2023b; Zhou et al., 2020). Additionally, prior work in text-to-speech synthesis and emotional voice conversion underscores the necessity of multi-level modeling (Lei et al., 2022b; Ming et al., 2016; Zhou et al., 2020). Consequently, developing a method to model the hierarchical structure of emotions is essential for generating nuanced speech synthesis. However, existing text-based emotion representation prediction networks in controllable models address phoneme-level variations (Lei et al., 2022a; Ren et al., 2022), overlooking the benefits of multi-level emotion modeling.

In this work, we build upon our previous work on a multi-level quantifiable method for speech emotion control (Inoue et al., 2024b) and editing (Inoue et al., 2024a) by proposing a multi-step prediction framework for hierarchical emotion distribution (ED) derived from textual cues. Our proposed pipeline supports three inference scenarios, as illustrated in Figure 1:

• Text-to-Speech (TTS) with Emotion Prediction, where the hierarchical emotion distribution (ED) is directly predicted from the input text.

• TTS with Emotion Control, where the ED is predicted from the text and can be modified by users.

• Emotion Editing, where the ED is extracted from input audio and manually adjusted by users.

In (Inoue et al., 2024a; Inoue et al., 2025; Inoue et al., 2024b), a hierarchical ED was introduced to enable both global and fine-grained emotion modification in speech generation. Unlike prior single-step ED prediction approaches (Inoue et al., 2024b), which treat different levels of emotion variance independently, we propose to explicitly model hierarchical dependencies by predicting EDs at the utterance, word, and phoneme levels in a multi-step manner. This structured approach ensures that higher-level emotional context influences lower-level prosodic details, resulting in a more coherent, expressive, and controllable emotional rendering. By leveraging multi-step ED prediction, our method provides fine-grained control, closely mimicking the way humans modulate speech—starting with an overall tone and refining intonation and articulation dynamically. This leads to an improved performance on both emotion expressiveness and speech naturalness. Furthermore, to demonstrate its flexibility, we explore two integration strategies for hierarchical ED: implementing it as a variance adaptor within FastSpeech2 (Inoue et al., 2024b) and incorporating it as an external module compatible with any text-to-speech (TTS) model (Inoue et al., 2024a). Through this approach, we bridge the gap between interpretability and fine-grained emotion control. Our contributions are summarized as follows ^[1]:

• We introduce a multi-step prediction framework for hierarchical emotion distribution (ED), where the utterance-, word-, and phoneme-level EDs are derived from textual cues in successive steps. This structured approach ensures that higher-level emotional context guides low-level prosodic details, resulting in synthesized emotional speech that is both natural and expressive.

• Leveraging the multi-step prediction of EDs, our method achieves refined control over emotion rendering and closely emulates human speech modulation. This approach not only enhances global and nuanced emotional expressiveness but also significantly improves performance in controlled emotional voices.

• We explore two integration strategies for hierarchical ED into TTS systems: (1) embedding it as a variance adaptor within FastSpeech2 and (2) implementing it as an external module, making it adaptable to various text-to-speech (TTS) systems.

The rest of this paper is organized as follows: In Section 2, we discuss the related works. Section 3 describes our proposed methodology. In Section 4, we summarize experimental setups. In Section 5, we report our experiments and results. Section 6 concludes our study.

In this section, we briefly introduce related studies to set the stage for our research and highlight the novelty of our contributions. We begin by discussing the hierarchical nature of speech emotions, emphasizing the need for multi-level and multi-step emotion modeling. We then review existing approaches to emotion rendering control in the TTS literature, identifying key advancements and gaps that our work addresses.

Speech emotions manifest hierarchically across three levels: utterance, word, and phoneme. At the utterance level, previous studies have shown that global prosodic patterns—such as pitch contour, range, mean, intonation, tempo, and rhythm—play a crucial role in conveying emotion (Rodero, 2011). At the word level, lexical cues shape emotional tone (Warriner et al., 2013) and enhance intensity through emphasis (Hirschberg and Ward, 1992). Additionally, research suggests that when lexical and prosodic signals conflict, listeners tend to rely on prosodic cues to interpret emotion (Snedeker and Trueswell, 2003). At the phoneme level, individual prosodic features such as pitch, energy, and duration contribute to emotional expression (Krothapalli and Koolagudi, 2013), as supported by multiple studies (Busso et al., 2009; El Ayadi et al., 2011; Leinonen et al., 1997). This hierarchical structure highlights the necessity of studying multi-level and multi-step emotion modeling for effective emotion modeling and control.

Recent advancements in emotional TTS have significantly enhanced expressiveness; however, achieving interpretable emotion control remains a challenge. Prior studies have primarily focused on refining emotion intensity control by treating speech emotion as a global feature and manipulating representations or attributes derived from reference audio. For instance (Oh et al., 2023), enables utterance-level control via a speech mixer that predicts pseudo-labels and modulates features such as pitch, duration, and energy (Li et al., 2022; Zhang et al., 2023), further controlled emotion by manipulating speaker-disentangled representations in cross-speaker scenarios. Recent studies employ relative attributes (Parikh and Grauman, 2011) for intensity control (Zhou et al., 2023a; Zhu et al., 2019). Approaches for mixed emotions include manipulating relative attributes (Zhou et al., 2022), incorporating noise mixing in diffusion models (Tang et al., 2023), and leveraging continuous emotional representations (Zhou et al., 2024). In contrast, EmoSphere-TTS (Cho et al., 2024b; Cho et al., 2024a) models emotional complexity via a spherical emotion vector through Cartesian-spherical transformations, while (Jing et al., 2024) integrates computational paralinguistic text prompts to enhance emotional expressiveness.

To achieve fine-grained emotion control, several studies have explored segmental-level representations. For example, MsEmoTTS (Lei et al., 2022a) employs a global emotion label and modifies phoneme-level intensity via relative attributes. EmoQ-TTS (Im et al., 2022) quantifies emotion intensity using a distance-based method, while the study in (Wang et al., 2023) refines control by examining inter- and intra-class distances. Additionally, CASEIN (Cui et al., 2023) leverages a speech emotion recognition module to predict phoneme-level emotion distributions. These multi‐ or phoneme‐level approaches outperform utterance‐level modeling in emotional speech synthesis (Cui et al., 2023; Im et al., 2022; Lei et al., 2022a) Building upon these efforts, our previous work introduced hierarchical emotion distribution (ED) (Inoue et al., 2024a; Inoue et al., 2024b), which enables multi-level emotion control, capturing both global and fine-grained emotional variations in speech synthesis.

However, existing approaches to hierarchical emotion modeling still face several limitations. Previous methods (Inoue et al., 2024a; Inoue et al., 2025) rely on single-step prediction strategies, which treat different levels of emotion variance independently and fail to capture the contextual dependencies between hierarchical emotion distributions. Additionally, current techniques often lack a structured mechanism to ensure that higher-level emotions influence lower-level prosodic variations, leading to inconsistencies in emotion expressiveness. Furthermore, integration with TTS remains another challenge, as most approaches are either model-specific or require reference audio, limiting their adaptability across different TTS architectures. Addressing these gaps, we propose a multi-step ED prediction framework that models hierarchical ED progressively, ensuring a more interpretable, flexible, and effective approach to speech emotion rendering and control.

We propose a novel approach, which supports the rendering of both single emotions and mixed emotions, that can be seamlessly integrated with various text-to-speech frameworks. Traditionally, speech databases label emotions at the utterance level, overlooking nuanced intensity variations within speech. To address this challenge, we automatically generate fine-grained, quantitative intensity labels, which serve as “soft labels” for speech generation models, eliminating the need for manual annotation. This method effectively enhances emotion control, enables mixed-emotion rendering, and can be readily adapted to speech generation frameworks, including text-to-speech and voice conversion.

Built upon our previous studies (Inoue et al., 2024a; Inoue et al., 2024b), the hierarchical emotion distribution (ED) extraction module integrates OpenSmile feature extractors (Eyben et al., 2010) with pre-trained ranking functions at each segmental level to quantify emotion intensities in an utterance, as shown in Figure 2. Grounded in relative attributes (Parikh and Grauman, 2011), our method measures emotion prominence by treating emotion style as a speech attribute and ranking its presence relative to other emotions, enabling a structured and interpretable approach to hierarchical emotion quantification.

Specifically, we define the ranking function as:

where $xi$⁠, $w$⁠, and b denote the acoustic features of the i-th sample, weight vector, and bias, respectively. We optimize these parameters using a support vector machine objective for binary classification (e.g., Angry vs. Non-angry) (Cortes and Vapnik, 1995) and normalize the outputs to the range [0, 1], with larger values indicating stronger emotion intensity. This process enables continuous labeling of training data and the quantification of emotion intensity in unseen utterances during run-time.

Figure 2(b) illustrates our hierarchical ED extractor. We begin by segmenting the input audio into phoneme, word, and utterance levels using the Montreal Forced Aligner (McAuliffe et al., 2017), and extracting an 88-dimensional feature set for each segment via OpenSMILE (Eyben et al., 2010). The pre-trained ranking functions then estimate an ED vector for each segment, where each element represents the intensity of a specific emotion. To ensure hierarchical consistency, we duplicate the utterance-level ED across all phonemes and replicate the word-level ED for the corresponding phonemes, as shown in Figure 2(a). These hierarchical ED vectors are subsequently incorporated into TTS training, which will be introduced in the next subsection.

We propose a multi-step strategy for modeling hierarchical emotion distribution (ED) to enable precise, multi-level control over emotion rendering in text-to-speech (TTS) synthesis. By progressively predicting ED at the utterance, word, and phoneme levels, our approach ensures that global emotional context guides local prosodic details. We explore two integration strategies to incorporate this multi-step hierarchical ED prediction into TTS frameworks.

In the external integration approach (“External”), as shown in Figure 3(a), we enhance a model-agnostic TTS pipeline by integrating a hierarchical ED embedding after text processing. In this paper, we choose FastSpeech2 (Ren et al., 2022) as our TTS backbone. A text encoder converts phoneme sequences into linguistic embeddings, while a fully connected network transforms the hierarchical ED into an ED embedding. A variance adaptor then predicts pitch, duration, and energy, followed by a decoder that reconstructs the Mel-spectrogram using an L1 loss. This design effectively captures emotion intensity and improves prosody. Since the “External” framework does not inherently predict hierarchical ED, we incorporate a dedicated multi-step hierarchical ED prediction module. In this module, EDs are predicted sequentially—starting from the utterance level, progressing to the word level, and finally to the phoneme level—while keeping the text encoder frozen as shown in Figure 3(b). This multi-step process allows a higher-level emotional context to guide finer, local prosodic adjustments.

The variation adaptor integration approach (“VA”) integrates hierarchical ED modeling directly within the variance adaptor of FastSpeech2 (Ren et al., 2022). This configuration extends the variance adaptor to jointly learn emotion representations and acoustic features, thereby tightly coupling emotion prediction with prosody generation. In contrast to the “External” setting, we train the linguistic encoder with a loss function that minimizes hierarchical ED differences. Specifically, we use a mean squared error (MSE) loss to reduce the discrepancy between the predicted and ground-truth EDs. Within the VA integration, ED prediction is also performed in a multi-step manner. The process begins with predicting the utterance-level ED to establish the global emotional tone, which then informs the word-level prediction. Finally, these outputs are combined to generate phoneme-level ED, enabling fine-grained emotion control, as shown in Figure 4(b). This hierarchical, multi-step approach ensures that a broader emotional context effectively influences lower-level prosodic details.

During run-time, our framework supports two primary tasks:

1. Emotion control.

2. Emotion editing.

For emotion control, where only text input is available, our model predicts a hierarchical emotion distribution that aligns with the textual content. This allows users to control the emotion intensity of individual speech segments, enabling fine-grained and quantifiable emotion rendering in synthesized speech. For the speech editing task, given an audio input and its corresponding transcript, our model extracts the hierarchical emotion distribution from the audio signal, enabling users control over emotion intensity for quantifiable emotion modification. In general, as depicted in Figure 3(c) and Figure 4(e), users are able to control the emotion rendering by adjusting the emotion distributions at three distinct levels, regardless of whether the ED is derived from audio or predicted from text.

We evaluated our system performance by conducting two experiments:

1. emotion prediction; and

2. emotion editing.

For emotion prediction, we train the TTS model on LibriTTS-R (Koizumi et al., 2023), a multi-speaker dataset containing approximately 580 hours of recordings from 2,306 speakers. Specifically, we utilized the “train-clean-100” and “train-clean-360” subsets for model training.

For the speech editing experiment, we used the Emotion Speech Dataset (ESD) (Zhou et al., 2022; Zhou et al., 2021), which comprises over 29 hours of emotional speech in five categories—Neutral, Angry, Happy, Sad, and Surprise—from 20 speakers (10 native English and 10 Mandarin). We exclusively used the English recordings and the full training split for TTS model training. We also use the ESD dataset to train the hierarchical ED extractor’s ranking functions, where we randomly selected 100 samples per speaker and emotion, resulting in a total of 5,000 audio samples for hierarchical ED extractor training.

We choose FastSpeech2 (Ren et al., 2022) as our backbone, which comprises a text encoder, variance adaptor, and decoder. We use a transformer (Vaswani et al., 2017)-based encoder to convert input phoneme sequences into linguistic embeddings. Variance adaptors predict hierarchical ED, duration, pitch, and energy. We utilize a transformer-based decoder to synthesize mel-spectrograms from these features. Our loss function combines the L1 loss on mel-spectrograms with the mean squared error for prosodic predictions in the variance adaptor. To address multi-speaker scenarios, we integrate speaker embeddings from Resemblyzer (Wan et al., 2020) into the encoder output. We adopt the Adam optimizer (Kingma and Ba, 2017). For TTS training, we use a batch size of 32 and perform 200,000 iterations over 48 hours on a single GPU. For text-based hierarchical ED prediction in “External”, we conduct 100,000 iterations. The ED embedding layers consist of fully connected layers with a Tanh activation function. Finally, we employ HiFiGAN (Kong et al., 2020) as the vocoder, trained on the ESD and LibriTTS-R datasets.

For emotion prediction experiments, we compared the model with our previous works (Inoue et al., 2024a; Inoue et al., 2024b) (“single-step”), which predict EDs from text in a single-step manner. For example, in the “VA” setting, our proposed model progressively predicted utterance-, word-, and phoneme-level EDs (Figure 4(b)), whereas the baseline (Inoue et al., 2024b) predicted all segments concurrently (Figure 4(d)). For emotion editing experiments, we re-implemented MsEmoTTS (Lei et al., 2022a) into the FastSpeech2 framework as the baseline (“MsEmoTTS”) to ensure a fair comparison.

For objective evaluation, we calculated the Word Error Rate (WER) using Whisper ^[2] (Radford et al., 2022) to assess the system’s robustness. To measure emotion similarity with the target, we evaluated spectral similarity using Mel-Cepstral Distortion (MCD) (Kubichek, 1993), prosody alignment through pitch and energy distortion, and duration deviation using Frame Disturbance (FD) (Sisman et al., 2017).

For subjective evaluation, we conducted three listening experiments with 20 participants, each evaluating 210 synthesized samples. First, we conducted MUSHRA tests, where participants rated each speech sample on a scale from 0 to 100, with higher scores indicating better quality or greater similarity. The first test assessed speech naturalness, instructing participants to disregard noise and emotion. The second test evaluated emotion similarity, asking participants to rate the synthesized audio solely based on its emotional expressiveness while ignoring speech quality. Additionally, we conducted best-worst scaling (BWS) tests (Kiritchenko, 2017) to compare word-level emotion controllability between our model and the baseline. In BWStests, we adjusted the emotion intensity of three words per utterance to 0.0, 0.5, and 1.0, and evaluators selected the least and most expressive samples.

In this section, we present the experimental results for two tasks: (1) Emotion Prediction and (2) Emotion Editing. For Emotion Prediction, we evaluated our models based on speech quality and emotional expressiveness, supplemented by qualitative analysis. For Emotion Editing, we assessed the controllability of emotions, measuring how effectively the model modifies and adjusts emotion intensity for different speech segmental levels.

We compared synthesized audio samples across seven different conditions, as detailed in Table 1 and Table 2. These tables organize these conditions into three key categories: “GT or Pred”, “TTS Model”, and “Pred Mode”:

• GT or Pred: This column specifies whether the hierarchical emotion distributions (EDs) are obtained from ground-truth data (GT) or predicted from text.

• TTS Model: This column indicates the TTS model used. “VA” refers to the model utilizing the Single-Step hierarchical ED Variance Adaptor (Figure 4(d)), while “VA (Multi-Step)” corresponds to the model employing the Multi-Step hierarchical ED Variance Adaptor (Figure 4(b)).

• Pred Mode: This column describes how the hierarchical ED is predicted. It is either generated progressively from longer to shorter segments (“Multi-Step”) or in parallel for all segments at once (“Single-Step”).

This configuration allows for a structured comparison of how different hierarchical ED configurations and prediction strategies impact synthesis quality and emotion expressiveness.

Table 1 summarizes the results of the MUSHRA and WER tests. With ground-truth hierarchical ED, the variance adapter with multi-step emotion modeling (“VA (Multi-Step)”) consistently outperforms the single-step approach (“VA”), achieving higher MUSHRA scores and lower WER. We also observe that, when using predicted hierarchical ED, the multi-step models significantly improve both speech naturalness and intelligibility compared to the single-step models. These findings suggest that aligning the EDs of shorter segments with those of longer segments is crucial for enhancing overall speech quality.

We further assess emotion expressiveness by conducting MUSHRA tests on emotional similarity and computing multiple objective prosody-related metrics, including Mel-Cepstral Distortion (MCD), Pitch/Energy Distortion, and Frame Disturbance (FD). As shown in Table 2, our proposed multi-step hierarchical ED prediction consistently achieves higher MUSHRA emotional similarity scores and lower distortion values across all objective metrics for both ground-truth and predicted hierarchical EDs. These results highlight the effectiveness of the multi-step scheme in enhancing emotion expressiveness and improving alignment with ground-truth emotions. However, we observe that in the “VA” setting, the multi-step scheme does not outperform the single-step approach in Pitch and FD. This discrepancy may stem from error accumulation across different levels of ED prediction (utterance, word, and phoneme levels). Compared to the “External” setting, where the linguistic encoder remains independent to emotion prediction, the “VA” setting incorporates hierarchical emotion distribution difference loss in training. This joint training may increase the model’s sensitivity to ED variations in longer segments, potentially leading to greater fluctuations in prosody-related metrics.

We further analyzed the predicted hierarchical emotion distribution (ED). Table 3 presents the mean absolute difference between the predicted and ground-truth values for each segment. The column “Longer Segments” denotes the longer segments used to predict shorter segments; “GT” indicates that ground-truth values were employed. For example, under the “GT” condition, we used the ground-truth utterance-level ED to predict the word-level ED, whereas under the “Predicted” condition, we utilized the predicted utterance-level ED.

Table 3 summarizes our results. We did not observe significant differences between single-step and multi-step predictions, despite substantial differences in synthesized audio performances (see Tables 1 and 2). These results suggest that our prediction modules not only reduce hierarchical ED differences but also generate emotion representations consistent with the textual emotional content in the audio domain. More importantly, when comparing “Predicted” and “GT” in the multi-step mode, we found error accumulation in ED prediction, evidenced by a smaller gap at the word level and an increased gap at the phoneme level. These findings suggest that our model prioritizes the dependency of EDs across segments over mere ED differences, which explains the improved speech naturalness (Table 1) and only marginally better emotion expressiveness (Table 2).

Next, we analyzed the predicted word- and phoneme-level EDs derived from different utterance-level EDs. Specifically, we systematically varied the intensity of a single emotion, setting it to 1.0 while keeping the intensities of all other emotions at 0.0. We then visualized the resulting ED distributions as histograms for each segment in Figure 5, where each row represents the intensified emotion and each column corresponds to a segment. We utilized a TTS model trained on the ESD dataset to highlight the impact of emotional variations in speech synthesis. From Figure 5, we observe that both word- and phoneme-level EDs exhibit a positive correlation with their respective utterance-level ED values, with the correlation being stronger at the word level. This suggests that emotion propagation is more consistent across words than phonemes. Additionally, we find that word-level anger and surprise intensities display a notably stronger inter-correlation compared to other emotions, indicating that these two emotions may share similar prosodic and acoustic patterns at the word level. This observation aligns with psychological studies suggesting that anger and surprise often exhibit overlapping acoustic characteristics, such as increased pitch and energy.

Following (Inoue et al., 2024a), we evaluated our models’ emotion controllability on the ESD dataset through subjective evaluations and objective analysis. We first conducted best-worst scaling (BWS) tests (Kiritchenko, 2017) to compare word-level emotion controllability between our model and the baseline MsEmoTTS (Lei et al., 2022a). As presented in Table 4, our model exhibited a stronger tendency than MsEmoTTS to select the least expressive sample at low intensity and the most expressive sample at high intensity across all five emotions. This trend was especially pronounced for Sad and Surprise emotions, where the distinction between intensity levels was more evident. These results demonstrate our model’s ability to capture fine-grained variations in emotional intensity, ensuring more precise and consistent emotion rendering control compared to the baseline MsEmoTTS.

We further validated controllability across utterance, word, phoneme, and word-and-phoneme levels. We incremented emotion intensity from 0.0 to 1.0 and computed prosodic features such as duration and the mean/standard deviation of pitch and energy (Figure 6). Because duration values vary between levels, we standardized them prior to visualization. Prior literature (Schuller, 2018) correlates these features with emotion intensity; for example, sadness is associated with a slower speaking rate and lower pitch and energy values. We analyzed the ESD dataset [50] to examine these acoustic-emotion relationships. A red background indicates a negative trend, while blue is a positive trend with increasing intensity. Our model followed these expected trends, showing a positive correlation between happiness and mean pitch and a negative correlation between sadness and pitch. Additionally, editing both word and phoneme-level emotions produced significant prosodic changes, with the standard deviation of pitch at the utterance level aligning with our expectations.

Figure 7 shows the spectrograms of synthesized audio samples with varying emotion intensities. We display pitch and energy contours (blue and green lines, respectively), noting that the y-axis for the energy contours is not relevant. Figures 7(a) and (b), present utterance‐ and word‐level intensity control. Each row corresponds to a different emotion, with the first column depicting acoustic features at an emotion intensity of 0.0, and the second column at 1.0. In (b), the three highlighted areas indicate regions where the intensity has been modified. For both segments, for anger, we observe more pronounced energy spikes, at higher intensities. In happiness, pitch and energy patterns are similar, with higher pitch values at an intensity of 1.0. Sadness prolongs duration and stabilizes pitch contour as intensity increases. For surprise, we note a rise in both pitch contours and energy spikes. These results align with Figure 6 and demonstrate that our model’s ability to manipulate duration and pitch/energy according to emotion intensity.

We present a multi-step prediction framework for hierarchical emotion distribution (ED), enabling multi-level control of emotion rendering in speech synthesis. By modeling ED at the utterance, word, and phoneme levels progressively, our approach ensures that higher-level emotional context influences lower-level prosody, resulting in more natural and expressive speech. We integrate ED into TTS systems through two strategies: embedding it within the variance adaptor of FastSpeech2 and incorporating it as an external module for other non-autoregressive TTS models, making our method flexible and widely applicable. At runtime, users can quantitatively control emotion intensity, enhancing the interpretability and adaptability of emotional speech synthesis. Objective and subjective evaluations demonstrate that our approach significantly improves speech quality, expressiveness, and controllability. In future work, we will extend our proposed method to additional languages, varied voice qualities, and more diverse emotional datasets, further exploring cross-lingual robustness and broadening the applicability of multi-level emotion intensity prediction.