Non-autoregressive speech translation with understanding, translation, reordering and LLM-augmented correction

With the rapid globalization of communication, the demand for speech translation (ST) has grown substantially. Traditional ST systems are typically built on cascaded architectures that integrate automatic speech recognition (ASR) and machine translation (MT). While effective, these systems suffer from error propagation and high latency. End-to-end speech translation (E2E-ST) has emerged as a more efficient alternative, removing intermediate steps to reduce latency and improve overall performance (Inaguma et al., 2020; Wang et al., 2020b; Weiss et al., 2017a; Zhao et al., 2021). Current E2E-ST models generally follow either an autoregressive (AR) or a non-autoregressive (NAR) decoding paradigm. AR models achieve strong translation quality by generating tokens sequentially, but at the cost of high inference latency (Fang et al., 2022; Gao et al., 2024; Ye et al., 2022, 2021; Zhou et al., 2023). By contrast, NAR models decode in parallel, substantially reducing latency and making them attractive for real-time applications (Inaguma et al., 2021a; Inaguma et al., 2021b; Xu et al., 2023). However, their inability to effectively handle word-order and syntactic differences across languages often leads to degraded fluency, particularly for structurally divergent language pairs.

Recent advances in NAR E2E-ST have largely centered on two decoding paradigms: connectionist temporal classification (CTC) (Graves et al., 2006) and conditional masked language modeling (CMLM) (Ghazvininejad et al., 2019). CTC-based methods directly generate output sequences from speech without requiring presegmentation or explicit length prediction (Xu et al., 2023). By introducing a blank symbol, CTC enables constant-time, fully parallel decoding. Nevertheless, the conditional independence assumption underlying CTC prevents the model from capturing contextual dependencies during inference, thereby limiting translation quality. CMLM-based methods address this issue by adopting an iterative refinement strategy, typically implemented with the mask-predict algorithm (Inaguma et al., 2021a, 2021b). These models repeatedly mask and regenerate low-confidence tokens to improve output quality. While CMLM alleviates the independence assumption of CTC, it relies on an auxiliary length predictor and multiple decoding iterations, which undermines efficiency compared to single-pass NAR methods.

Building a fully integrated NAR E2E-ST model is inherently complex, difficult to debug, and challenging to optimize due to the tight coupling between components (Omachi et al., 2023; Ran et al., 2021). Motivated by a divide-and-conquer philosophy, we propose decomposing the ST process into simpler, more interpretable subtasks. To this end, we introduce UTRo-NAST, a novel NAR-ST framework comprising an acoustic encoder, a word-by-word mapping encoder and a reordering decoder, corresponding to the subtasks of source speech understanding, word-level translation and target-side reordering (Kuan and Chen, 2025). Each component is optimized with task-specific objectives under a multitask learning scheme, enabling the system to progressively transform source speech into coherent target-language output.

With the rapid progress of large language models (LLMs), many state-of-the-art ST systems have integrated them to enhance translation fluency and robustness. Building on these advances, we extend UTRo-NAST with a plug-and-play LLM-augmented correction strategy. To leverage the strengths of LLMs while maintaining a lightweight and efficient design, we introduce three prompting schemes that further improve translation performance.

We evaluate UTRo-NAST on the MuST-C benchmark (Di Gangi et al., 2019), which spans eight language pairs. Experimental results show that UTRo-NAST consistently outperforms strong NAR-ST baselines (Inaguma et al., 2021a; Xu et al., 2023). UTRo-NAST achieves an average BLEU improvement of +1.02 across all language pairs, compared to the best-performing NAR model. Moreover, relative to AR systems, UTRo-NAST provides a 3.97$×$ speedup in inference while maintaining competitive translation quality. Finally, incorporating LLM-augmented correction, the UTRo-NAST variants deliver competitive or superior performance relative to recent LLM-enhanced ST systems, thereby validating the effectiveness and scalability of the framework for real-world speech translation applications.

Speech-to-text translation (ST) aims to directly translate spoken utterances in a source language into text in a target language. Early approaches relied on pipeline architectures, in which automatic speech recognition and machine translation systems were trained independently and cascaded sequentially (Stentiford and Steer, 1990). While modular and effective, pipeline models suffer from error propagation, as ASR errors inevitably degrade translation quality. To overcome this limitation, researchers explored end-to-end ST (E2E-ST) models that map speech directly to translated text without intermediate transcription. Pioneering works demonstrated the feasibility of end-to-end training using sequence-to-sequence architectures with attention mechanisms (Bérard et al., 2016; Weiss et al., 2017b). These models eliminate intermediate text representations and enable joint optimization, though they typically require large-scale data. Depending on the decoding paradigm, E2E-ST systems can be broadly categorized into autoregressive (AR) and non-autoregressive (NAR) architectures.

Autoregressive ST models achieve strong performance by generating each output token conditioned on previously generated tokens (Inaguma et al., 2020; Wang et al., 2020b; Zhao et al., 2021). This sequential dependency allows the model to capture rich context and perform accurate language modeling. The typical formulation of AR-based ST models relies on the conditional probability decomposition:

where X denotes the input speech features and $Z=(z1,z2,…,zU)$ represents the target translation sequence.

Recent studies have further advanced AR-based ST models through multitask learning strategies (Chen et al., 2024c; Cheng et al., 2023; Fang and Feng, 2023; Fang et al., 2022; Gao et al., 2024; Tian et al., 2025; Tsiamas et al., 2024b; Wang et al., 2024; Yan et al., 2024; Ye et al., 2022, 2021; Zhang et al., 2024; Zhou et al., 2023), which jointly optimize ST, ASR, and MT by sharing encoders and/or decoders, thereby enhancing model generalization. Xu et al. (2024) and Yan et al. (2023) further combined CTC with cross-entropy loss to improve alignment and accuracy. While these techniques enhance translation quality and robustness, the sequential nature of auto-regressive decoding still limits inference speed.

In contrast, NAR models reduce latency by generating tokens in parallel, making them attractive for real-time use cases (Gu et al., 2018). The basic NAR formulation assumes conditional independence among target tokens:

Here, each token $zu$ depends only on the input speech X, without reference to other target tokens. This assumption makes it difficult to model token regularity and syntactic differences, often degrading fluency for structurally divergent languages.

Recent work on NAR E2E-ST can be categorized into two paradigms:

1. purely CTC-based approaches; and

2. conditional masked language modeling (CMLM) with iterative refinement.

Inaguma et al. introduced the Orthros series (Inaguma et al., 2021a, 2021b), a flexible framework supporting both CTC and CMLM decoders. Chuang et al. (2021) systematically examined reordering in CTC-based models and proposed a multitask strategy that jointly optimizes source and target CTC losses to improve alignment. More recently, Xu et al. presented CTC-NAST (Xu et al., 2023), which incorporates intermediate CTC supervision, prediction-aware encoding, and curriculum mixup to further improve robustness and translation quality.

Recent advances in ST increasingly integrate LLMs into end-to-end architectures. Given their proven capacity for language understanding and generation, LLMs are being leveraged to address persistent limitations in ST, particularly in contextual reasoning and fluency. Zhang et al. proposed LST (Zhang et al., 2023a), a joint fine-tuning framework that combines speech-pretrained models with LLMs via prompt-based adaptation, yielding notable quality gains. Building on this idea, LLaST (Chen et al., 2024a) embeds LLMs directly into the E2E-ST pipeline and uses a Dual-LoRA training scheme. In parallel, SALM (Chen et al., 2024e) introduced a speech-augmented prompting paradigm, in which acoustic representations are fused with task-specific instructions as LLM prompts. This design enables in-context learning and zero-shot generalization across diverse speech–language modalities.

Beyond integrated pipelines, other methods enhance ST with LLMs via postprocessing or auxiliary training. GenTranslate (Hu et al., 2024) leverages N-best hypotheses to construct semantically consistent outputs with LLM support. Dou et al. (2025) refined translations through postdecoding prompts that jointly improve ASR and ST outputs. TDR (Zhou et al., 2025) improves context modeling through multitask learning across ASR, MT and CoT-based ST, with Jensen–Shannon divergence to enforce consistency. CoT-Prompting (Hu et al., 2025) further extends CoT reasoning by integrating speech encodings, ASR hypotheses and fixed prompts to strengthen semantic inference. Collectively, these approaches highlight the growing role of LLMs in advancing ST performance.

In summary, AR-based ST models deliver strong translation quality through sequential decoding but are constrained by high inference latency, limiting their applicability in real-time scenarios. NAR-based approaches address this limitation with parallel decoding, yet often struggle to maintain fluency and accurately model cross-lingual reordering. Recent efforts have attempted to mitigate these issues through CTC and CMLM variants, but challenges remain in balancing efficiency with translation quality. At the same time, LLMs have opened new directions for improving fluency, contextual reasoning and robustness, either through integration into end-to-end pipelines or as postprocessing modules. These developments highlight both the promise and limitations of existing approaches, motivating our proposed UTRo-NAST framework, which combines modular NAR modeling with an LLM-augmented correction process to achieve efficient and high-quality speech translation.

In this study, we propose a novel framework called UTRo-NAST, short for Understanding, Translation and Reordering for Non-Autoregressive Speech Translation. The core idea of UTRo-NAST is to decompose the end-to-end speech translation (E2E-ST) task into three explicitly defined, consecutive subtasks: source speech understanding, word-level translation and target-side reordering. To this end, UTRo-NAST is composed of three specialized components: an acoustic encoder, a word-by-word mapping encoder and a reordering decoder. By adopting a modular design with a multitask learning strategy, UTRo-NAST improves interpretability, enhances modeling flexibility, and boosts overall translation performance, while preserving the inference efficiency characteristic of NAR models. The complete architecture is illustrated in Figure 1.

Beyond the core NAR framework, we introduce a plug-and-play LLM-augmented correction strategy that refines the preliminary translations generated by UTRo-NAST. This component leverages three prompting variants: text-only prompting, ASR-enhanced prompting and ASR + chain-of-thought prompting. These progressively incorporate additional contextual grounding and reasoning – ranging from direct refinement of translations, to conditioning on ASR hypotheses, to stepwise error identification and correction – thereby improving fluency and grammaticality without compromising efficiency. Together, the NAR backbone and the LLM-augmented correction strategy form a comprehensive framework that balances speed, interpretability and translation quality.

Given a speech translation data set, each training example is represented as a triplet $(X,Y,Z)$⁠, where $X=(x1,…,xT)$ denotes the input acoustic feature sequence (e.g. log-Mel filterbanks), $Y=(y1,…,yS)$ is the corresponding source-language transcription, and $Z=(z1,…,zU)$ is the target-language translation.

To capture both local temporal patterns and global dependencies in the acoustic sequence X, we adopt a stack of Conformer layers (Gulati et al., 2020) to construct the encoder:

where $h(l)$ denotes the hidden representation at the l-th layer and $h(0)=X$⁠. The final output $h(L)$ serves as input to the downstream translation module.

To explicitly preserve the semantic content of the source speech, we use the source-language transcription Y as an auxiliary supervision signal, optimizing the encoder with a CTC loss:

Furthermore, we incorporate two auxiliary techniques, Intermediate CTC (InterCTC) (Lee and Watanabe, 2021) and Prediction-Aware Encoding (PAE) (Nozaki and Komatsu, 2021; Xu et al., 2023), to mitigate the conditional independence assumption of CTC and improve training stability at selected intermediate layers. InterCTC facilitates gradient flow and encourages transcription-aware features by applying CTC loss to intermediate layers:

In parallel, PAE injects token-level prediction feedback into intermediate features. Specifically, a softmax distribution $PCTC(h(l))$ is projected back to the hidden space via a linear transformation and added to the original representation:

where $Linear(·)$ maps the token distribution to the hidden dimension. The feedback-enhanced representation $h˜(l)$ is then passed to the next Conformer block.

After extracting acoustic features, the next step is to translate the source speech content into the target language. To simplify this process, we design this stage to operate in a monotonic, word-by-word manner.

To bridge the modality gap between acoustic and textual representations, we apply a linear projection to map the acoustic embeddings $h(L)$ into a latent space aligned with token embeddings:

The projected sequence $h^$ is then processed by a stack of sequence modeling blocks (e.g. Transformer or Conformer):

where $f(k)$ denotes the k-th sequence modeling layer, and $h¯(0)=h^$⁠.

To realize word-by-word translation, the encoder is trained to produce a rough target sequence $Z′$ that preserves the source sentence word order (see Section 4.1 and Table 1). Although $Z′$ may not conform to correct target-language syntax, it serves as an intermediate objective that encourages the model to learn a monotonic word-level translation mapping. The training objective is defined using a CTC loss:

where $h¯(K)$ denotes the final representation from the sequence modeling stack.

Building on the preliminary word-by-word translation, the final stage reorders the intermediate sequence into a fluent and grammatically correct sentence in the target language. To this end, the reordering decoder takes $h¯(K)$ as input and applies a stack of decoder layers:

where $g(j)$ denotes the jth decoder block and $h˜(0)=h¯(K)$⁠. Each decoder layer incorporates a cross-attention mechanism, using $h˜(j−1)$ as the query and $h¯(K)$ as the key and value, thereby guiding the reordering process with global semantic context.

The decoder is optimized with a CTC loss defined over the ground-truth target sequence Z:

As in the earlier modules, InterCTC and PAE are applied at selected intermediate layers to provide auxiliary semantic supervision and further enhance reordering quality.

In summary, UTRo-NAST decomposes the end-to-end speech translation task into three consecutive subtasks: source speech understanding, word-level translation and target-side reordering. Each subtask is handled by a dedicated module with its own training objective and the entire framework is optimized end-to-end with a multitask loss:

where $LInterCTC-AE(m)$ denotes the auxiliary InterCTC loss at the m-th selected layer of the acoustic encoder, with M being the total number of such layers. Similarly, $LInterCTC-Reorder(n)$ and N represent the corresponding InterCTC losses and number of selected layers in the reordering decoder.

During inference, UTRo-NAST performs fully parallel decoding. The final translation is generated by the reordering decoder via greedy decoding over the CTC outputs, followed by the removal of blank symbols and repeated tokens. This procedure enables fast and efficient non-autoregressive decoding while maintaining high translation quality.

To further enhance performance, we leverage the power of LLMs through a postcorrection process. This approach provides a lightweight and effective alternative to fully end-to-end LLM architectures: it requires no joint training, incurs minimal engineering overhead and can be flexibly enabled or disabled at inference time.

Our design is motivated by two key considerations. First, directly fine-tuning or tightly coupling LLMs with ST models is computationally expensive and typically demands large amounts of supervised data, which restricts practical deployment. Second, recent studies have shown that carefully crafted prompts can elicit strong correction capabilities from off-the-shelf LLMs in zero- or few-shot settings (Dou et al., 2025). Based on these insights, we decouple generation and correction: the ST backbone produces a preliminary translation, and an external LLM refines it via prompting. This decoupled design allows flexible substitution of different pretrained LLMs without re-engineering the ST backbone.

To harness LLMs efficiently, we design a three-tier prompting framework, as illustrated in Table 2:

1. Text-Only Prompting (TP) – the LLM receives only the preliminary translation and is instructed to refine it as a translation expert.

2. ASR-Enhanced Prompting (AP) – in addition to the preliminary translation, the ASR recognition result of the source speech is provided, anchoring corrections to the spoken content and reducing semantic drift.

3. ASR + Chain-of-Thought Prompting (ACoT) – building upon AP, the LLM is further guided through a stepwise reasoning process: first identifying potential errors, then proposing a refined translation.

These three levels of prompting (TP$→$AP$→$ ACoT) progressively increase contextual grounding and reasoning support, thereby enabling LLM-augmented correction to flexibly balance simplicity, robustness, and translation accuracy, while highlighting its scalability and adaptability within real-world speech translation pipelines.

We conduct experiments on the MuST-C v1 corpus (Di Gangi et al., 2019), a multilingual ST data set derived from TED talks. It covers eight translation directions from English (En) to German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru). All models are trained in a bilingual setting, with model selection based on the development set and final results reported on the tst-COMMON test set for each language pair.

To construct supervision signals for the word-by-word mapping encoder, we generate word-level translation targets using bilingual dictionaries obtained via Google Translate. Specifically, all unique English words in the training set are extracted and individually translated into the target language. Each English sentence is then converted into a word-level translation by replacing its words with their dictionary translations while preserving the original order. This procedure is applied uniformly across all language pairs to ensure consistency. An illustrative example is shown in Table 1.

We acknowledge that this dictionary-based construction may introduce noise, particularly for polysemous words and context-dependent lexical choices. However, these word-level targets are intended only as a rough, monotonic intermediate signal for learning cross-lingual correspondences while preserving source-side ordering. The subsequent reordering decoder incorporates sentence-level context to refine the intermediate sequence and generate fluent target sentences, which is expected to alleviate the impact of local imperfections in the word-by-word supervision. In addition, our framework is not tied to a specific dictionary source and can readily incorporate alternative lexicons or alignment-based mappings as stronger intermediate supervision, which we leave for future work.

Preprocessing follows the standard Fairseq pipeline (Wang et al., 2020b). Utterances shorter than 5 frames or longer than 3,000 frames are discarded. Acoustic features are extracted as 80-dimensional log-Mel filterbanks with a 25 ms window and 10 ms stride. SentencePiece is applied for tokenization, using a shared vocabulary of 10,000 subwords across source and target languages to support cross-lingual modeling. To improve robustness, we apply SpecAugment (Park et al., 2019), while speed perturbation is not used.

Our models are implemented in Fairseq (Wang et al., 2020b) and optimized with Adam (⁠$β1=0.9,β2=0.98$⁠) under the default learning rate schedule. Dropout is set to 0.15 during training.

To control complexity and isolate architectural effects, we adopt a consistent setup for the sequence modeling blocks $f(·)$ and $g(·)$ used in the word-by-word mapping encoder and reordering decoder. We evaluate two variants: UTRo-NAST-T, where both components use Transformer blocks, and UTRo-NAST-C, where both use Conformer blocks. This comparison highlights architectural differences without introducing the confounding effects of hybrid configurations.

The acoustic encoder, word-by-word mapping encoder and reordering decoder contain $L=12$⁠, $K=3$⁠, and $J=9$ layers, respectively. Each layer consists of 512 hidden units, 8 attention heads and a feed-forward dimension of 2,048.

To stabilize training and improve representation learning, InterCTC and PAE are applied at layers six and nine of the acoustic encoder, and at layers three and six of the word-by-word encoder. For the reordering decoder, a curriculum mix-up strategy is integrated into PAE with a fixed ratio of 0.8 (Qian et al., 2021; Wang et al., 2020a; Xu et al., 2023).

All models are trained on four NVIDIA RTX 3090 GPUs with a maximum batch size of 20,000 tokens. Gradient accumulation is performed with an update frequency of four. Training runs for up to 100 epochs per language pair, with early stopping based on development BLEU and a patience of 20 epochs. During inference, the final model is obtained by averaging the top ten checkpoints ranked by development set BLEU. Evaluation is conducted with case-sensitive SacreBLEU (Post, 2018).

We first evaluate the proposed UTRo-NAST framework under two architectural variants: UTRo-NAST-T and UTRo-NAST-C. Results are reported with and without sequence-level knowledge distillation (Seq-KD) (Kim and Rush, 2016), where the distilled translations are obtained from an autoregressive MT model with a beam size of five. The overall results are summarized in the bottom section of Table 3.

Without Seq-KD, UTRo-NAST-C consistently outperforms UTRo-NAST-T, achieving an average BLEU score of 29.2 versus 28.6. This advantage is particularly notable for syntactically flexible languages such as Romanian (25.1 vs 24.1) and Russian (18.6 vs 17.5), suggesting that Conformer-based modeling better captures long-range dependencies and structural variation. With Seq-KD applied, both variants improve further: UTRo-NAST-C + Seq-KD reaches 29.7 BLEU, while UTRo-NAST-T + Seq-KD achieves 29.6 BLEU. The reduced gap indicates that knowledge distillation mitigates architectural differences by providing smoother and more deterministic training targets.

We next compare UTRo-NAST with representative non-autoregressive ST (NAR-ST) baselines, including Orthros-CTC (Inaguma et al., 2021a, 2021b), Orthros-CMLM (Inaguma et al., 2021a, 2021b), CTC-NAST (Xu et al., 2023), Hard Multitask CTC (Yan et al., 2024) and NAST-S2T (Ma et al., 2024). The results are summarized in the lower section of Table 3. Overall, UTRo-NAST achieves stronger translation performance across most language pairs. Among the compared NAR baselines, CTC-NAST is the strongest prior method; nevertheless, UTRo-NAST-T outperforms our CTC-NAST* implementation on seven out of eight directions, with a small drop on Romanian (24.1 vs 24.7). Moreover, UTRo-NAST-C consistently improves over CTC-NAST* across all directions and increases the average BLEU from 28.2–29.2, highlighting the benefit of Conformer-based modeling for capturing acoustic cues and longer-range semantic dependencies.

We also observe that applying sequence-level knowledge distillation (Seq-KD) tends to compress performance gaps among student architectures by aligning them to the teacher’s output distribution, which explains why several Seq-KD variants fall into a narrow BLEU range (e.g. $∼$29.5–29.7) in Table 3. Under Seq-KD, the average BLEU differences become small: CTC-NAST + Seq-KD achieves 29.5, while UTRo-NAST-T + Seq-KD reaches 29.6 and UTRo-NAST-C + Seq-KD reaches 29.7. Importantly, the advantage of UTRo-NAST is more visible without Seq-KD, where architectural differences are less masked by distilled supervision (e.g. 28.2 for CTC-NAST* vs 28.6 for UTRo-NAST-T and 29.2 for UTRo-NAST-C). Consistent with this observation, the per-language-pair results suggest that UTRo-NAST yields robust gains across multiple directions, complementing the averaged BLEU comparison.

Beyond BLEU, UTRo-NAST offers practical advantages that are not directly reflected by aggregate scores. By explicitly separating monotonic word-level mapping from target-side reordering, UTRo-NAST provides an interpretable intermediate representation and enables targeted error diagnosis (e.g. distinguishing lexical mapping errors from reordering errors), which is less accessible in single-stage CTC baselines. In summary, UTRo-NAST provides consistent improvements over existing NAR-ST models without iterative refinement or complex decoding, while offering strong scalability, cross-lingual robustness and improved interpretability as a fully end-to-end ST framework.

We also compare UTRo-NAST variants with recent autoregressive ST (AR-ST) systems, shown in the upper section of Table 3. AR-ST approaches such as CTC-Aug ST (Xu et al., 2023), BiL-CTC + Rescoring (Xu et al., 2024), ZEROSWOT-MEDIUM (Tsiamas et al., 2024a) and CAST (Tian et al., 2025) generally achieve strong performance, with average BLEU scores between 29 and 31. However, performance varies considerably across AR-ST models, from 25 to 31 BLEU. Many of these systems incorporate rescoring or rely on large pretrained foundation models for enhancement. In contrast, UTRo-NAST is trained in a purely end-to-end fashion without such external augmentation. Despite this, it surpasses several AR baselines, including FCCL (Medium) (Zhang et al., 2023b), CMOT (Zhou et al., 2023) and RoPE-ST+Joint (Li et al., 2024), even without Seq-KD. With Seq-KD, UTRo-NAST achieves further gains and reaches performance comparable to stronger AR-ST baselines. Nonetheless, compared with top-performing AR models such as ZEROSWOT-MEDIUM, a gap remains, indicating that NAR-ST systems still trail AR models in ultimate accuracy.

To assess efficiency, we measure the inference speed of UTRo-NAST against both AR-ST and NAR-ST baselines. Speed is evaluated on the En–De subset using an NVIDIA RTX 3090 GPU with a batch size of 1. As shown in Table 3, UTRo-NAST achieves a speedup of over 3.97$×$ relative to AR-ST models such as CTC-Aug ST. While UTRo-NAST-C typically produces better translations than UTRo-NAST-T, it incurs slightly higher latency due to the heavier computation of Conformer blocks. This illustrates a tradeoff between accuracy and efficiency. Overall, UTRo-NAST provides a strong balance: it consistently decodes faster than AR models while achieving comparable quality across multiple language pairs, demonstrating its practicality for real-time ST.

To better isolate the contribution of the word-by-word mapping stage in our modular decomposition, we conduct a controlled ablation by setting the loss weight of the word-by-word mapping objective $LWbW$ to zero while keeping the network architecture unchanged. This setting removes the explicit word-level intermediate supervision but preserves the full forward computation of the word-by-word mapping encoder and the downstream reordering decoder, enabling a like-for-like comparison of translation quality.

Table 4 reports the BLEU scores on MuST-C tst-COMMON. For UTRo-NAST-T, disabling $LWbW$ by setting its weight to zero decreases the average BLEU from 28.6–28.4, with consistent drops on most language pairs (e.g. De: 26.7$→$25.9; Es: 32.0$→$31.2; Fr: 37.8$→$37.5; It: 27.5$→$27.3; Nl: 31.2$→$30.8). For UTRo-NAST-C, the degradation is more pronounced, where the average BLEU drops from 29.2–28.3, and decreases are observed across all directions (e.g. Fr: 38.4$→$37.1; Nl: 32.2$→$31.1; Pt: 32.7$→$31.2; Ro: 25.1$→$24.4). These results demonstrate that the explicit word-level intermediate supervision provided by $LWbW$ contributes meaningfully to the final translation quality, and that the benefit is particularly salient when the backbone has stronger sequence modeling capacity (i.e. the Conformer-based variant).

Overall, this ablation substantiates the importance of the proposed intermediate word-by-word mapping objective: even though the final translation is produced by the reordering decoder, enforcing monotonic word-level correspondence during training provides a useful inductive bias and improves end-to-end speech translation quality.

Finally, we evaluate the proposed inference-time correction framework that integrates LLMs, and compare it systematically with existing LLM-based ST approaches. In this study, we adopt the ChatGPT-4o API to implement the strategy, though the choice of LLM remains open depending on resources and deployment considerations. As shown in Table 5, we report results for UTRo-NAST-T and UTRo-NAST-C under different prompting strategies.

Several interesting observations can be made. First, both UTRo-NAST-T and UTRo-NAST-C degrade in performance when combined with the text-only prompt (TP). In contrast, ASR-Enhanced Prompting (AP) and ASR + Chain-of-Thought Prompting (ACoT) yield substantial improvements over the uncorrected baselines, confirming that LLMs can effectively leverage contextual information to enhance translation quality. These results signal the importance of ASR guidance: without it, corrections may drift from the original content, thereby reducing translation fidelity.

Next, somewhat surprisingly, AP achieves better performance than ACoT. A plausible explanation is that while CoT encourages more detailed reasoning, it may also introduce unnecessary complexity or overcorrection, leading to minor semantic deviations. In contrast, AP provides a more direct and efficient alignment between source ASR output and translation, which appears more reliable in this setting.

Finally, we compare LLM-augmented correction with recent state-of-the-art LLM-based ST systems, including GenTranslate, Refine _Both, and TDR. Even without fine-tuning the LLM, our modular framework with carefully designed prompts achieves results comparable to or exceeding these systems. For example, on German (De), UTRo-NAST-T with AP reaches 31.01 BLEU, outperforming Refine _Both -GPT3.5 (30.52) and TDR (29.8). On Italian (It), it achieves 30.84 BLEU, surpassing GenTranslate (29.4). These findings highlight the strong cross-lingual consistency and generalization ability of the proposed framework.

As shown in Table 5, incorporating the LLM correction step (AP or ACoT) yields clear improvements over the uncorrected baseline in our main setting (ChatGPT-4o). However, these gains come with additional inference latency and monetary cost, especially when the correction module is deployed via a commercial API.

To provide rough estimates for real-world usage, we measure the latency of the LLM-augmented correction on the entire En–De tst-COMMON set (average sentence length $≈$ 17 words). Using the ChatGPT-4o API in our setting, the average per-sentence latency is approximately 1.60 s for AP and 1.70 s for ACoT. These numbers reflect an API-based deployment and may vary with network conditions and service load; nevertheless, they provide a concrete reference for the incremental overhead introduced by enabling postcorrection.

Regarding cost, it scales approximately linearly with the number of input/output tokens under token-based pricing. Therefore, in practical deployments, one should weigh the translation-quality gains against the additional latency and API expenses, particularly under real-time constraints.

We present two qualitative case studies (En$→$De) to illustrate both the benefits and potential failure modes of the proposed LLM-augmented correction strategy. As shown in Table 6, text-only prompting (TP) may occasionally introduce semantic drift by altering core attributes such as possessives or gendered noun choices (e.g. “seines Lebens” and “Hausbesitzer” vs “Hausbesitzerin”), leading to a mismatch with the reference. In contrast, AP/ACoT incorporate ASR grounding (i.e. the transcript context) and structured refinement, which helps preserve the intended meaning and recovers a translation closer to the reference in this example.

Table 7 further highlights that the correction behavior is not always monotonic. While TP improves the baseline output by reducing disfluency and repetition, AP/ACoT may over-edit the hypothesis by introducing redundant phrasing (e.g. “und das sind diejenigen”) or heavier rephrasing such as expanded relative-clause rewrites (e.g. “aus denen ich das meiste Feedback erhalten habe”). These cases indicate that, while LLM-based postcorrection generally improves translation quality, it is not flawless. In particular, the correction model may occasionally hallucinate or over-generate content (i.e. unnecessary expansions or rephrasings) that deviates from the reference. Addressing such non-monotonic behaviors remains an important direction for future work toward more robust and reliable LLM-based refinement.

To assess whether the observed behaviors of different prompting variants are robust across LLM backends (rather than being specific to a single model), we evaluate the proposed LLM-augmented correction using two LLMs: ChatGPT-4o and a smaller open-weight model, Mistral (Bérard et al., 2016), which has 12B parameters. The results are summarized in Table 8. Across both backends and for both UTRo-NAST-T and UTRo-NAST-C, we observe a consistent ordering among prompting variants: AP achieves higher BLEU than ACoT, and both outperform TP. These results suggest that the AP-over-ACoT trend is robust across LLM choices and is not limited to a particular proprietary model.

Meanwhile, the absolute impact of postcorrection is clearly LLM-dependent. While ChatGPT-4o provides substantial gains with AP/ACoT, Mistral-12B does not surpass the uncorrected baseline in our current setting and may even degrade BLEU. A plausible explanation is that, as a smaller model, Mistral-12B may have weaker instruction-following or refinement capability for this postediting style task, making it more prone to suboptimal edits. These results highlight an important practical consideration: the proposed prompting framework appears transferable across LLMs in terms of relative behaviors, but the magnitude (and even the sign) of the quality change depends on the underlying LLM capability. Improving the robustness of LLM-based refinement for smaller or open-weight LLMs remains an important direction for future work.

In this work, we proposed UTRo-NAST, a non-autoregressive speech translation framework that decomposes the complex ST process into three modular subtasks: source speech understanding, word-level translation, and target-side reordering. By adopting a divide-and-conquer philosophy with multitask optimization, UTRo-NAST achieves competitive translation quality while maintaining high inference efficiency and architectural flexibility. Experiments on eight language pairs show that UTRo-NAST consistently outperforms existing NAR-ST baselines and achieves performance comparable to strong autoregressive models.

Beyond the core framework, we further introduced a plug-and-play LLM-augmented correction strategy that refines preliminary translations through prompting-based correction. This lightweight extension significantly improves fluency and robustness, achieving results competitive with state-of-the-art LLM-based ST systems, while avoiding iterative refinement or costly LLM fine-tuning.

Overall, these findings show the effectiveness and scalability of UTRo-NAST for real-world speech translation. For future work, we plan to explore its application in low-resource and real-time scenarios, investigate tighter integration of large language models to further boost translation quality, and extend the framework to multimodal translation tasks.

This work was supported by the National Science and Technology Council (NSTC), Taiwan, under Grants NSTC 112-2628-E-011-008-MY3 and NSTC 114-2640-B-002-005. Additional support was provided by the Empower Vocational Education Research Center at the National Taiwan University of Science and Technology (NTUST) through the Featured Areas Research Center Program, as part of the Higher Education Sprout Project funded by the Ministry of Education (MOE), Taiwan. The authors also acknowledge the National Center for High-Performance Computing, National Applied Research Laboratories (NARLabs), Taiwan, for providing essential computational and storage resources.