Semi-supervised text-audio contrastive learning method using pseudo-text input

There is a growing demand for models that reliably associate audio with natural-language descriptions, enabling applications such as language-based audio retrieval (Munakata et al., 2024; Xie et al., 2022), automated audio captioning (Mei et al., 2022), text-conditioned sound generation (Liu et al., 2023) and large audio-language models (Ghosh et al., 2024). Building on the success of vision-language contrastive learning (Radford et al., 2021), audio counterparts such as CLAP (Elizalde et al., 2023a), WAV2CLIP (Wu et al., 2022), AudioCLIP (Guzhov et al., 2022) and MuLan (Huang et al., 2022) have been developed. These methods embed audio and text in a shared latent space and have become a foundational component of modern multimodal systems.

Despite this progress, training audio-text contrastive models remains constrained by data scarcity; such training relies on large volumes of paired audio-text data, which are expensive and labor-intensive to curate. In contrast to vision-language data sets like LAION-5B with over a billion image-text pairs (Schuhmann et al., 2022), commonly used audio-text data sets offer only tens of thousands of pairs, e.g. about 5,000 in Clotho (Drossos et al., 2020) and 46,000 in AudioCaps (Kim et al., 2019), resulting in a significant disparity in data availability.

Two mainstream strategies attempt to mitigate this scarcity. Augmentation-based methods mix audio clips and compose their captions (Cho et al., 2023; Kim et al., 2023a; Kim et al., 2023b), sometimes inserting temporal connectors such as “followed by” to encode ordering (Wu et al., 2023a; Xie et al., 2023; Yuan et al., 2024). Generation-based methods use large language models (LLMs) to synthesize captions from tags or metadata (Mei et al., 2023; Primus et al., 2023a; Sun et al., 2023). Although these methods increase data diversity, the former still requires captions for all source clips, and the latter depends on textual side information and can introduce bias or distributional drift in LLMs outputs (Shumailov et al., 2023; Shumailov et al., 2024; Wyllie et al., 2024), often resulting in limited improvements in downstream performance (Munakata et al., 2024; Primus and Widmer, 2024).

A parallel line of research leverages unlabeled audio for audio-text applications, including weakly or unsupervised captioning (Deshmukh et al., 2024; Kouzelis and Katsouros, 2023; Zhang et al., 2025), text-queried source separation (Saijo et al., 2025) and audio-language modeling (Wang et al., 2025). These approaches typically pass unlabeled audio through a frozen CLAP encoder and fine-tune task-specific heads, improving task performance while keeping the shared audio-text representation fixed. How to use unlabeled audio to directly improve the contrastive encoder itself remains an open problem.

To tackle this problem, we propose a semi-supervised framework that trains an audio-text contrastive model using large collections of unlabeled audio together with available labeled clips. The key idea is to map embeddings of unlabeled audio into the tokenized text embedding space through an audio-to-text (a2t) mapper, producing pseudo-text inputs that act as textual counterparts for unlabeled audio within the contrastive objective. Our approach is inspired by Pic2Word (Saito et al., 2023) from the vision domain, which maps query images into the text space for zero-shot composed-image retrieval. Unlike Pic2Word, which keeps the CLIP backbone frozen, our framework jointly optimizes both the audio and text encoders together with the mapper, directly updating the shared audio-text representation rather than keeping the backbone fixed. We explore two mapper designs:

1. a simple multi-layer perceptron (MLP)-based a2t whose effectiveness we verified in our earlier study (Komatsu et al., 2025b); and

2. a sequence-flexible Transformer-based a2t that can attend to variable-length audio embeddings.

To further enhance robustness, Gaussian noise is injected into the audio embeddings before they enter the a2t mapper. This perturbation smooths the pseudo-text representations and improves retrieval accuracy, as shown in ablation studies.

Our main contributions are summarized as follows:

• We introduce the first (to our knowledge) semi-supervised framework that updates the CLAP backbone using unlabeled audio, in contrast to prior work that either relies on fully paired captions during pre-training or freezes the encoder and tunes only task heads.

• We propose an audio-to-text mapping that projects audio features into the tokenized text embedding space, enabling the use of unlabeled audio without generating synthetic captions.

• We demonstrate that incorporating unlabeled audio through our framework leads to consistent improvements in retrieval performance without requiring any additional captions.

Early audio-text models, including CLAP (Elizalde et al., 2023a), WAV2CLIP (Wu et al., 2022), AudioCLIP (Guzhov et al., 2022) and MuLan (Huang et al., 2022), adapt CLIP to the audio domain and rely on large amounts of paired data. More recent studies explore improved alignment mechanisms and more compact architectures (Komatsu et al., 2025a; Wang et al., 2024; Xin et al., 2024). Hierarchical Alignment Transformers refine multilevel correspondences between audio and text features (Xin et al., 2024). Multiscale contrastive text-audio pre-training (MC-TAP) introduces multigranularity objectives to reduce label reliance (Wang et al., 2024). TinyCLAP distills CLAP into a compact model with competitive retrieval accuracy (Paissan and Farella, 2024). Temporal-enhanced CLAP (T-CLAP) injects temporal cues via LLM-generated captions and mixed-audio augmentation (Yuan et al., 2024). However, pre-training in these pipelines remains fully supervised. In contrast, our approach performs semi-supervised contrastive training that leverages unlabeled audio to update the backbone itself.

Audio mixing has been widely used as a general-purpose augmentation principle [e.g. Mixup (Zhang et al., 2018)] and as a complementary time-frequency perturbation strategy [e.g. SpecAugment (Park et al., 2019)], motivating mixed-sample training in audio representation learning. In audio-text contrastive learning, however, modifying the audio also requires adapting the corresponding text description. Methods such as AL-MixGen (Kim et al., 2023b) and PairMix (Kim et al., 2023a) augment training data by mixing two audio clips and concatenating their captions “sound A and sound B.” Subsequent work replaced the conjunction with temporal connectors such as “followed by” to encode ordering information (Wu et al., 2023a; Yuan et al., 2024). Although these methods effectively increase the diversity of training pairs, they still require a caption for every source clip. Our approach avoids this dependency by pairing unlabeled audio with pseudo-text embeddings in the text space rather than with human or synthetic captions.

LLM-based data sets such as WavCaps (Mei et al., 2023), ClothoV2-GPT (Primus et al., 2023b) and Auto-ACD (Sun et al., 2023) generate captions from tags or metadata. While this approach is cost-effective, it relies on textual side information and can introduce bias or distributional drift (Shumailov et al., 2023; Shumailov et al., 2024; Wyllie et al., 2024), leading to limited improvements in downstream tasks (Munakata et al., 2024; Primus and Widmer, 2024). Beyond LLM-based caption synthesis, a related direction in multimodal learning constructs supervision from noisy or automatically induced pairings, as exemplified by ALIGN (Jia et al., 2021) and BLIP (Li et al., 2022) in vision-language pre-training. These approaches highlight the practicality of leveraging imperfect textual surrogates at scale. In contrast, we take a different route: rather than generating text, we align mapped audio vectors directly with tokenized text embeddings.

In parallel, large-scale self-supervised audio representation learning has produced strong acoustic encoders [e.g. wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021)] without requiring paired text. However, these approaches primarily strengthen audio-only representations and do not directly address cross-modal alignment in a shared audio-text space.

Recent studies have explored learning from unlabeled audio for various audio-text tasks. Most of these studies keep a pretrained CLAP encoder frozen and update only task-specific heads, e.g. in weakly or unsupervised captioning (Deshmukh et al., 2024; Kouzelis and Katsouros, 2023; Zhang et al., 2025), text-queried source separation (Saijo et al., 2025) and large audio-language models (Wang et al., 2025). DRCap further limits supervision by using unlabeled audio clips solely at inference time (Li et al., 2025). However, because these approaches keep a pretrained CLAP encoder frozen, the shared audio-text representations remain unchanged. Our work differs fundamentally: we perform semi-supervised contrastive training that updates both the audio and text encoders through pseudo-text mapping, directly enhancing the backbone representation itself.

This section reviews the conventional contrastive framework that is widely used by many state-of-the-art audio-language models; the next section extends it to the semi-supervised setting.

Figure 1 shows the overview of audio-text contrastive learning. We assume mini-batches of audio-text pairs $(xa,xt)$⁠, where $xa$ denotes an audio waveform and $xt$ its caption. The waveform is first converted to a time-frequency feature $Xa∈RT×Da$⁠, and the caption is tokenized into $Xt∈RL×Dt$⁠, where T and L denote the sequence lengths and $Da$⁠, $Dt$ the feature dimensions. Modality-specific encoders $fa$ (audio encoder) and $ft$ (text encoder) map them to a shared latent space:

with $za,zt∈Rd$⁠, where d is the embedding dimension.

Let $sim(u,v)=u⊤v||u||2||v||2$ be the cosine similarity and $τ>0$ the temperature. For the audio $→$ text direction, the InfoNCE loss is:

where the denominator sums over all captions $zt(j)$ in the mini-batch. The text $→$ audio counterpart is:

The overall loss is the sum of the two directions:

Minimizing equation (5) pulls matched pairs together and pushes unmatched pairs apart, yielding modality-invariant embeddings that support cross-modal retrieval, zero-shot classification and other downstream tasks.

We propose a semi-supervised contrastive method that leverages unlabeled audio by synthesizing a pseudo-text representation. The key idea is to mix a labeled clip with an unlabeled one. Because unlabeled audio lacks a textual counterpart, we generate a pseudo-text input through an audio-to-text mapper (a2t). This strategy turns every mixture into a valid audio-text pair, enabling InfoNCE training without additional annotations. Note that our framework does not generate natural-language captions; “pseudo-text” denotes latent vectors in the tokenized text embedding space.

Figure 2 depicts the proposed semi-supervised contrastive pipeline. We assume mini-batches of paired samples $(xa,xt)$ and additional unlabeled waveforms $xau$⁠. First, we form the mixture:

Mixing is performed in the waveform domain before feature extraction, with a fixed coefficient of 0.5 as shown in equation (6). Because the CLAP audio encoder accepts fixed-length inputs, no additional duration matching is required. After converting to a time-frequency representation (e.g. a log-mel spectrogram), its embedding is obtained as:

The unlabeled clip is processed by the audio encoder, and the resulting representation is passed to the a2t mapper $fa2t$⁠:

where $X^tu$ lies in the text-token space. Section 4.2 details two mapper variants (MLP-a2t and Transformer-a2t), which take different levels of audio representation as input.

We concatenate the tokenized labeled text $Xt$ with the pseudo input:

and encode it to obtain $ztmix=ft(Xtmix)$⁠. Linear waveform mixing can be interpreted as a natural superposition of concurrently occurring acoustic events, which is common in real-world recordings. On the text side, concatenating the labeled caption with the pseudo-text tokens allows the text encoder to represent a composite semantic description corresponding to the mixture. This design enables us to construct valid audio-text pairs for contrastive learning without requiring human or synthetic captions for the unlabeled component.

The mixed pair $(zamix,ztmix)$ is contrasted against other pairs in the mini-batch using the InfoNCE loss:

By augmenting labeled data with unlabeled clips in this way, the model learns from a broader distribution of audio conditions. No additional annotation is required, yet retrieval accuracy improves.

The a2t module synthesizes a pseudo-text input from an audio embedding, allowing unlabeled clips to participate in contrastive training. We investigate two variants (Figures 3 and 4): a simple MLP that emits a single vector and a Transformer-based decoder that produces a token sequence.

This variant uses the two-layer MLP mapper from our prior work (Komatsu et al., 2025b), originally inspired by Pic2Word (Saito et al., 2023), to convert each unlabeled audio embedding into a single-vector pseudo-text representation. It consists of two linear layers with ReLU activation and dropout:

where $W1∈RH×d$⁠, $W2∈RDt×H$⁠, $H=1024$ is the hidden dimension, and the dropout rate is set to 0.1. Because this mapper outputs a single vector, the resulting pseudo-text sequence has a fixed length of one.

This variant uses a Transformer-based a2t mapper that converts frame-level audio embeddings into an $Lu$-token pseudo-text sequence.

Let $Zau=[z1,…,zT]∈RT×Da$ denote the frame-level audio features output by the audio encoder before pooling. We introduce a trainable query matrix $Q∈RLu×Dt$⁠, where the sequence length $Lu$ is a hyper-parameter. $Q$ serves as a set of query vectors in the cross-attention of the Transformer-based a2t mapper and is updated end-to-end through backpropagation from the contrastive objective.

Unlike the MLP-based mapper, this learnable query sequence enables the model to generate a multi-token pseudo-text input. A stack of N Transformer decoder layers then performs cross-attention:

We use $N=2$⁠, eight attention heads, and a feed-forward network (FFN) width of $4Dt$⁠. Positional encodings are added to $Zau$ to preserve temporal structure.

Although both mappers project an audio embedding into the text-token space, they operate at different levels of granularity. The MLP-a2t compresses an entire clip into a single vector (⁠$Lu=1$⁠), which is adequate when the associated text is expected to describe a dominant foreground event (e.g. “a dog is barking”).

In contrast, Transformer-a2t produces a sequence of $Lu$ learnable query tokens. Each token serves as a slot that attends to a distinct temporal or spectral pattern in the frame-level audio embeddings through cross-attention. This design enables the mapper to disentangle multiple acoustic events within the same clip and to encode them at different positions of the pseudo-text input.

To regularize the mapper, we inject a small isotropic Gaussian perturbation into each unlabeled audio embedding before it is passed to the a2t module.

Given the audio embedding $zau∈Rd$⁠, we sample:

and feed the perturbed representation to the mapper. For MLP-a2t, noise is added to the pooled embedding $zau$⁠; for Transformer-a2t, noise is added to the frame-level features $Zau$⁠. The noise scale $σ$ is set to 1.0 in this study. The effect of the noise perturbation on the final performance is examined in the ablation study presented in Sec. 5.5.

The overall loss is the sum of three contrastive terms:

1. the labeled‐pair loss in equation (5);

2. the mixed‐pair loss in equation (10); and

3. a loss that aligns each unlabeled clip with its pseudo-text counterpart.

For each unlabeled audio input $Xau$⁠, we obtain the audio embedding $zau=fa(Xau)$⁠, which the a2t mapper converts into a pseudo-text token sequence $X^tu$⁠. To inject a minimal natural-language prior, we prepend the fixed prompt “A sound of” and feed the result to the text encoder:

where the prompt is tokenized in advance and concatenated in the same way as in (9). We then align the two views with an additional InfoNCE loss:

The final objective combines all three components:

In all experiments, we use an unweighted sum of the three losses (i.e. equal weights). Each mini-batch contains an equal number of labeled and unlabeled samples; negatives are drawn from the mini-batch instances in the standard InfoNCE manner. All three losses are evaluated within the same mini-batch, and gradients are back-propagated through every module, so that the encoders and the a2t mapper are optimized jointly. The effect of equation (15) on the final performance is examined in the ablation study presented in Sec. 5.5.

We evaluate the proposed semi-supervised audio-text contrastive learning framework on audio-to-text retrieval tasks. Our experiments aim to assess (i) the effectiveness of incorporating unlabeled audio clips into the contrastive pipeline and (ii) the impact of the a2t architecture, including its sequence length $Lu$ and the use of noise perturbation.

We adopt two widely used captioning data sets, Clotho (Drossos et al., 2020) and AudioCaps (Kim et al., 2019), as sources of labeled data, and AudioSet (Gemmeke et al., 2017) as unlabeled audio. For each experiment, we target either Clotho or AudioCaps. When AudioCaps is the target, Clotho or AudioSet are treated as unlabeled clips; when Clotho is the target, AudioCaps or AudioSet are used as unlabeled clips.

All models fine-tune MS-CLAP (2023) (Elizalde et al., 2023b), initializing its audio and text encoders with publicly available pretrained weights.

All network dimensions follow the MS-CLAP (2023) configuration. The shared embedding dimension of both audio and text encoders is $d=512$⁠, and the text tokenizer (RoBERTa-base) uses an embedding dimension of $Dt=768$⁠.

For the MLP-a2t mapper, the input layer takes the 512-dimensional CLAP embedding, the hidden layer has dimension $H=1024$⁠, and the final linear layer outputs a 768-dimensional vector matching the text tokenizer.

The Transformer-a2t variant adopts the same hidden size as CLAP (⁠$Dt=768$⁠), with 8 attention heads and a feed-forward network width of $4Dt$⁠. Only the final output dimension is aligned to the text tokenizer size (768). $Q$ in equation (12) is randomly initialized following the default initialization scheme of PyTorch linear layers. Dropout of 0.1 is applied in attention and feed-forward sublayers, and layer normalization follows the default PyTorch Transformer configuration. The frame-level audio features $Zau$ are taken from the CLAP audio encoder output before pooling. Transformer-a2t variants with query length $Lu∈{1,2,4}$ are evaluated in Section 5.4. The temperature $τ$ in the InfoNCE loss is set to 0.1.

We fine-tune for 30 epochs using Adam (Kingma and Ba, 2015) with the scheduler-free optimizer (Defazio et al., 2024). The batch size is 256. All experiments are conducted on four NVIDIA A100 GPUs and complete within one to two hours each.

Evaluation metrics are Recall@1, Recall@5, Recall@10, and mean Average Precision at 10 (mAP@10) (Manning et al., 2008). Higher scores indicate stronger alignment within the joint embedding space.

Table 1 presents the results when AudioCaps is used as the only labeled corpus, whereas Table 2 shows the symmetric setting in which Clotho serves as the target. In both scenarios, incorporating unlabeled clips enables our semi-supervised framework to consistently outperform the MS-CLAP baseline.

AudioCaps as the target. The best configuration, Transformer-a2t with $Lu=2$ and unlabeled = AudioSet, achieves a 6.6% relative improvement in Recall@1 over the fine-tuned CLAP model, confirming that AudioSet clips can be effectively used through a2t pairing. This configuration also surpasses a multi-corpus supervised baseline fine-tuned on both AudioCaps and Clotho.

Clotho as the target. For Clotho, the absolute gains are smaller but remain significant: Recall@1 improves by 4.4% relative to the fine-tuned CLAP baseline, again exceeding the multi-corpus supervised counterpart.

Comparison with larger models. Despite using less labeled data, our semi-supervised model is competitive with AL-MixGen (Kim et al., 2023b), which requires captions for every mixed clip, and even outperforms large open-domain models such as LAION-CLAP (Wu et al., 2023b) and T-CLAP (Yuan et al., 2024), both of which were pre-trained on larger data sets.

Table 3 examines how the pseudo-text length $Lu$ affects performance. Replacing the MLP with a Transformer already improves results at $Lu=1$⁠, suggesting that cross-attention captures more detailed acoustic information than simple projection. Extending the query sequence to $Lu=2$ yields the best overall scores on both data sets, whereas further increasing the length to $Lu=4$ provides no additional benefit and occasionally degrades performance, likely because longer pseudo texts become redundant and do not align well with the expected input distribution of the text encoder.

Table 4 shows an ablation study on the noise perturbation and the unlabeled loss term in equation (15). Disabling the Gaussian perturbation reduces mAP@10 by approximately one point, indicating that the perturbation acts as a useful regularizer. Removing the unlabeled loss in equation (15) has an even larger effect, eliminating most of the semi-supervised gain and reducing Recall@1 by approximately two points on both targets. These observations confirm that the unlabeled loss term is essential for binding each audio clip to its generated pseudo-text input and for preventing the mapper from producing arbitrary outputs.

Our experiments are based on fine-tuning a pretrained CLAP model; the effectiveness of the proposed method in a pretraining-from-scratch setting remains to be verified. In addition, the quality of pseudo-text mapping strongly depends on the distribution of the labeled audio-text pairs used to train the a2t mapper. When the unlabeled audio comes from a different domain, the mapper may not generalize well, potentially limiting the gains from semi-supervised learning.

We presented a semi-supervised extension to audio-text contrastive learning. The core contribution is an audio-to-text mapper (a2t) that generates a pseudo-text representation for any unlabeled audio clip, allowing the clip to be mixed with a labeled clip and incorporated into the standard InfoNCE objective. Two mapper instantiations were explored: a two-layer MLP and a Transformer decoder that outputs a short sequence of pseudo-text representations. Combined with a Gaussian perturbation and an explicit audio-pseudo-text alignment loss, the framework consistently improves cross-modal retrieval in a limited-label scenario. Experiments on Clotho and AudioCaps showed that the best configuration, Transformer-a2t with two query tokens, achieved a 6.6% relative improvement in Recall@1 on AudioCaps and a 4.4% gain on Clotho over a fine-tuned CLAP baseline, surpassing baselines trained on larger data sets, all without requiring captions for the additional audio.

This work was partly supported by a project, JPNP20006, commissioned by NEDO, Japan.