Meta Soft Prompting and Learning Open Access

A large-scaled language model has been trained by the supervised and selfsupervised schemes and has achieved the cutting-edge performance in a wide range of speech and language tasks [5, 9]. The bidirectional encoder representations from transformers (BERT) [41, 15] is known as a successful pre-trained language model (PLM) which was trained from a large corpus by using the tricks of masked language model and next sentence prediction. However, in real applications, the feature representation using PLM is strictly bounded for an unseen downstream task. The previous PLM did not cope with the issue that a specific domain of input sentences is missing and far apart from those of training articles. The generalization issue is serious, and the resulting performance is dropped drastically. Such a phenomenon, also called the domain shift [16, 42], is quite common in practice. The larger the domain shift, the more the restriction on the system performance. In addition, collecting a large amount of labeled data over a variety of domains is time-consuming and sometimes intractable. Blindly predicting the unknown domain is challenging. It is practical to conduct the supervised or even unsupervised domain adaptation [40] in few-shot setting. However, fine-tuning the large-scaled PLM to a new domain by few-shot learning is prone to be overfitting. To cope with this issue, the parameter efficient learning (PEL) was implemented by incorporating the adapters in transformer layers [22] or augmenting the inputs with the prompt templates. The PEL based on model reprogramming [43] was also developed. With the emerging of the powerful generative pre-trained transformer (GPT) [2] or particularly ChatGPT, it is crucial to utilize PLM for various downstream tasks via prompt-based learning or tuning where a large scaled PLM can be frozen. PEL is basically performed by estimating a very limited set of parameters in model construction. Model capacity can be boosted by using various PEL methods.

Conventionally, the learning objective in previous studies [42] was designed by adapting PLM to a target domain which should be provided beforehand. In addition, the performance was likely degraded by pre-training and finetuning according to different learning objectives. The fine-tuned model was also sensitive to the variations in domain knowledge [33]. To deal with the aforementioned problems, this study presents a new prompt-based learning for unsupervised domain adaptation [29] where the parameter efficient adaptation of PLM across multiple domains is developed for natural language understanding (NLU). The domain-agnostic soft prompt is proposed to carry out the generalization of adaptive prompt based on PLM over different unseen domains. The fast adaptation in multi-domain language modeling is activated in the low-resource few-shot setting. A few-shot or even zero-shot domain adaptation is performed through prompt-based learning where few unlabeled samples or even no any sample in an unseen target domain are enrolled. Particularly, a meta learning approach to the proposed domain-agnostic soft prompt is addressed. Model generalization for unseen domains is tackled accordingly.

In particular, this paper integrates two machine learning paradigms. The first paradigm is the soft prompting where a number of continuous random vectors or soft prompt tokens are learned and augmented with the input text tokens to enrich the capacity of input representation. Different from the discrete vectors of word tokens as the hard prompt through a trial-and-error search procedure [38, 38, 39], the proposed continuous vectors are automatically estimated from adaptation data as the soft prompt in accordance with an iterative gradient descent algorithm [27, 27, 37]. Prompt optimization is performed to calculate the data-driven soft prompt which looks more realistic and attractive than hard prompt since both specific domain expertise and human-engineering process can be avoided. The second paradigm is the model-agnostic meta learning (MAML) [17, 45] which is employed to estimate the general soft prompt from support data and then adapt the estimated soft prompt to various unseen target domains from query data [11, 10]. The adaptability of soft prompt for an unseen domain is enabled. The meta soft prompt is proposed by implementing the gradient-based MAML where the meta learner is estimated by using the gradients accumulated from individual task-specific learners over different downstream tasks. The individual metatraining tasks are merged in an unsupervised domain representation which is handled by soft prompt learning from the source-domain labeled data as support data as well as the target-domain unlabeled data as query data. The meta learning is run across a variety of meta-training tasks where the soft prompt is learned from a source domain and then generalized to an incoming target domain through an auxiliary task in present of few-shot unlabeled samples. A new type of meta learning is developed for multi-domain NLU based on the domain-agnostic soft prompt which is learned through a doubly-looped learning algorithm consisting of an inner loop for individual learners and an outer loop for a meta learner. Owing to the integration of individual learners and meta learner, the issue of sub-optimal performance in model construction and domain adaptation which is caused by the inconsistent objectives in pretraining stage and fine-tuning stage is tackled, respectively [33, 44]. In the experiments, a series of analyses and evaluations are conducted to illustrate the performance of the proposed prompt-based language model for multi-domain sentiment classification.

The remaining of this paper is organized as follows. First of all, the fundamentals of prompt-based learning for unsupervised domain adaptation are introduced in Section 2. The basics of meta learning are surveyed. The soft prompt is estimated for multi-domain natural language understanding (NLU). In Section 3, a new solution to learn the domain-agnostic soft prompt is proposed by utilizing a pre-trained language model which is frozen during model construction. The unsupervised domain adaptation based on meta learning for soft prompt is developed. The implementation algorithm is developed and detailed. In Section 4, a set of experiments are conducted to analyze and evaluate the proposed meta soft prompt in multi-domain NLU. The ablation studies on the proposed prompt through latent visualization and classification accuracy are evaluated. Finally, the conclusions drawn from this study are given in Section 5.

This study aims to enrich the learning representation and facilitate the domain knowledge in text representation across multiple domains for natural language understanding. The fundamentals of prompt engineering and domain adaptation are first introduced.

Prompt-based language model has been showing dramatic performance in the era of generative artificial intelligence since GPT-3 [2] or particularly ChatGPT [36] has achieved state-of-the-art results in a variety of NLU tasks. Such a text representation or embedding basically considers the pre-trained language model (PLM) as a frozen backbone model where the input sentence is augmented by merging with an adjustable prompt. Prompt-based language model has been recognized as a parameter efficient learning approach in low-resource setting by utilizing a pre-trained PLM and only estimating a small set of masked language model (MLM) head and continuous prompt tokens where a small amount of domain-specific data are enrolled for domain adaptation. MLM head was added on top of PLM and estimated by means of unsupervised scheme through prediction of the randomly masked words. Basically, the performance of PLM can be leveraged for a specific downstream task.

Figure 1 compares the fine-tuning and the prompt-based learning with MLM heads which are illustrated for sentiment classification. PLM is seen as an encoder to extract general features and MLM head serves as the switch to a downstream task. The PLM parameters and MLM head, which are frozen or adjustable, are depicted by either blue or yellow, respectively. Traditionally, fine-tuning a large set of PLM parameters and estimating the task-specific

head are inefficient. There was no masking scheme. The parameters are easily overestimated especially in case of low-resource setting where only a limited set of adaptation are available. Prompt-based learning is seen as a parameter efficient solution which is designed to improve the fine-tuning of PLM when the generative pre-trained transformer (GPT) has been publicly released. In general, the strategy of prompt-based learning allows training a model with a cloze-style input sentence which adds some textual string prompt “It was —’ to the original sentence that has some unfilled or masked slot —’ which is either ‘good’ or ‘bad’. Accordingly, the MLM head in prompt-based language model is learned to predict the masked token in the augmented sentence. In addition to this unsupervised scheme, MLM head can be further learned in a supervised manner to predict target label of an input text. Such an unsupervised or supervised scheme was also employed to strengthen the small-sized PLMs in Schick and Schütze [38, 39] and Gao et al. [18] where the whole model parameters were fine-tuned and the masked tokens were predicted.

There are two types of prompting, namely hard prompt and soft prompt, in the implementation of prompt sentence. The hard prompt is formed in natural language which consists of ‘discrete’ tokens from the vocabulary. However, selecting a suitable hard prompt for specific domain or task requires the domain-specific expertise with many trial-and-errors in a prompt engineering procedure. When the hard prompt template is selected and fixed, both PLM parameters and MLM head are fine-tuned for a downstream task in a new target domain. More recently, the soft prompt was constructed in the continuous space with adaptability. Since the real-valued soft prompt {v₁v_2—} or [v₁v₂v₃} is differentiable and can be easily updated by gradient descent algorithm, the optimized prompt with the masked token `—` can be obtained in a handcraft-free fashion. The learning objective is formulated for optimization in accordance with an end performance criterion as well as a prediction of masked token. A recent approach was proposed to carry out the soft prompt which was represented in the continuous space and was optimized in accordance with a real-valued objective function where both PLM and MLM head are frozen. Only the soft prompt tokens are estimated. Relative to the fine-tuning and hard prompt, the number of tunable parameters in soft prompt is significantly reduced and the issue of overestimation can be mitigated. Nevertheless, either fine-tuning or prompting the PLM for a specific downstream task could not sufficiently represent the variations in different downstream tasks. A rapid and unsupervised domain adaptation is required in multi-domain language modeling for natural language understanding. In what follows, this paper surveys the related works and basics for unsupervised domain adaptation and optimization-based meta learning.

In general, pre-trained language model has recently demonstrated convincing results for natural language understanding through a fine-tuning or prompting stage. However, fine-tuning, hard prompt and soft prompt using PLM only work for a specific test domain, and could not easily generalize for the diverse input sentences which may come from a variety of domains in practical circumstances. In the implementation, the additional task-specific layers are configured as the MLM head for domain adaptation to a downstream task. The learned model likely faces the diverse inputs from unseen domains. The unsupervised domain adaptation (UDA) [8, 8, 28] has become crucial to deal with multi-domain language modeling in recent years [12, 6]. Basically, the goal of UDA is to compensate the issue of domain bias or shift caused by the varying distributions in source and target documents where the class labels of enrollment data in a new target domain are unavailable. A simple and meaningful approach to UDA is to fine-tune PLM or provide a prompt to input text so that the pre-training process is continued and extended by adopting the domain-specific documents. BioBERT [26] was developed to fulfill this strategy based on fine-tuning. In addition, the domain-specific pre-training [19] and the task-specific pre-training [20] were jointly performed in a way that the domain-specific pre-training was conducted by further using the task-specific unlabeled documents to adjust the text representation to be closer to the task distribution. In Karouzos et al. [25] and Chen and Chien [3], UDA was implemented under the framework of language modeling where the losses due to the masked language model and the downstream text classification were jointly minimized to enhance the model robustness and improve the sample efficiency in an unsupervised learning and adaptation. However, in the aforementioned works, there existed an inconsistency between the learning objectives in pre-training stage and fine-tuning stage for downstream task, which considerably constrained the utilization of domain knowledge using PLM [33]. This study introduces a new approach to UDA where the soft prompt is adapted across various domains through meta learning where a consistent hybrid learning objective is implemented for both domain adaptation and prompt learning.

This study presents an approach to multi-domain adaptation for prompt-based language model and text representation where the optimization-based meta learning [23] via the model-agnostic meta learning (MAML) [17] is implemented. Basically, MAML conducts a supervised learning to train individual learners and a meta learner where rapid adaptation is realizable under few-shot setting. MAML is feasible for implementation of any kinds of model architecture with learning objective through the gradient descent method. The underlying concept of MAML aims to learn a good initialization for model parameters so that the model can perform well or adapt rapidly on the new tasks or domains through a small number of enrollment data with a few steps of gradient updating. Considering a feature representation model θ, the learning process of a generalization to new tasks can be seen as a construction of internal representation which is broadly suitable for a number of tasks which are sampled from the distribution p(𝒯) of different tasks 𝒯 = {𝒯_i}. From the perspective of dynamic systems, meta learning can be implemented by maximizing the sensitivity of a learning objective to new tasks with respect to model parameters θ. High sensitivity in task adaptation corresponds to the condition that the performance improvement due to new tasks is significant while only a few gradient updates are required for system optimization. Basically, the gradient-based method to estimate a model is performed by minimizing a loss function $ℒTi$ of a single task 𝒯_i through the gradient descent updating

where α is the learning rate. To activate the process for meta learning, the parameter of a meta learner is optimized to find the updated parameter θ’ where a number of tasks sampled from p(𝒯) are learned to achieve the minimum of total loss across different tasks

In the implementation, the meta optimization across different tasks is performed such that the meta parameter θ is updated by

The training procedure for optimization-based meta learning is shown in Figure 2 where a meta learner θ_meta is optimized by using the gradients of individual tasks in the training stage and then the rapid adaptation of θ_meta to a specific task is realizable in the test stage.

This paper presents a hybrid solution to soft prompt language model and unsupervised domain adaptation under low-resource settings as shown in Figure 3. The pre-trained language model (PLM) is utilized and frozen, and the labels of target-domain data Ɗ_T are waived and ignored. There are two implementations for unsupervised domain adaptation (UDA). In the first implementation, the zero-shot domain adaptation is performed by using the learned prompt which is augmented with the test sample from Ɗ_T. Only the test sentence in target domain is used. There is no extra adaptation data required in UDA. In the second implementation, two stages are planned in

a joint scenario of prompt estimation and low-resource domain adaptation where the frozen PLM is shared and applied. The first stage is to learn and adapt the soft prompt where the labeled data from source domain Ɗ_s and few-shot unlabeled data from target domain Ɗ_T are provided for semi-supervised domain adaptation where meta learning is implemented. Subsequently, the second stage is to utilize the adapted soft prompt, shown in red, and further adjust the adapted soft prompt by using a test sentence in the target domain Ɗ_T. To cope with the downstream tasks in multiple domains, the resulting soft prompt is estimated and adapted through the optimization-based meta learning.

In a practical situation, the input query in a natural language processing task is originated from a variety of domains. It is important to design a general solution to soft prompt over different domains, and present a rapid unsupervised adaptation to a new domain. This paper presents a new prompt-based language model for multi-domain natural language understanding. A system overview is shown in Figure 4 where the pre-trained language model (PLM) using BERT encoder and the masked language model (MLM) head are frozen and utilized. The meta soft prompt is seen as the controllable model parameter which is trained and adapted by using support data (shown by purple) and query data (shown by orange) across multiple domains.

Figure 5 shows an example of various meta-training tasks containing support and query sets from two different domains. There are two stages in implementation of meta learning in the system overview. First, the support data $Disup$ of a domain i are used to adapt the soft prompt parameter θ of a meta learner to an individual learner θ_i’ with the prompt, e.g. consisting of three continuous-valued tokens {v₁v₂v₃} (shown by yellow), via a prompt-based learning objective from support data ℒ_support (or ℒ_prompt$(Disup,θ))$⁠. In the second stage, the query data $Diqry$ of a domain i are used to calculate the gradients based on the updated prompt tokens of a learner θ_i = {v₁v₂v₃} (here, the learner index i is ignored in soft prompt token v) via a learning objective from query data ℒ_query (or ℒ_prompt$(Diqry,θ))$⁠. After accumulating the gradients from query data of individual learners, the soft prompt tokens of the meta learner θ = {v₁v₂v₃} (shown by red) are accordingly updated. In the whole process, the frozen PLM is responsible for encoding the input sentence and extracting the embedding or feature representation of soft prompt and masked token [7, 30]. Next, the extracted features are used to find the output of MLM head to calculate the likelihood over the vocabulary of language model for classification prediction using support data and query data in the natural language understanding tasks where the learning objectives ℒ_support and ℒ_query are calculated, respectively. Notably, the learning objectives are calculated from the labeled data of source domain Ɗ_S as well as the unlabeled data of target domain Ɗ_T. The masked token [MASK] is arranged for supervised and unsupervised domain adaptation. The back propagation algorithm is run to minimize ℒ_support and ℒ_query to update the parameters of meta learner θ and individual learners θ_i, as shown by red dashed lines. In such a meta training procedure, support set is used to update the soft prompt of learner θ_i = {v₁v₂v₃}. Query set is then used to calculate the gradients for updating the parameters of earner θ_i = {v₁v₂v₃} followed by updating the parameters of meta learner θ = {v₁v₂v₃}. Different from the standard meta learning [17], which learns the whole model from a large number of few-shot classification tasks to allow the model to quickly adapt to an unseen classification task, the proposed meta learning focuses on tackling the domain shift by introducing the soft prompt learning conditioned on a frozen PLM. All of the meta-training tasks are designed to simulate the scenarios which will be encountered during test stage. Therefore, the frozen PLM is feasible to provide a seed model which is generalized from multiple source domains Ɗ_s to a target domain by only tuning the parameters of soft prompt.

Soft prompt language model is implemented through the prompt-based learning which is driven by the soft prompt template and the label word [14]. The soft prompt template is composed of a set of k continuous trainable vectors or tokens which are appended to the input word string as the description to make prediction in a target task. The label word is defined as the word with the highest probability that the prompt-based language model would like to predict. For the example of binary sentiment classification, the training pairs of input string of L words x = {x₁,…,x_L} and the corresponding output label y ∈ Y = [positive, negative} are collected. Given the template function T(•), the input string x is transformed to an embedding of prompt input x_prompt as the MLM input, i.e.

where the masked token [MASK] is merged for class prediction and e(·) denotes the embedding function of BERT-based PLM ℱ. For the label words, a verbalizer ℳ: 𝒴 ↦ 𝒱* is introduced to construct the label space by a set of label words 𝒱* ∈ 𝒱 where 𝒱 is the vocabulary of ℱ. This paper considers ℱ as a function of mapping the prompt input x_prompt to a vocabulary distribution given the masked token [MASK], or equivalently finding the likelihood for predicting the output class y as

Therefore, the conditional likelihood of the predicted label word y* ∈ 𝒱* due to a masked token [MASK] by using the proposed soft prompt language model driven by a verbalizer ℳ is calculated as a softmax function by

where $Vy*$ denotes the vocabulary of label words corresponding to the class label y, and $vmask(Vy*)$ denotes the probability of label word $Vy*$ in the vocabulary distribution v_mask. The learning objective of soft prompt language model is calculated as a cross entropy error function between predicted output y^* and true output y in a form of

where Ɗ = {x, y} denotes the training data, y is the ground truth label corresponding to a given input x and θ = {v₁,…,v_k} denotes k trainable parameters for soft prompt. In this study, the soft prompt parameter θ using soft prompt language model is estimated by minimizing ℒ_prompt(Ɗ, θ). Notably, the meta soft prompt is estimated across a number of domains to construct the domain agnostic soft prompt. The prompt-based learning objectives from support data ℒ_support (or ℒ_prompt(Ɗ^qry, θ)) and query data ℒ_query (or ℒ_prompt(Ɗ^sup, θ)) are calculated in the implementation.

This study would like to enhance the generalization of soft prompt to unseen domains based on meta learning [24, 21] which acquires domain knowledge from various meta-training tasks. Different from traditional meta learning [17], which trains the model from a large number of few-shot classification tasks to allow the model to quickly adapt to an unseen few-shot classification task, the proposed meta learning is developed to tackle the domain shift issue by introducing soft prompt tuning, conditioned on a frozen PLM. The meta-training tasks are executed and integrated to simulate the scenario that an unseen test condition may encounter. Accordingly, the backbone PLM could generalize to a new unseen domain by simply tuning the parameters of soft prompt. The learning procedure of the proposed domain agnostic soft prompt is implemented through a nested-loop optimization. The inner loop of this optimization is to initialize the individual learners by using the parameters of meta learner and then update the parameters of learners in accordance with Equation (7) by using the support set$Disup$ in each individual meta-training task. In the outer loop, the meta learner is trained by optimizing the performance of individual learners which is examined by using the query set D^qry in each meta-training task. The overall learning process is continuously run until the convergence condition is met. The parameters of a learner are optimized through the gradient-based method by

using support data $Disup$⁠. The parameters of meta learner are then trained by pursuing the goodness of each individual learner θ_i’, which is learned over a lot of meta-training tasks 𝒯 = {𝒯_i} from the distribution of tasks p(𝒯) using query data $Diqry$ as

Algorithm 1 illustrates the entire training for domain-agnostic soft prompt across m domain pairs or meta-training tasks. This solution is implemented for multi-domain supervised domain adaptation by using the prompt-based learning objective ℒ_prompt where the prompt samples {x_prompt, y] with input string x and its class label y from source domains are required.

To activate the unsupervised domain adaptation (UDA) by using the targetdomain data without labels, the domain agnostic soft prompt based on supervised learning via ℒ_prompt in Figure 4 is accordingly modified to that via semi-supervised domain adaptation in Figure 6 by using source-domain data with labels and target-domain data without labels in accordance with [13]

with a hyperparameter λ which tunes two learning objectives. In a nested-loop optimization, firstly, the learners are initialized by sharing the parameters from meta learner in the inner loop. Then, the parameters of individual learners are updated by using support set in each meta-training task based on ℒ_uda. The support set $Disup$ consists of k_s labeled samples from source domains ${xSj,ySj}j=1ks$ and k_t unlabeled samples from target domains ${xTj}j=1kt$ which are used to calculate the prompt-based objective ℒ_prompt in Equation (7) and the masked language model (MLM) objective ℒ_mlm, respectively, and the query set $Diqry$ consisting of k_q unlabeled samples from target domain ${xTj}j=1kq$ are used to calculate the MLM objective ℒ_mlm. Both MLM losses are calculated in a form of cross-entropy error function. Different from the cross-entropy loss ℒ_prompt calculated for predicting the class words from labeled data Ɗ_S, the MLM loss ℒ_mlm is measured for predicting those randomly-masked word

tokens from unlabeled text strings in target domains. Next, in the outer loop, the meta learner is optimized according to the performance of those learners which is evaluated on the query set in each meta-training task by using the unlabeled data in target domain Ɗ_T. The overall learning process is repeated until reaching convergence. Meta learner $θ^$ is estimated by minimizing the prompt-based and MLM losses over individual learners θ_i and the MLM loss over meta learning across various meta-training tasks 𝒯_i sampled from p(𝒯) without class labels

where α can be implemented via the popular optimizer using AdamW. The overall meta soft prompt learning and adaptation is shown in Algorithm 2. A kind multi-task learning over various domains is implemented. A combination of soft prompt learning and masked language modeling is fulfilled for unsupervised

domain adaptation. The adapted soft prompt is able to simultaneously capture the target domain information from MLM objective and the task language knowledge from prompt objective. The domain adaptation loss and prompt-based classification loss are jointly optimized in the proposed meta soft prompt.

The proposed meta soft prompt learning was evaluated by two multi-domain text classification benchmarks which were Amazon review dataset [1] and FDU-MTL dataset [31].

Amazon review dataset contained text documents from Amazon.com consisting of four domains including books, DVD, electronics and kitchen (or precisely kitchen and housewares). Each domain contained 1000 positive reviews and 1000 negative reviews. All reviews had the ratings from 1 to 5 stars which were converted into binary labels as the positive review and negative review. Each domain also contained additional unlabeled data which were collected for evaluation of unsupervised domain adaptation and text categorization as shown in Table 1 where the sizes of labeled and unlabeled data, and the document length are provided. FDU-MTL was a relatively challenging dataset consisting of up to 16 domains, which were broadly categorized into Amazon product reviews and movie reviews, where the statistics of experimental data are shown in Table 2. In the experimental setting for domain adaptation, one of the domains was selected as the target domain, and the remaining domains were viewed as the source domains. For meta learning over various source domains, the supervised domain adaptation was performed and the test data in a target domain were evaluated. Zero-shot domain adaptation was then implemented. For the setting of unsupervised domain adaptation, the classification labels in target-domain data were missing. Meta soft prompt was learned from labeled source data and few-shot unlabeled target data, and was evaluated by the rest of test data in target domain. Few-shot domain adaptation was implemented accordingly. Classification accuracy was reported in different sets of evaluation over various domains.

In the implementation, the stochastic gradient descent algorithm based on AdamW optimizer [34] was used. The mini-batch size was 256 tokens, and the number of meta-training tasks in one batch was eight. BERT [15] was used as the pre-trained language model (PLM). In meta learning, the learning rates in inner-loop optimization and outer-loop optimization were set differently. The outer-loop learning rate was fixed as 10⁻³ for different lengths of soft prompt, and the inner-loop learning rate was set as 10⁻² and 5 × 10⁻³ for the length of soft prompt as 2 and 10, respectively. The dimension of the input to the task-specific layer was 768. The parameters of task-specific layer and soft prompt tokens were trained with the same dimension 768 with two layers and two soft tokens, respectively. In the evaluation, two-dimensional (2D) latent visualization based on t-distributed stochastic neighbor embedding (SNE) [35] was analyzed.

In the experiments, the ablation study on the effect with and without the learned meta soft prompt was first conducted by showing the comparison of two-dimensional visualization samples from original 768 dimensional samples of the masked tokens for the positive and negative reviews from four individual domains in Amazon review dataset. The results of those review samples without and with meta soft prompt are displayed for comparison in Figure 7 where blue and red samples correspond to the positive and negative emotions for test reviews in unseen domains for sentiment classification, respectively. As we can see, without considering the meta soft prompt, the frozen PLM using BERT could not properly separate different classes of Amazon review samples. However, the samples from positive and negative reviews are clearly separated by introducing the proposed meta soft prompt which could correctly guides the frozen BERT to extract the distinguishable features for two sentiment classes.

Next, the usefulness of meta soft prompt language model is further examined by comparing two-dimensional latent visualization over positive versus negative as well as source domain versus target domain as illustrated in Figure 8 where the perspectives of classes and domains are analyzed, respectively. In this comparison, the samples of the masked tokens in the product reviews of ‘Books’ by using the prompt sentences based on meta soft prompt are shown. The test samples in unseen domains are evaluated. From the perspective of classes, it is found that the learned meta soft prompt language model does extract useful feature representations for masked tokens where the latent variables look separately for two individual classes. More interestingly, 2D latent samples of the masked tokens of two classes in source domain and target domain are close together. This finding reveals that the proposed meta soft prompt does pursue the domain-matching property for multi-domain adaptation. In the

above evaluation of latent visualization, the soft prompt length was fixed as 5 and λ = 0.8 was used.

Subsequently, the effect of the length of meta soft prompt is investigated [27]. The experiments on classification accuracy are carried out by considering various prompt lengths. Figure 9 displays the classification accuracy when the length of meta soft prompt was varied from 2 to 20. The four domains in Amazon review dataset are individually analyzed in the comparison. It is found that domain ‘kitchen’ receives the highest accuracy while domain ‘DVD’ obtains the worst accuracy. The accuracy was increased when the length of soft tokens in meta learning was increased, but was saturated with the length around 10.

Furthermore, the prediction of masked tokens based on the hard prompt and the meta soft prompt is compared. Table 3 shows the examples of the templates of prompt sentences based on hard prompt and meta soft prompt where the input text and masked token [MASK] are included. A popular hard prompt is designed as “This {domain} is [MASK]” by introducing the domain name for input representation in a sentiment classification task. In this study, the soft tokens {v₁,v₂,v₃} with prompt length 3 are estimated via meta learning to simulate and act as the hand-crafted hard tokens. As a result, Table 4 compares the top five predicted words of two examples in the domains of product reviews of ‘Music’ and ‘Magazine’ in FDU-MTL dataset by using hard prompt and meta soft prompt. It is found that using hard prompt could not correctly guide the BERT backbone model to seek right classification. The prediction is prone to be irrelevant or wrong. In contrast, using the proposed meta soft prompt likely finds the right sentiment class in different applied domains.

(a) Music product review

(b) Magazine product review

For comparative study, the sentiment classification accuracy of review data by using fine-tuning (FT) method, standard soft prompt (SP) [27] and the proposed meta soft prompt (MSP) is illustrated in Table 5 where four individual domains in Amazon review dataset are evaluated. The experiments were setup by choosing one domain as the target for testing and the remaining domains as the sources for meta training. Meta training over various pairs of source domains was performed in a supervised manner. Notably, the testing is performed over an unseen target domain without any adaptation data. Zero-shot adaptation is examined. To conduct a fair evaluation, the number of parameters in different methods was controlled to be comparable. For fine-tuning, the number of trainable parameters in task-specific layer was 768 × 2. The length of 768 dimensional soft tokens in SP and MSP was therefore fixed as 2. In this set of experiments, the proposed meta soft prompt achieves the highest accuracy when compared with fine-tuning and standard soft prompt in most of domains. In addition, the experimental setting on review or document classification is extended to the case of SP and MSP where the length of soft prompt is increased and fixed as 5. For this case, the accuracy using MSP is consistently higher than that using SP for various domains. Meta soft prompt does learn different cross-domain knowledge from multiple domains which is helpful for a better domain adaptation than that using standard soft prompt where multi-domain information is simply integrated without the pairwise cross-domain learning.

Next, the proposed MSP language model is compared with the previous multi-task learning methods over individual 16 domains of FDU-MTL dataset. The adapted meta soft prompt is collaborated with the frozen PLM to conduct the evaluation on target domain. Similar to the experimental setting in Table 5, different learning methods were performed over various source domains in a supervised way, but the testing was run on an unseen target domain without additional adaptation data. The classification result is shown in Table 6. The result using the proposed meta soft prompt (MSP) is compared with strong baselines including the multi-task deep neural network (MT-DNN) [32], the adversarial multi-task learning (ASP-MTL) [31], the multinomial adversarial network with the least square loss (MAN-L2), the negative log-likelihood loss (MAN-NLL) [4], and the BERT model [15] which is fine-tuned (FT) on each domain. ASP-MTL [31] used the additional abundant unlabeled samples from the target domain. This set of experiments shows that MSP performs better than the other methods in most of domains. In average, MSP obtains the classification accuracy of 89.5% over 16 domains which is the highest in this set of comparison.

In addition, the comparison of classification accuracy is extended to a recent work called MoE-Tr [42]. MoE-Tr is a transformer-based multi-source domain adaptation method, which introduced a mixture of experts (MoE) technique where multiple PLMs were involved in adaptation. This method required additional unlabeled target domain data for domain adversarial training. MoE-Tr performed the domain adversarial learning [40] where many shots were used. Different from MoE-Tr, the proposed MSP only involves one single PLM which is frozen and used to train soft prompt via meta learning. The length of soft prompt is now extended to 10 for both SP and MSP. Table 7 shows that MSP performs better than SP. This likely happens because MSP is learned from the pairwise cross-domain information which is richer than the integrated domain information in SP without pairwise domain adaptation. Meanwhile, the performance of MSP without additional unlabeled adaptation data is not as good as that of MoE-Tr where the unlabeled adaptation data are required. But, the number of trainable parameters using MSP is only 7.68K which is extremely smaller than 264M by using MoE-Tr. MoE-Tr requires 33.6K times of trainable parameters relative to MSP. MoE-Tr suffers from high parameter size and computational cost.

To conduct a fair comparison, the set of experiments in Table 7 also include the results of SP and MSP where additional few-shot unlabeled samples are provided for unsupervised domain adaptation. The results of classification accuracy show that MSP considerably works better than SP. This is likely because MSP is learned from different pairs of domain adaptation which result in an optimized model for a better adaptation than SP where the learning to learn for cross-domain adaptation is ignored. In addition, relative to MoE-Tr which involved multiple PLMs and utilized abundant adaptation data in target domain, SP and MSP are implemented by only using 4 or 8 shots of unlabeled adaptation samples. When eight shots of adaptation data are employed for unsupervised meta soft prompt learning, the resulting classification accuracy is close to that using MoE-Tr where a large-scaled controllable model and multiple PLMs were required and a large number of adaptation samples were enrolled. On the other hand, the additional few-shot unsupervised domain adaptation (either 4 or 8 shots) is further evaluated by using the FDU-MTL dataset as shown in Table 8. In this set of experiments, the ablation study without and with unsupervised domain adaptation (UDA) is also shown. The length of soft tokens in SP and MSP is fixed as 10. There are some findings.

First, similar to Table 7 for Amazon review dataset, the classification accuracy using FDU-MTL dataset is consistently improved from SP to MSP in a setting of unsupervised domain adaptation. Using SP, UDA is not working as well as that in MSP. This is possibly because MSP learns a better foundation of soft prompt than SP. Also, increasing number of shots in MSP elevates the classification accuracy in most of target domains. In general, the improvements by increasing shot number and soft prompt length are obtained through the proposed meta soft prompting and learning.

This paper has presented a meta learning for domain-agnostic soft prompt for multi-domain text representation and language understanding. This soft prompt was augmented with the observed and the masked word tokens as the prompt sentence to enrich input representation. Based on the doubly-looped optimization, the learned meta learner was able to extract useful features from the frozen pre-trained language model where the fine-tuning process was avoided. On the other hand, an additional unsupervised domain adaptation was developed in presence of very limited trainable parameters without the need of labeled data in target domain. Such an unsupervised domain adaptation was proposed to pursue the extra benefit from learning with unlabeled data in target domain. The experiments on multi-domain representation showed that this method was able to obtain latent spaces with obvious class separation. The results have shown that the meta soft prompt could successfully boost a frozen pre-trained model to capture domain-specific information and achieve desirable classification performance by only training a few parameters. This method¹ has obtained higher accuracy than the other methods in most of domains over two text datasets. For future work, the proposed method could be integrated with other unsupervised domain adaptation in the inner optimization for meta learning. The proposed method is feasible to collaborate with not only masked language model but also sequence-to-sequence model or autoregressive model for different applications.

This work was supported in part by the National Science and Technology Council, Taiwan, under Contract NSTC 112-2634-F-A49-006.

Jen-Tzung Chien is currently the Lifetime Chair Professor at National Yang Ming Chiao Tung University, Hsinchu, Taiwan. He has authored more than 250 peer-reviewed articles in machine learning with applications on natural language processing and computer vision, and three books including Bayesian Speech and Language Processing, Cambridge University Press, 2015, Source Separation and Machine Learning, Academic Press, 2018, and Machine Learning for Speaker Recognition, Cambridge University Press, 2020. He was a Tutorial Speaker of AAAI, IJCAI, ACL, MM, KDD, ICASSP and Interspeech. He received the Best Paper Award in IEEE Workshop on Automatic Speech Recognition and Understanding in 2011, and IEEE International Workshop on Machine Learning for Signal Processing in 2023.

Ming-Yen Chen received the M.S. degree from the Artificial Intelligence Graduate Program at National Yang Ming Chiao Tung University, Hsinchu, Taiwan in 2022. His research interests include meta learning, prompt engineering and natural language processing.

Ching-hsien Lee is currently the Lead of the Computational Intelligence Technology Division in the Information and Communications Labs, Industrial Technology Research Institute, Hsinchu, Taiwan. His research interests include large language model, natural language technology and generative artificial intelligence.

Jing-Hao Xue received the Dr.Eng. degree in signal and information processing from Tsinghua University in 1998, and the Ph.D. degree in statistics from the University of Glasgow in 2008. He is a Professor of Statistical Pattern Recognition in the Department of Statistical Science, University College London. His research interests include statistical pattern recognition, machine learning, and computer vision. He received the Best Associate Editor Award of 2021 from the IEEE Transactions on Circuits and Systems for Video Technology, and the Outstanding Associate Editor Award of 2022 from the IEEE Transactions on Neural Networks and Learning Systems.

References