Robust Multi-Domain Multi-Turn Dialogue Policy via Student-Teacher Offline Reinforcement Learning

Designing a faultless dialogue system is challenging, especially in the case of multi-domain multi-turn dialogue tasks where each conversation with multiple turns may comprise multiple domains such as finding the best Italian restaurant and the tourist attraction in the nearby location. Recent progress has shown two different approaches to design a dialogue system. The first one is the end-to-end approach in which all dialogue system components are represented by a single learning machine. This approach shows state-of-the-art results in the multi-domain dialogue task with convincing performance [17, 21, 51]. However, this approach is computationally expensive due to the utilization of large pretrained language models (PLMs) such as the generative pre-trained transformer 2 (GPT-2) [28]. Furthermore, completely relying on large PLMs for end-to-end dialogue system is problematic, especially in case of the trustworthy issues. It is because that PLMs are prone to generate out-of-context sentences even given by in context input. Addressing this problem is required. For example, the two best models in DSTC9 track 2 applied the complicated pre-processing and post-processing stages to make sure the model could handle multi-domain dialogue task [21, 51]. The pre-processing stage involved fine-tuning the PLMs with several dialogue datasets, like Schema-Guided [29], Taskmaster [3] and CamRest676 [47]. Meanwhile, the post-processing stage introduced some special modules like fault tolerance mechanism and customized user interface that allowed manual revision.

Due to the aforementioned problems, a practical solution is to construct a pipeline dialogue system. As we can easily integrate any trustable rule function in a pipeline setup, the potential trustworthy issues can be resolved. Importantly, pipeline dialogue systems offer flexibility in optimizing each dialogue system component. Each component of the dialogue system can be optimized individually or jointly. Furthermore, the computation cost of the pipeline system is relatively cheaper compared to the end-to-end approach. Unfortunately, designing a high-performance pipeline dialogue system is demanding. It is because the performance of earlier components has a significant impact on output quality in later components. Especially in the multi-domain multi-turn dialogue task, like in MultiWOZ 2.1 [12], the errors from the natural language understanding (NLU) component considerably degrades the performance of the dialogue system. Although the dialogue policy is well trained given a clean dataset, such a dialogue policy could not handle the noisy input due to the mispredicted user dialogue acts by the NLU component which contain unseen information. In Yeung et al. [50], a learning to learn method was proposed to handle noisy input in a classification task.

To cope with this challenge, this paper presents a robust dialogue policy which is trained to handle the noisy representation due to the errors caused by NLU component. This is different from Henderson et al. [14] where the noise robustness was developed on dialogue state tracking. This work focused on policy learning. The resulting dialogue policy is merged in the pipeline dialogue system that considerably saves computation time. The robustness of dialogue policy is pursued by using offline reinforcement learning (RL) with two different objectives which are optimized to improve noisy feature representation and action decision. To achieve the first objective, this study conducts a student-teacher learning where the student model is trained to map the extracted noisy features to be near to the clean features produced by the teacher model. A new kind of knowledge distillation [15, 20] is implemented for reinforcement learning of dialogue policy. Meanwhile, the second objective is pursued by using focal loss to deal with the multi-label classification problem. Before optimizing the dialogue policy, this paper introduces a data augmentation process by following a dialogue self-play [40] using the ConvLab-2 framework [54]. Such data augmentation is performed to collect additional expert trajectories so as to implement offline RL to train the policy. This study proposes a new approach to enhance the robustness of dialogue policy learning due to the errors introduced by NLU component. The merit of this work is illustrated by the experiments on multi-domain multi-turn dialogue policy optimization under various evaluation metrics.

The remainder of this paper is organized as follows. In Section 2, the multi-domain task-oriented dialogue system is introduced and the previous dialogue policy optimization approaches are surveyed. Section 3 presents the proposed robust dialogue policy learning. The overall learning process including data pre-processing, data augmentation and student-teacher model training are explained in details. Section 4 addresses the experimental settings for evaluation of multi-domain multi-turn dialogue policy followed by the experimental results to illustrate the benefits of the proposed work. The summary of findings from this study is given in Section 5.

The current methods to build multi-domain dialogue system along with dialogue policy learning [35] are surveyed.

Multi-domain task-oriented dialogue is a dialogue task in which there may exist more than one domain in each dialogue session. For example, in the travel assistant task, a user may request information about a train ticket, restaurant and hotel in a dialogue session. It is practical and interesting to work out a multi-domain dialogue system compared to a single domain dialogue [40, 1, 18, 8]. However, achieving a desirable performance for a multi-domain dialogue system is challenging. One of the main difficulties arise by the complexity of dialogue structure due to the information from several domains. Another challenge is the problem definition itself where the dialogue system must satisfy the user goal in a limited number of turns or steps, commonly under 10 conversation turns. In order to promote researches on such a dialogue system scenario, DSTC9 track 2 has provided ConvLab-2 [54] as an evaluation framework that integrates with the MultiWOZ 2.1 dataset [12]. Compared to the other frameworks designed for multi-domain dialogue task [44, 12], Convlab-2 offers two main advantages for system evaluation. Firstly, ConvLab-2 provides a simulated user which enables an end-to-end system evaluation with multi-turn evaluation. Such an evaluation can represent real-world phenomena compared to the component-wise and single-turn evaluation similar to the traditional evaluation [43]. Secondly, this framework offers the flexibility to train the dialogue system components consisting of natural language understanding (NLU), dialogue state tracking (DST), dialogue policy (POL) and natural language generation (NLG) [25], either individually or jointly trained. Figure 1 displays an illustration of how the simulated user and the multi-domain dialogue system interact with each other using the MultiWOZ 2.1 dataset under the ConvLab-2.

Dialogue policy is an important component of the dialogue system which serves as a system brain that determines the response conditioned on dialogue context [32]. Dialogue policy optimization can be formulated under partially observable Markov decision process (POMDP) where the dialogue agent does not have any access to the complete state information s = {ɡ, o} consisting of user goal ɡ and observation o. Instead, the dialogue agent can only obtain o to optimize its policy. Basically, dialogue policy π is optimized as π^* by using RL where the main objective is to maximize the discounted reward accumulation r given by state s_t= {ɡ, o_t} and system action a_t along a whole trajectory over different time steps t via

with a discount factor γ and termination time step T. a_t is a sparse vector where the positive labels or one-hot values represent specific system dialogue acts in the current time step. Previously, the dialogue policy is optimized by utilizing deep RL techniques [7] such as the proximal policy optimization (PPO) [39], adversarial inverse reinforcement learning (AIRL) [13], generative adversarial imitation learning (GAIL) [16] and deep RL with human-in-the-loop paradigm [34, 33]. Methods such as deep Q network [27, 6] or REINFORCE [48] are also feasible to train dialogue policy. However, the dialogue policies optimized using those two methods did not perform well in the multi-domain case due to their inability to handle large state and action spaces. In the Convlab-2 framework, the observation vector o is described as a 340-dimensional vector composed of six distinct components which are user action, system last action, belief state, book info, database pointer and termination with the detailed dimensions shown in Figure 2. Meanwhile, the system action a is specified as a 209-dimensional vector which reflects the system dialogue acts.

Commonly, before starting RL procedure, the dialogue policy must be initially trained by using the behavior cloning method through leveraging the observation-action pairs Ɗ = {o,a^*} in MultiWOZ 2.1 dataset. Here, a^* is the ground truth action provided by the dataset. Behavior cloning aims to find the policy network π_θ with parameter θ by minimizing the cross-entropy loss. Due to the multi-domain case, the learning process needs to deal with the multi-label classification task with sparse one-hot values. Therefore, most of the previous works applied the cross-entropy (CE) loss that merges a weight β to the positive labels or one-hot values in a form of

Once the behavior cloning or training is done, the dialogue policy is fine-tuned by using the RL algorithm via interacting with the simulated user in a predefined environment. In this setting, the reward is very sparse which may cause an unstable learning process. The dialogue agent will receive -1 in every conversation it makes, +5 if the current domain is satisfied, and +40 if all domains in the user goal are satisfied.

Other approaches were developed by jointly training the dialogue policy and the NLG component [53, 5] or even jointly training all of the dialogue components in an end-to-end fashion. In this setting, the action space is the number of all possible vocabularies. Therefore, the dialogue policy is trained with autoregressive learning. Notably, most of the previous works only showed good performance in the component-wise and single-turn evaluation [43]. Meanwhile, this work considers end-to-end system evaluation with the multi-turn evaluation as done in the DSTC9 track 2. Therefore, dealing with the error propagation from the preceding component is essential for a practical dialogue system. Based on the results in DSTC9 track 2, only limited end-toend approaches showed desirable performance [21, 51]. Those methods also required a huge amount of computational cost due to the implementation of large-sized PLM, which is GPT-2.

To relax the computational issue, this paper presents a new approach to train a robust dialogue policy for pipeline dialogue system that offers a low computational cost. The dialogue policy is trained by using student-teacher offline RL to improve its robustness when dealing with noisy input representation. The teacher is designed to guide the student to map the noisy input into the same representation as the clean input. Meanwhile, the offline RL is exploited to show a potential dialogue learning technique when the abundance of trajectories from data augmentation [9] is available.

This section addresses the proposed policy learning strategy for multi-domain multi-turn dialogues. This strategy is generally divided into three stages, as shown by Figure 3. The first is the data pre-processing followed by data augmentation using the dialogue self-play [40]. After the dataset is augmented, the student-teacher dialogue policy model is trained by using the offline reinforcement learning.

Before starting the training procedure of dialogue policy, the conversations in each dialogue session in the MultiWOZ 2.1 dataset need to be processed by converting the raw text data into the observation-action pairs {o,a^*} by using the vectorized functions provided by ConvLab-2. This pre-process results in the 340-dimensional observation and 209-dimensional action vector representations, o ∈ ℝ³⁴⁰ and a ∈ ℝ²⁰⁹. The total number of the generated pairs reaches around 50K observation-action pairs. This number may not be large enough for training an offline RL algorithm that commonly requires huge numbers of trajectories as the training data. Therefore, this work introduces a data augmentation process by using the dialogue self-play method [40]. In this setting, the rule-based policy will interact with the simulated user driven by the agenda-based policy. Besides adding the augmented data for offline training, the data augmentation process aims to generate noisy data to improve the robustness of dialogue policy in the subsequent offline RL training stage. The dialogue self-play is implemented by utilizing the provided RL environment in the ConvLab-2 where all of the stored trajectories in the replay buffer are saved as the augmented data. There will be no overlap with test data which follows the DSTC-9 track 2 evaluation where the evaluation was done under an end-to-end system evaluation environment in the Convlab-2.

However, generating meaningful noisy data in the multi-domain dialogue is non-trivial due to the enriched complexity of observation representation. In the multi-domain dialogue tasks, the user dialogue act consisting of four entries is formed as [intent, domain, slot, value]. Randomly introducing noise to observation o is not helpful since this may produce irrelevant domain information. For instance, considering a user input “I want to find an Italian restaurant.”, the correct user dialogue act would be [‘Inform’, ‘Restaurant’, ‘Type’, ‘Italian’]. However, the dialogue act due to random noise might be produced as [‘Inform’, ‘Taxi’, ‘Type’, ‘Indian’], which bears no relevant information to user input. Particularly, the entries of slot and the value have no connection to that of domain within this noisy dialogue act.

As the main objective of this work is to compensate the noise from NLU component which is mostly represented by BERTNLU, an NLU that uses BERT [11, 36] as a base language model, then the noisy observation ō is generated by using the user dialogue acts ${u^1,…,u^m}$ predicted by BERT as shown by Figure 3 in the middle part. Each predicted dialogue act û contains various information in four entries intent, domain, slot and value from the current user input. The BERTNLU predicted user dialogue acts will change the information of user action, belief state and database pointer stored in observation vector o. The process of generating user dialogue acts can be expressed by

where ƒ_MLP is the multi-layer perceptron (MLP) network added on the top of BERT model ƒ_BERT for intent classification and slot tagging. The input of BERT {x₁…, x_m} is the ground-truth conversation history between simulated user and dialogue system with perfect NLU. To ensure that the observation ō may reflect reasonable noise, the state history in the noisy DST is set to be identical with the state history in the perfect DST, as shown by the dashed line. The generated noise can be said as the noise that occurs in the current time step only. The action a^* is the output of rule-based policy given clean observation o. After the data augmentation process is completed, both MultiWOZ 2.1 and the generated dataset are used to train the dialogue policy.

Before training the dialogue policy by using student-teacher offline RL, we initially train the teacher model by using the behavior cloning to minimize the action probability discrepancy between the learned policy π_θ and the expert policy π_E that can be measured by using the Kullback-Leibler (KL) divergence with approximation as expressed by

where $ηπE$ and $ρπE$ denote the state and trajectory distributions under the expert dialogue policy π_E, respectively. Both $ηπE$ and $ρπE$ can be represented as the dataset used for offline RL training. The most common offline RL method used in this pre-training stage is the behavior cloning which is the simplest approach to learn the expert policy. Although the previous works mentioned that behavior cloning is prone to the compounding errors [26, 46], as the empirical evidences shown in the experimental results, we argue that the compounding errors which can degrade the performance of the dialogue policy do not really exist in the task-oriented dialogue, as long as we can make sure that the policy discrepancy between π_E and π_θ in Eq. (3) is very small with a bound ϵ, namely

It is because each conversation in the task-oriented dialogue is made under very short trajectory in a finite horizon which accordingly minimizes the probability of occurring distributional shift issue leading to severe compounding errors.

It can be verified by deriving the difference of value functions between expert policy π_E and imitated policy π_θ in the finite horizon of POMDP. First of all, consider the value function of a finite horizon as the expected total reward obtained by policy π without any discount factor γ

t is a trajectory consisting of (o, a) pairs from initial until terminal time steps. By using the definition of ||µ – v||₁= 2D_TV (µ||v) where D_TV(µ||v) denotes the total variance distance over two probability distributions µ and v, the difference of values between policies π_E and π_θ can be expressed and manipulated to derive an upper bound as follows

The last two inequalities were provided in Ross et al. [37] and further explained in Ke et al. [19]. Next, by using the Pinsker’s inequality [10] that states $DTV(μ‖ν)≤12DKL(μ‖ν)$⁠, the upper bound U_V in Eq. (6) for finite horizon case can be further tightened by considering the bound ϵ in Eq. (4) in a form of

By considering the fact that finite number of turns T in dialogue task is likely small, around 10 turns at most which only happens in the dialogues containing three domains and at the average only 6 turns in the dialogues containing one and two domains, it is reasonable that the compounding errors of behavior cloning in a short horizon will not be really harmful as long as we can guarantee that the discrepancy of policies between π_E and π_θ is small enough.

Unlike the previous works [34, 16, 42] which employed the cross-entropy or balanced cross-entropy loss to minimize Eq. (3), in this work, the dialogue policy is trained by minimizing the focal loss (FL) [22] which addresses the sparse multi-label classification via

where ω is a focusing parameter, ⊙ is the element-wise product, ℒ₊ and ℒ₋ are the losses for positive and negative labels. The total loss can be calculated through

If we set ω > 0, the focal loss will down weight the easy negatives which have low probability (π_θ(a|o) ≪ 0.5). This means that the learning process will focus more on the harder samples. Meanwhile, if we set ω = 0, it will be reduced to the standard binary cross-entropy loss. Even though the balanced cross-entropy loss introduces a weighting factor β to balance the importance of positive and negative samples, the weighting factor in the focal loss is adaptive to the current learning progress to distinguish between easy and hard examples.

Since the main objective of this paper is to train a robust dialogue policy, the objective stated in Eq. (3) should be changed by considering the noise from BERTNLU model that can be simply formulated as ō = BERTNLU(o). The probability discrepancy between π_E and π_θ is then minimized and approximated as

In order to mitigate such noise, this work introduces a student-teacher offline RL which mainly aims to improve the representation learning. Under this learning stage, we define the teacher parameters as θ = {θ_fe,θ_cls} and student parameters as ϕ = {ϕ_fe,ϕ_cls} which consist of the parameters of feature extractor and dialogue act classifier. Then, the learning objective can be defined to force the noisy features extracted by the student model π_ϕ given ō to be similar to the clean features extracted by the teacher model π_θ given o. The learning process of this stage is depicted by Figure 2. As the teacher classifier can show convincing performance given clean input, then the student-teacher offline RL only focuses on how to improve the feature extractor. The objective of this learning can be formulated by minimizing the following student-teacher (ST) loss function

The first term is the focal loss given noisy input ō that will be used to update ϕ_fe to find the best representation for $fϕcls$ to correctly predict a, which is the same label for o. ϕ_cls directly copies θ_cls and the parameters are frozen during training. Meanwhile, the second term in Eq. (11) is a feature distance loss that will be directly used to train $fϕfe$ to map the noisy input representation to be as close as possible to the clean data representation extracted by $fθfe$⁠. The overall learning process of student-teacher offline RL is shown by Algorithm 1 which first calculates teacher model and then student model.

The experiments were done by using ConvLab-2 framework. ConvLab-2 provides a framework to allow interactions between simulated user and dialogue agent in an environment by using MultiWOZ 2.1 [12] dataset. MultiWOZ 2.1 was an updated version of MultiWOZ 2.0 [2], known as a multi-domain, multi-intent task-oriented dialog corpus [41] that contained 7 domains which are hotel, attraction, restaurant, train, taxi, police and hospital, 13 user intents, 25 slot types, 10,483 dialog sessions, and 71,544 dialog turns. By using ConvLab-2, the end-to-end system evaluation was performed to sufficiently reflect real-world scenarios.

The MultiWOZ 2.1 dataset was initially split into the training, validation and test data with 8434, 999 and 1000 samples, respectively. The feature extractor of both student model $fϕfe$ and teacher model $fθfe$ were formed by MLP networks with two hidden layers consisting of 100 neurons with activation function ReLU. Meanwhile, the teacher and student classifiers, $fθcls$ and $fϕcls$⁠, respectively, were based on MLP networks with sigmoid activation function. The focusing parameters ω and λ in the feature distance loss Eq. (11) were set to 2 and 1, respectively. The dialogue policy parameters {θ, ϕ} were optimized by using Adam with the decoupled weight decay (AdamW) [24] with learning rate α =5e-4 and batch size N =32. The data generated in the data augmentation process were around 36000 observation-action pairs.

The benefit of the proposed method for improving dialogue system robustness was evaluated with the end-to-end system evaluation over a set of 1000 dialogues, consisting of 337, 523 and 140 dialogues containing 1, 2 and 3 domains, respectively. The dialogue system performance was measured through the interactions with the simulated user where the NLU, DST and NLG in a pipeline system were identical to the ConvLab-2 default settings, which included BERT [11] based NLU (BERTNLU), rule-based DST and template NLG, respectively. The experimental results were carried out on a PC with a CPU i9-10900K, 128GB of RAM, and a GPU NVIDIA RTX 2080Ti. Comparative studies were performed by using five primary metrics listed below

• –

  success rate: judges whether constraints (e.g. hotel location or hotel price) and requests (e.g. hotel phone number) in the user goals have been satisfied by system.

• –

  F1 score: judges if all requested information like taxi type or taxi phone number has been informed. This score is computed from precision and recall.

• –

  complete rate: ratio of the completed user constraints.

• –

  booking rate: ratio of the successful dialogues for booking a request which is only available in the domains of hotel, restaurant and train.

• –

  average conversation turns between dialogue system and user for successful and all dialogues.

For the first four metrics, the higher the better. For the last metric, the lower the better. The proposed method was compared with two types of baseline methods. The first type of baselines was the methods which only optimized the dialogue policy, mentioned as follows

– behavior cloning optimized with balanced cross-entropy loss (Eq. (2)) and asymmetric loss (ASL) [30]. ASL was shown good benefit in the multi-label image classifications, leading to state-of-the-art results. ASL is an extension of focal loss where the weights for positive and negative labels are different, ω⁺ and ω⁻, respectively, by

The total loss is accumulated just like Eq. (9). $πθm(a∣o)$ is an asymmetric probability with a shift m calculated by $πθm(a∣o)=max(πθ(o)−m,0)$⁠. There are three hyperparameters ω⁺, ω⁻ and m in this method.

– policy gradient (PG) [48]: a standard policy based method in RL where the gradient with respect to the cumulative reward is calculated to update π_θ(·)

– proximal policy optimization (PPO) [39]: an actor-critic method implemented by maximizing the clipped surrogate objective to train the actor and minimizing the regression error to train the critic.

– guided dialogue policy learning (GDPL) [42]: a solution based on adversarial inverse RL [13] method which learns a reward function by using the expert trajectories and uses this information to train the dialogue policy agent sequentially in the same loop.

Another type of baselines conducted the optimization in an end-to-end fashion. All components from NLU until NLG were optimized jointly.

• –

  domain aware multi-decode (DAMD) [52]: a multi-action data augmentation scheme to produce diverse response by using additional state-action pairs.

• –

  minimalist transfer learning (MinTL) [23]: a transfer learning framework offering plug-and-play approach for task-oriented dialogue system.

• –

  UBAR [49]: a task-oriented dialogue model which used distilGPT-2 model [38] as the base model. The model was fed not only with the user and response sentences, but also with database search result and belief state from the previous steps.

• –

  offline RL methods, which are GPT-critic [17], critic regularized regression (CRR) [45] and decision transformer [4]. All of them used GPT-2 as the base model.

• –

  GPT-TDAPT (GPT-2 model with task and domain adaptive pretraining) [51]: proposed five stages of learning. The first was the domain adaptation using the pre-trained GPT-2 where the datasets including Schema [29], Camrest [47], Taskmaster 2019, Taskmaster 2020 [3] and MSR-e2e were used. Multi-task fine-tuning using MultiWOZ 2.1, data pre-processing and post-processing, fault tolerance mechanism, and rule-based postprocessing for refining the agent utterances were the other four stages.

• –

  AUGPT (GPT-2 finetuned with auxiliary tasks) [21]: conducted similar implementation as the GPT-TDAPT with two distinctions. First, there was no post-processing approach in this work. Second, the auxiliary tasks were employed to increase the consistency in sentence generation given the identical system action responses.

Firstly, the evaluation was carried out to investigate the effect of data augmentation by utilizing the dialogue self-play method under different behavior cloning objectives. The configuration of the dialogue system was identical for the methods including BERTNLU, rule-based DST and template NLG. We set β, m,ω⁺,ω⁻ to be 5, 0.05, 0 and 2, respectively. As shown in Table 1, the proposed data augmentation process successfully produced the meaningful data, as indicated by the performance gaps between the models trained by using only MultiWOZ 2.1 dataset and the models trained with the generated data. One of the main reason of this phenomena is the simplicity offered by the generated data indicated by the number of system dialogue acts in the action vector a, as shown by Figure 4. Compared to the generated data, MultiWOZ 2.1 contains more complicated observation-label pairs as more than one-fifth of data samples have at least 4 positive labels. Furthermore, it also contains the data with more than 8 positive labels which might be hard for the model to understand them. Therefore, dialogue policy could learn better from the generated dataset than the original dataset. Although the performance gap between models trained using MultiWOZ 2.1 and generated data is significant, the best results were obtained by combining those two datasets. Training the models solely by using the generated data only resulted in suboptimal performances. This evidence implies that each dataset has unique important information that must be learned by the dialogue policy. Another interesting finding is the suboptimal performance of the behavior cloning with asymmetric loss in comparison to the behavior cloning with focal loss, even after adjusting the hyperparameters in the Eq. (12). The results are shown by Table 2. Since the behavior cloning with asymmetric loss is quite sensitive to the hyperparameter tuning, the resulting performance is worse than that with focal loss. Using asymmetric loss, the most notable evidence is seen when the precision score drastically declined after the value of ω⁻ was raised. Despite the fact that the success rate difference between behavior cloning trained with asymmetric loss and focal loss is relatively not significant, the focal loss effectively helped the model in achieving a high F1 score of 88.6 as compared to 85.1 for the model trained with asymmetric loss.

The next experiment was performed in order to demonstrate the benefit of the proposed method, called the student-teacher offline RL (denoted by STORL). In this evaluation, the proposed method was compared with both pipeline and end-to-end dialogue system. The evaluation results are shown by Table 3. From the result, it can be seen that the proposed method properly learned how to map the noisy input oō into the better representation that was closer to the representation of teacher model given clean input o. As a result, the dialogue policy could reduce the impact of the noisy input while still generating appropriate actions, indicated by absolute improvements 1.4% and 0.8% in the success rate and complete rate, respectively, when compared to the teacher model. Conditioned on the clean input, which means the true user dialogue acts were directly passed to the DST component, the learned student encoder $fϕfe$ could maintain to map the clean input to the appropriate representation indicated by the nearly identical performance between teacher model and the proposed method given by the perfect NLU setting. Furthermore, compared to the rule-based policy which is always set as the upper bound of the dialogue policy optimization, the proposed learning scenario even performs better than the rule-based policy, especially in terms of success rate and booking rate by 0.9% and 0.6%, respectively. Unfortunately, there was a tradeoff that should be paid in this learning strategy. Due to the enforcement to map noisy representation to be as close as possible to the clean representation, F1 score of the learned dialogue policy was degraded from the teacher model, by around 1.8.

In order to show further comparison with the dialogue policy baselines, additional experiments were conducted by training all of the dialogue policy baselines by utilizing a combination of MultiWoZ 2.1 with the augmented data. The augmented data were leveraged in the pre-training stage. The optimization process followed the default setting of each baseline method which mainly used the behavior cloning in accordance with the balanced cross-entropy loss. The results are shown in Table 4. As can be observed, all of the dialogue policy baselines are significantly improved due to the addition of augmented data during the pre-training phase. The best baseline performance is obtained by PPO with human-in-the-loop (PPO HITL) which can reach the performance close to that using the STORL (Teacher) model. However, by using this setting, the learning strategy in PPO HITL introduced expert feedback twice. The first was in the pre-training stage which utilized the expert trajectories, and the second was in the PPO training in which an expert provided the action corrections in every interaction with the simulated user. As all of the dialogue policy baselines could not perform better than STORL (Teacher) model, this finding indicates that the offline RL is enough for solving the dialogue policy learning in the dialogue task. Also, the resulting training data are sufficient so that the learning strategy can guarantee to reach the derived bound in the Section 3.2 which is achieved by using the focal loss. Furthermore, even though all of the baselines were trained by using an additional dataset, all of them still could not outperform the rule-based policy, while the STORL showed better performance than the rule-based policy due to the well-trained encoder that can compensate for the errors from BERTNLU.

Compared to the other baselines that applied either RL or offline learning with large PLMs, the proposed method was only outperformed by the GPT-TDAPT [51] which achieved the best performance in the DSTC9 track 2. Table 5 shows the performance comparison between STORL and GPT-TDAPT which also includes the performance of each model given different test data categorized by the number of domain occurrences in each dialogue. Even though GPT-TDAPT outperformed STORL in overall testing, in terms of computational cost comparison, the proposed method only required 7.5 minutes to complete the testing consisting of 1000 test dialogues. GPT-TDAPT required nearly 74 minutes to complete the testing process. Moreover, the learning process introduced by GPT-TDAPT implemented some non-trivial tricks in both preprocessing and post-processing stages which required human manual design to make sure the model could achieve a convincing result. As a result, the other end-to-end approaches that employed GPT-2 model such as AUGPT [21] and GPT-critic [17] showed suboptimal performances due to the missing of the manual design tricks. On the other hand, this work proposes a much simpler and straightforward learning process to improve dialogue policy performance which makes the reproduction of the results becomes much easier. Conditioned on different categories of test data, the results show that the proposed method could achieve competitive performances in the simple and hard cases, where each dialogue consisted of 1 domain and 3 domains, respectively. In case of dialogues containing 2 domains, the STORL was significantly better than GPT-TDAPT by around 10% in success rate and 5% in both complete rate and booking rate. This empirical evidence may suggest a future research for representation learning methods that can handle all possible cases.

In comparison with the teacher model, Table 6 shows the benefit of using the trained encoder through STORL. In the goal IDs 304 and 357, the STORL model generated the responses identically to the teacher model given clean input although BERT did not predict the first user dialogue act “[‘Inform’, ‘Train’, ‘none’, ‘none’]”. On the other hand, the teacher model, fed with noisy input from BERT, produced the irrelevant response by offering the train schedule in the goal ID 304, and failed to generate any response in goal ID 357. In the goal ID 623, even though BERT outputs a wrong dialogue act “[‘Inform’, ‘Hotel’, ‘Type’, ‘hotel’]”, the STORL model could provide a nearly similar response to the teacher model given the clean input while the teacher model given noisy input could not find any hotel for the user. All of the findings are in line with the distance metrics (in both ℓ₁ and ℓ₂) and the distributions of latent features due to the introduction of the STORL and the teacher model given the noisy input (Teacher) relative to the teacher model given the clean or ground truth of user dialogue acts (Teacher*) as shown by Table 7. It can be seen that the STORL sufficiently maps the features conditioned on the noisy input to be closer to the teacher features given by the clean input.

More empirical evidence about the benefit of using STORL is shown by Table 8 which illustrates a qualitative comparison of the conversations made by STORL policy and rule-based policy given the noisy input from BERTNLU. Due to the mispredicted slot in the second conversation turn, ‘Destination’ to ‘Departure’, the rule-based policy repeated the question which asked for the user destination until it obtained slot ‘Destination’. However, the rule-based policy could not find any schedule for the train departing to Peterborough due to the inability for handling the conflicted DST information where the DST stored two different values for ‘Departure’ which were Cambridge and Peterborough. Meanwhile, the proposed STORL agent could handle the mispredicted user dialogue acts due to the well-trained encoder which could map the noisy input into a better representation. Therefore, although BERT mispredicted the slot ‘Destination’ to ‘Departure’, STORL could provide an appropriate response by offering a train schedule to Peterborough.

A novel method for improving the robustness of dialogue policy in addressing noisy input due to the error output from NLU component in the dialogue system, named as the student-teacher offline reinforcement learning (STORL) has been proposed. The proposed method was designed according to two strategies including student-teacher learning and offline reinforcement learning. Student-teacher learning aimed to force the student model to map the extracted features of the noisy input to be close to the clean features extracted by the teacher model. Meanwhile, the offline reinforcement learning which aimed to minimize the focal loss was used to train the dialogue policy to be able to provide appropriate response given the user input by only utilizing the observation-action pairs stored in the dataset. The experimental findings demonstrated that the proposed hybrid learning did enhance the performance of the pipeline dialogue system in the ConvLab-2 end-to-end system evaluation under MultiWOZ 2.1 dataset. Furthermore, the competitive results were also obtained when compared to the end-to-end approaches by using the pre-trained GPT-2 model with lower computational cost and simpler learning process. The source codes based on STORL can be accessed in https://github.com/NYCU-MLLab/. Future works will include the extension of this method to the other dialogue domain [31].

This work was supported in part by the National Science and Technology Council, Taiwan, under Contract NSTC 112-2634-F-A49-006.

Mahdin Rohmatillah received his M.S. degree and Ph.D. degree from the National Sun Yat-sen Univesity, Taiwan, and the National Yang Ming Chiao Tung University, in 2018 and 2024, respectively, both in electrical and computer engineering. His research interests include machine learning, large language models and dialogue systems.

Jen-Tzung Chien is currently the Lifetime Chair Professor with National Yang Ming Chiao Tung University. He has published extensively, including three books and more than 250 peer-reviewed articles, many on machine learning, deep learning and Bayesian learning with applications on natural language processing and computer vision. He was the recipient of the Best Paper Award in IEEE Workshop on Automatic Speech Recognition and Understanding in 2011 and IEEE International Workshop on Machine Learning for Signal Processing in 2023.