Speaker-Specific Articulatory Feature Extraction Based on Knowledge Distillation for Speaker Recognition

Speaker recognition has been an important task for artificial intelligence applications [1, 3, 20] for years. To represent the short-term spectral characteristics of speakers, traditional methods for speaker recognition typically used Gaussian mixture models (GMM-UBM) and i-vector [9, 10, 17]. Villalba et al.[43] presented a framework based on the variational autoencoder paradigm to deal with latent variables between i-vectors. Chin et al. [7] combined i-vector and sparse representation classifier for speaker identification. Kinnunen et al. [18] adopted i-vector as the basic unit for voice conversion. In recent years, deep neural networks (DNN) have been widely used to generate features for speaker recognition [5]. In these studies, DNN was applied to directly capture speaker characteristics and produce speaker embedding as a speaker representation [2, 23, 42, 45, 46]. Recently, most speaker verification (SV) systems were based on x-vector features [37, 39], and the architecture consisted of frame-level and segment-level feature transformations. The frame-level feature transformation was based on time delay neural network (TDNN) structure [44]. It has been proven that using TDNN, speech characteristics extracted from multi-frame signals with shift-invariance were more efficient than those from single-frame signals [36]. The segment-level feature transformation applied statistics pooling to aggregate variable-length features to obtain a fixed-dimensional vector. In [15], a statistics pooling time delay neural network was proposed to improve the x-vector learning ability by capturing more robust speaker characteristics. Nowadays, many studies were focused on improving speaker recognition performance. Tang et al. [41] integrated TDNN and long short-term memory (LSTM) to capture speaker information at different levels. Zhu et al. [51] proposed a self-attention mechanism for DNN-based embedding and computed the embedding as a weighted average of the speaker’s frame-level features. However, speech attributes, such as articulatory features (AFs), which can be used to characterize speaker-specific information, are rarely considered in speaker recognition.

AF is an important representation of phonological properties during speech production. More precisely, AFs are abstract classes for describing the movements or positions of different articulators during speech production [6]. AFs have been successfully used as features in speech recognition in recent years [19, 26, 27, 31, 40, 48], and commonly used methods combine acoustic features and AFs to improve speech recognition performance. For the speaker recognition task [13, 14], Shen et al.[34] utilized decision tree-based phone cluster models to cluster the speech segments with speaker characteristics for speaker diarization. Li et al. [22] proposed a feature-level fusion and a score-level fusion approach by combining acoustic and AF information for SV, and the authors indicated that concatenating AF with acoustic features can improve the performance dramatically, but access to the AF is impractical for real world applications. Thus, using an acoustic-to-articulatory inversion technique can deal with this issue [22]. Siniscalchi et al. [35] showed the relationship between phone and attribute, and computed a score by articulatory feature detectors to describe the activation level of the specified speech phonetic features that the current frame exhibits. Wu et al. [47] integrated senones with an AF posterior probability vector to model a wide range of acoustic-phonetic phenomena in a language for code-switching event detection.

For feature fusion, most of the studies combining various vectors are based on the concatenation of different features [16, 33]. Using an AF extractor is a direct way to generate the AF features of the training speaker dataset for feature fusion. However, the AF features obtained from individual modules may cause performance drops due to representation specificity [49]. Therefore, this study proposes a speaker-specific AF extraction model via knowledge distillation (KD) to capture the speaker characteristics from acoustic features and then learn the speaker-specific information from AFs to distinguish speakers. By applying KD rather than feature concatenation, speaker embedding model learning based on acoustic and AF features simultaneously cannot only achieve a better fusion performance, but also reduce the impact of representation specificity.

In speaker recognition, the state-of-the-art (SOTA) method for speaker discrimination is based on the probabilistic linear discriminant analysis (PLDA) backend [4], by comparing the speaker embedding of the input utterance with the embedding features of the speakers. McCree et al. [25] extended the PLDA model to include segment duration as well as to distinguish between session and channel variability. Rohdin et al. [32] developed an end-to-end speaker recognition system that is initialized to mimic an i-vector with a PLDA baseline. Snyder et al. [38] used a diarization system based on PLDA as a front-end for speaker recognition.

In this paper, we propose a speaker-specific AF extraction technique based on KD for speaker recognition. First, an articulatory feature (AF) extractor is trained as a teacher model to extract the AF profiles, in which each AF profile can be seen as the speaker-specific information. Next, a student model as low-level layers of a speaker embedding model is trained based on multi-task learning using KD to learn more robust speaker embedding. Finally, speaker embeddings are extracted from the high-level layer, and the obtained speaker embeddings are further used to train a PLDA model for scoring.

This paper is organized as follows. An improved structure of frame-level transformation of x-vector for speaker embedding extraction is described in Section 2. The relationship of phones and AFs for AF extractor training is presented in Section 3. A KD model based on acoustic and AF features for speaker embedding extraction is presented in Section 4. Section 5 introduces the dataset and details the experimental results of different comparisons. Finally, conclusions are given in Section 6.

In this section, the structure proposed in [15] is used to improve the x-vector representation, which is regarded as the SOTA feature representation for speaker recognition. As the TDNN layer focuses on local feature extraction, the high-level features extracted through non-linear transformation in the preceding layers may lose some subtle information using low-level features. Therefore, this study integrates TDNN with statistics pooling to exploit the potential ability of the network by considering the variation in temporal context.

Researchers have been working to improve the performance of speaker recognition [39] using the standard x-vector. For x-vector feature extraction, a speaker discriminative network is trained with a large amount of speech data from speakers for enrollment and evaluation. Table 1 shows the x-vector architecture. Let X_l = [x_l;1x_l,2 … x_l,T]^T represent an input sequential vector with T frames at the l-th layer. Suppose that s_l,t = flatten(X_l,t), where X_l,t = [x_l,t−τx_l,tx_l,t+τ]^T is the spliced output of X_l at frames {t − τ, t, t + τ} with a dilation factor τ. The frame-level transformation of the TDNN can be written as

where $Wl+1∈ℝdls×dl+1x$ the weight matrix of size $dls×dl+1x,dls$ is the length of $sl,t,dl+1x$ is the length of x_l+1,t, b_l+1 is the bias vector at the (l + 1)-th layer and α(·) is the activation function.

After the transformation of L frame-level layers, statistics pooling is performed by aggregating all output vectors of the last frame-level layer (i.e., L-th layer) to form a fixed-dimensional vector as

where mean(•) is the mean function and std(•) is the standard deviation function.

By pooling the frame-level features to the segment-level features, the discriminative probabilities of speakers can be predicted by I dense layers.

where h₀ = z and the predicted probabilities of N speakers Ŷ = {ŷ_n ∈ ℝ: 0 ≤ ŷ_n ≤ 1 and $∑ny^n=1}$ is determined by a softmax function. In this study, a cross-entropy loss L_SE for speaker recognition task is defined as follows.

where B is the batch size and y_i,n represents the label of the n-th speaker of the i-th sample in the training process. After model training is completed, x-vector embedding can be obtained from the output of the penultimate dense layer.

As the TDNN focuses on local feature extraction, segment-level feature extraction through statistics pooling and non-linear transformation may lose some subtle information using frame-level features. In order to further improve the information representation in TDNN, a feature combination method for each time-delay layer proposed in [15] (named stats-vector) is adopted to integrate the TDNN with statistics pooling to exploit the potential ability of the network by considering the variation in temporal context. As shown in Equation (1), the TDNN output is obtained by context input transformation. To further consider the variation in the input features, we directly combine s_l,t and the statistics pooling result of X_l,t to form a new input feature vector, which is then fed into the transformation layer.

During the stats-vector training, assuming that the input sequential vectors of $Xl,tT$ are equal at the l-th layer (x_l,t−τ = x_l,t= x_l,t+τ), the output of mean(•) is the same as the current time vector x_l,t, and std(•) produce a zero vector, thus, $x^l+1,t$ is an approximation of x_l+1,t as a result of the assumptions on stationarity. Otherwise, the variation of the input sequential vectors of $Xl,tT$ will produce different local means and standard deviations, which provide more helpful features for model training.

This study builds an AF extractor to predict the probabilities of the speech attributes present in the speech signal and helps the speaker embedding extraction model to extract more representative speaker features.

AFs can be distinguished based on pronunciation places and manners by speaker voices [21]. As shown in Table 2, 20 attributes, defined in this study, are used to train an AF extractor.

According to linguistics, a word is composed of several phonemes to represent the changes in pronunciation (place or manner of articulation). Thus, the labels of speech attributes in signals can be defined by phones. Table 3 shows the relationship of phonetic symbols and speech attributes in English [50]. The phonetic symbols are defined by the CMU dictionary, which is based on the ARPAbet symbol set developed for speech recognition tasks, and could be downloaded at http://www.speech.cs.cmu.edu/cgi-bin/cmudict.

In this study, a DNN for acoustic-to-articulatory inversion is constructed for AF recognition. This study constructs a multilayer perceptron (MLP)-based model and a TDNN-based model to explore the effects of multi-frame integration at different layers and evaluate the performance of AF recognition, as shown in Figure 1. As each speech phone may correspond to one or several attributes, in this study, the Kaldi automatic speech recognition (ASR) toolkit [29] is used to align the phone positions of the speech signals in the GMM-HMM-based acoustic model training procedure. According to the alignment information, every segment of the training speech signals can be labeled exactly with the attributes which the phone corresponds to. As shown in Figure 2, the AF extractor training is based on the acoustic features of a segment of 𝒯 frames. And the acoustic features are extracted from the raw signals according to the duration of each phone labeled in the GMM-HMM-based acoustic model training procedure. Acoustic features S can be labeled with 20 binary values to indicate which speech attribute is present in the acoustic features.

Through the non-linear hidden layer transformation, the AF extractor maps acoustic features into the posterior probabilities of all speech attributes. In this study, the silence attribute SIL was ignored for AF extractor training, because silent signals will be removed in speaker recognition to reduce the interference of non-verbal signals. AF extractor training is different from traditional classifier training. As the acoustic features may contain more than one attribute and the model is trained with a distributed (not one-hot) representation, the prediction score is estimated based on a sigmoid function to scale the value to lie between 0 and 1. The loss function is the mean squared error ℒ_AF defined as follows.

where B is the batch size, C is the number of attributes, 𝒜_i,j represents the target label of the j-th attribute of the i-th sample, and $A^i,j$ is the actual DNN output for the j-th attribute given the i-th sample.

As described in [42], speaker representation is derived by averaging all corresponding d-vectors for the utterances of a specific speaker. Even if speaker embedding is extracted from the same speaker, the speaker embedding is slightly different from each other. Therefore, making the model learn more robust representation to precisely recognize speakers is still a challenge.

As speech pronunciation can reflect the speaker’s speaking styles and habits, in which not only different speakers have different pronunciations, but also the same speaker has different presentations. In this study, considering that speech signals are constructed by several different phones, a knowledge distillation model is proposed to characterize speaker-specific information from AF profiles. The speaker embedding model not only captures the speaker characteristics from acoustic features, but also learns the speaker-specific features from the AF profiles to form a more robust speaker representation.

Inspired by [24], speaker embedding extracted with phonetic information by multi-task learning can effectively improve the performance of speaker recognition. In [24], the frame-level shared layers of the x-vector are shared with the ASR network, in which the shared layers learn more informative features by classifying the phonetic features in frame-level transformation. Assuming there are four parameters {θ_s, θ_a, θ_f, θ_l} in the model, where θ_s denotes the frame-level shared layers, θ_a denotes the remaining parameters of the ASR network at frame level, θ_f denotes the remaining parameters of the x-vector at frame level, and θ_l denotes the segment-level parameters of the x-vector. These four parameters will be updated simultaneously at the training stage by gradient backpropagation.

In [24], a training strategy is proposed to deal with the following situation. That is, when the training speaker dataset does not have phonetic labels, it is desirable to use another dataset containing phonetic labels to train the ASR network at different mini-batches. For example, given two specific domain datasets, the speaker dataset only contains speaker labels, while the phonetic dataset only contains phonetic labels. The two datasets are thus merged into different mini-batches. In the training process, when the speaker samples are fed into the model, {θ_s, θ_f, θ_l} are updated, while when the phonetic samples are fed into the model, {θ_s, θ_a} are updated.

As mentioned in Section 4.1, multi-task learning does not train speaker classifier and phonetic label classifier from the same samples as the training speaker dataset does not have phonetic labels, which will reduce the dependency of the model on these two tasks. Therefore, this study proposes a KD-based speaker embedding extraction model based on multi-task learning to eliminate the problem. In addition, as phonetic information only corresponds to a specific speech sound (one-hot representation) and the AF profiles can be seen as the activation levels of places and manners, we replace the phonetic information with AF profiles as the features for robust speaker representation. Figure 3 shows the proposed KD architecture for speaker embedding extraction, which consists of four procedures based on multi-task learning as described in the followings.

1. An AF extractor is trained as a teacher model for extracting the AF profiles of the input speaker dataset.

2. The predicted AF profiles are used as the soft targets for shared layers learning. The shared layers are transferred to the student model to capture the speaker-specific AF by minimizing a loss function in which the target is the distribution of attribute probabilities predicted by the teacher model.

3. At the same time, the shared layers are used for speaker embedding model training based on multi-task learning.

4. After model training is completed, the speaker embedding is obtained from the output of the penultimate dense layer.

In AF extraction, the AF profiles obtained from different samples are utilized to learn the speaker-specific information. First, according to the context of F frames at the frame-level shared layer S, each sample (T frames) is fed into the model and is divided into T − F +1 frame-level data for shared layers training. Second, an AF extractor is used to obtain the AF profiles of these frame-level data as soft targets, which can represent the different activation levels of speech attributes of speakers. Finally, the output of the shared layer S is fed to a dense layer with a sigmoid function to obtain the AF probabilities. The loss function is defined as follows.

where p_i(attr_j) represents a soft target for the j-th attribute of the i-th data obtained from the AF extractor, and $p˜i(attrj)$ represents the corresponding AF probability determined from the shared layers.

In speaker embedding extraction, the training loss function is the same as in Equation (5). The loss is back-propagated to update the parameters.

Assuming the proposed model is composed of four parameters {θ_s, θ_af, θ_f,θ_l}, where θ_s denotes the frame-level shared layers, θ_af denotes the parameters of the last dense layer of the AF predictor, θ_f denotes the remaining frame-level parameters and θ_l denotes the segment-level parameters. In the training process, θ_f and θ_l are updated based on ℒ_SE and θ_af is updated based on $ℒ˜AF$⁠. As the shared layers are used for AF extraction and speaker embedding extraction simultaneously, θ_s is updated by the total loss ℒ_KD defined as follows.

During the shared layer training, the network not only captures the speaker characteristics from acoustic features, but also learns the speaker-specific information of the AFs by KD.

Speaker recognition model is generally trained by a large number of speakers to learn the rich phonetic characteristics of speakers, and then the trained model is used to further extract speaker embedding. The speaker embedding is extracted from the output of the high-level layer of the trained model for speaker discrimination.

PLDA is utilized to compute the likelihood ratios during the test. In standard PLDA [30], speaker embedding ϕ_n,r representing sample r from speaker n is given by

where μ is the global mean, $Vyn$ represents the between-speaker variation, $Uxn,r$ is the within-speaker variation and ϵ_n,r represents the residual noise. The PLDA parameters Θ = {μ, V, U, Σ} are updated by the expectation-maximization (EM) algorithm. After that, assuming there are two speaker embeddings ϕ₁ and ϕ₂ for similarity comparison, the score between the two speaker embeddings is calculated as

where H_s is the hypothesis for the same speaker and H_d is the hypothesis for different speakers. The details of the PLDA scoring could be found in [11].

In this study, two datasets were used for model training and testing, including VoxCeleb dataset and LibriSpeech dataset. The VoxCeleb dataset was used to evaluate the performance of the proposed mechanism on the speaker recognition task. The VoxCeleb dataset consisted of two versions, VoxCeleb1 [28] and VoxCeleb2 [8], which were released for commercial and research purposes for speaker recognition. The LibriSpeech dataset was used to evaluate the performance of the AF extractor.

• Training data of the embedding extraction model: The baseline system and the proposed system were trained on VoxCeleb2 dataset without any data augmentation. The VoxCeleb2 dataset provided two subsets for evaluation: DEV and TEST sets, which contained over 1 million utterances recorded from 6,112 speakers, extracted from YouTube video-sharing platform. The DEV set contained 1,092,009 utterances from 5,994 speakers and had no overlap with the speakers in the Vox-Celeb1 dataset. In the experiments, the DEV set was used to train the speaker embedding models and the PLDA models.

• Training data of AF extractor: LibriSpeech is a corpus of approximately 1,000 h read English speech, and this corpus was released for automatic speech recognition (ASR) task. The LibriSpeech consists of 460h “clean” speech and 500 h “other” speech. The clean speech (from 1,172 speakers) was used to train the GMM-HMM-based acoustic models using the Kaldi ASR toolkit, and the phone alignment information was obtained during the training process. After that, an AF extractor was trained with the phone alignment information corresponding to the 20 speech attributes.

• Testing data: The VoxCeleb1 dataset contained 153,516 utterances from 1,251 speakers, which was also obtained from YouTube videos. In this study, VoxCeleb1 dataset was used to evaluate the speaker recognition performance.

In the experiments, the audio files in the VoxCeleb dataset were labeled with speaker identities and the silence interval was removed using energy-based voice activity detection. The input features were 40-dimentional Mel-frequency cepstral coefficients (MFCCs), and the spectrogram was extracted based on a 25 ms window with a stride of 10 ms.

• AF extractor: Five different models were trained for AF extraction evaluation, including mlp-1f, mlp-9f, mlp-15f, tdnn-9f and tdnn-15f (the first term denotes model name; the second term denotes the number of frames for input). Totally, five hidden layers, each with 512 nodes, were used for non-linear transformation, parametric ReLU (PReLU) was used as the activation function, and the final output layer used a sigmoid function to scale the output values to lie between 0 and 1 as the predicted probabilities of 20 speech attributes. For MLP model, the input was a one-dimensional vector, which directly concatenated all framelevel features as input. For example, mlp-9f concatenated sequential features of 9 frames as input vector. For TDNN model, the frame-level features (two-dimensional matrix) were fed to the model, and each layer received input from the temporal outputs of the previous layer. The main difference between tdnn-9f and tdnn-15f was: tdnn-9f architecture as shown in Figure 1, in which the input of the 2nd-layer and the 4th-layer do not consider the temporal context, but tdnn-15f architecture considered the temporal context as input at the 2nd-layer and the 4th-layer, and the input was the spliced output of the previous layer at frames {t − l, t, t + 1}.

• Embedding extraction model: The baseline and the proposed KD models were built using the same architecture. In the training stage, in order to reduce the training cost, the inputs of mini-batches were fixed to three-second long spectral features; each input feature was the MFCC spectrogram extracted from a 25 ms window with a stride of 10 ms to obtain a sequence of spectral features. For the baseline x-vector as shown in Table 1, there were 512 output nodes in frame1 to frame5 layers and 1,500 output nodes in frame6 layer. For stats-vector model, if the time-delay layer considered temporal context, the subsequence of the output vectors from previous layer were concatenated with the statistics pooling results in the same context to form a new input feature vector, e.g., frame2, frame3 and frame4 layers. For KD, the shared layers further considered the AF recognition loss to learn the speaker-specific information from AF profiles. After that, the statistics pooling produced 3,000 output nodes that were twice the length of the input nodes (consisting of 1,500 nodes for mean operation and 1,500 nodes for standard deviation operation). Segment7 and segment8 layers also consisted of 512 output nodes, and the final softmax layer consisted of 5,994 output probabilities of the training speakers. Rectified linear unit (ReLU) activation function and batch normalization (BN) were applied to each transformation layer for non-linear mapping. For the testing stage, the entire speech signals were fed into the model to extract speaker embeddings. Because the models were trained by segment-based data and the speaker embeddings were extracted by utterance-based data, there is a mismatch between training and testing.

Because different AFs in a speech pronunciation have rapid change and different durations, five AF extraction models with different input durations were trained to evaluate the performance of AF recognition. In the spectrogram, a single frame is the shortest duration, and most AF durations are shorter than 15 frames. This study analyzed the AF extraction for a duration less than or equal to 15 frames, in which the mlp-1f is the AFs extracted from a single frame (i.e., 25 ms); mlp-9f and tdnn-9f are the AFs extracted from 9 frames (i.e., 105 ms); and mlp-15f and tdnn-15f are the AFs extracted from 15 frames (i.e., 165ms). As the AF label of LibriSpeech clean set was complete, we further divided the data into a training set and a testing set for AF recognition. In the training set, 921 speakers containing 104,014 recordings were selected randomly for model training. In the testing set, the remaining 251 speakers containing 28,939 recordings were selected for evaluation. As shown in Table 4, this experiment evaluated the performance on multiple attributes prediction and single attribute prediction. The multiple attributes prediction means that the accuracies were calculated based on the correctness of 20-dimensional rounded AF profiles and AF labels; the single attribute prediction means that the accuracies were calculated based on the correctness of each element of the rounded AF profiles and AF labels. Furthermore, the predicted result of each speech attribute is shown in Figure 4. Obviously, when the AF extractor was trained by single-frame features, the MLP model achieved the worst accuracy for AF recognition due to insufficient information in the input. The MLP model with an input of 9 frames achieved the best accuracies of 76.70% and 96.46% on multiple attributes prediction and single attribute prediction, respectively. Therefore, using multi-frame features as input can provide more information than single-frame features and achieve the best performance.

Moreover, we further evaluated the performance on phone prediction from the AF profile to ensure the dependency between phone and AF profile. According to Table 3, each AF profile was mapped to a phone label by the Euclidean distance between the predicted AF profile and the phones with one-hot representation of speech attributes. As shown in Figure 5, mlp-1f still performed the worst in phone predictions. It is worth noting that the models with the input of 9 frames and 15 frames on vowel prediction achieved similar performance. But in consonant prediction, the performance of models with an input of 9 frames was better than the models with an input of 15 frames. The reason may be that consonants have short duration and rapid change in pronunciation, and the input with 15 frames has the feature differences at the beginning and the end of a phone. On the other hand, the experiments showed that the performances did not have much difference between the MLP and TDNN models, thus, the mlp-9f (best performance) was selected for the next experiments.

According to the results in [12], using soft targets to train the models can provide more information than hard targets and improve the performance of predictions. As the AF can be used to characterize speaker-specific information, this experiment aimed to prove whether the AF profiles are easier to distinguish speakers than hard targets.

First, the AF extractor was used to obtain the corresponding AF profiles in the VoxCeleb2 and VoxCeleb1 datasets for training and testing, respectively. Next, the x-vector and stats-vector models were trained for speaker embedding extraction. In the training stage, each input was fixed to a sequential AF profiles of 3s. In the testing stage, each corresponding sequential audio AF profile was used to extract speaker embedding. Finally, the VoxCeleb1 (cleaned) list was used to evaluate the performance by PLDA scoring. In the following results, “AF hard” denotes that the input data were obtained by rounding the AF profiles to binary values. “AF soft” denotes that the input data were the AF profiles represented by decimal values. In the SV task, Table 5 shows the equal error rate (EER) for SV. For x-vector and stats-vector, the AF soft method performed better than the AF hard method by 29% and 28% in EER, respectively. Therefore, this experiment showed that AF profiles can be used to characterize speaker-specific information, and using AF soft as input can provide more helpful information to improve the performance of speaker recognition.

In this experiment, distilling the information into different layers was analyzed in frame-level transformation. We used the VoxCeleb1 speaker identification task of 1,251 speakers (see iden_split.txt on the VoxCeleb1 webpage) for audio enrollment and evaluation. In the enrollment, the speaker needed to enroll in the system and the corresponding speaker model was constructed. Through enrollment utterances, the obtained speaker-specific embeddings were averaged to form an average embedding as the speaker model. As shown in Table 6, compared to the baseline x-vector and stats-vector systems without KD, the KD with 3 shared layers for x-vector and stats-vector achieved the best accuracies of 63.68% and 62.84%, respectively. Furthermore, we found that when KD was used at higher layers, the performance of KD-based models was worse than the baseline. Conversely, when KD was used at lower layers, the performance of KD-based models was better than the baseline. Thus, distilling the knowledge to a long temporal context (high layers) may cause the learning confusion and degrade the performance, but using KD at lower layers reduces the confusion and achieves the best performance. Combining multimodal results to improve prediction performance has been widely used in recent years. As KD was used in a speaker embedding model, this experiment focused on investigating whether combining the results of multiple KD-based models can further improve the performance. For feature concatenation, we directly concatenated the speaker embeddings obtained from different models to form a combined embedding. According to the results in Table 6, the KD-based x-vector with 3 shared layers and the KD-based stats-vector with 3 shared layers were selected and compared to the method without KD. We can see that in feature concatenation, our proposed KD model achieved the performance better than the method without KD. Therefore, this experiment showed our proposed KD model performed better than the baseline x-vector, and was suitable to be used in multimodality for performance improvement.

In this paper, a knowledge distillation model was proposed to distill speaker-specific information to make the model learn more robust speaker representation. First, as AF can be used to characterize speaker-specific information, an AF extractor is proposed to obtain AF profiles of the speech signal. We find that the predicted AF profiles can be used as input features for speaker recognition, and the AF profiles can be seen as the speaker-specific features to provide more helpful information. Next, the knowledge distillation method was used to distill the speaker-specific information of the AF profiles from the teacher model to the student model. The experimental results showed that our proposed knowledge distillation model achieved the best performance in speaker recognition, and using the feature concatenation of multimodal results can further improve the performance.

In the future, several problems, listed in the followings, should be considered for further improvement.

• As the AF extractor was trained by the LibriSpeech dataset and tested by the VoxCeleb1 dataset, there is a mismatch between training and testing which will degrade the performance.

• In this paper, model training does not apply data augmentation to improve performance.

Therefore, in the future we will explore the learning ability of deep neural networks to deal with the mismatch problems. And combining another mechanism, such as attention, to obtain more robust speaker embedding and increase the degree of aggregation in clustering.

This work was supported in part by the Ministry of Science and Technology, Taiwan, under Contract No. MOST 111-2221-E-006-150-MY3.

Qian-Bei Hong received the B.S. and M.S. degrees in electrical engineering from Southern Taiwan University of Science and Technology, Tainan, Taiwan, in 2009 and 2011, respectively. He is currently pursuing the Ph.D. degree in the Graduate Program of Multimedia Systems and Intelligent Computing, National Cheng Kung University and Academia Sinica, Tainan, Taiwan. His research interests include multimedia signal processing, deep learning, and speaker recognition.

Chung-Hsien Wu received the B.S. degree in electronics engineering from National Chiao Tung University, Hsinchu, Taiwan, in 1981, and the M.S. and Ph.D. degrees in electrical engineering from National Cheng Kung University (NCKU), Tainan, Taiwan, in 1987 and 1991, respectively. Since 1991, he has been with the Department of Computer Science and Information Engineering, NCKU. He became the Chair Professor in 2017. He served as the Deputy Dean of the College of Electrical Engineering and Computer Science, NCKU, from 2009 to 2015. He also worked at Computer Science and Artificial Intelligence Laboratory of Massachusetts Institute of Technology (MIT), Cambridge, MA, USA, in summer 2003, as a Visiting Scientist. He was the Associate Editor of IEEE Transactions on Audio, Speech and Language Processing (2010–2014), IEEE Transactions on Affective Computing (2010–2014), ACM Transactions on Asian and Low-Resource Language Information Processing, and APSIPA Transactions on Signal and Information Processing (2014~2020). He was the APSIPA BoG Member in 2019~2021. He received 2018 APSIPA Sadaoki Furui Prize Paper Award in 2018, and the Outstanding Research Award of Ministry of Science and Technology, Taiwan, in 2010 and 2016. His research interests include deep learning, affective computing, speech recognition/synthesis, and spoken language processing.

Hsin-Min Wang received the B.S. and Ph.D. degrees in electrical engineering from National Taiwan University, Taipei, Taiwan, in 1989 and 1995, respectively. In October 1995, he joined the Institute of Information Science, Academia Sinica, Taipei, Taiwan, where he is currently a Research Fellow. He also holds a joint appointment as a Professor with the Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, Taiwan. His major research interests include spoken language processing, natural language processing, multimedia information retrieval, machine learning and pattern recognition. He was an Associate Editor for IEEE/ACM Transactions on Audio, Speech and Language Processing from 2016 to 2020. He is currently on the Editorial Board Member of APSIPA Transactions on Signal and Information Processing. He was the General Co-Chair of ISCSLP2016 and ISCSLP2018 and a Technical Co-Chair of ISCSLP2010, O-COCOSDA2011, APSIPAASC2013, ISMIR2014, and ASRU2019. He was the recipient of the Chinese Institute of Engineers Technical Paper Award in 1995, and the ACM Multimedia Grand Challenge First Prize in 2012. He was an APSIPA Distinguished Lecturer for 2014–2015. He is a Member of the International Speech Communication Association and ACM.