Heterogeneous Convolutional Recurrent Neural Network with Attention Mechanism and Feature Aggregation for Voice Activity Detection

In recent years, the application of deep learning (DL) in voice activity detection (VAD) has achieved remarkable performance in various downstream tasks. These DL-based VAD methods have demonstrated promising results, outperforming traditional algorithms in numerous applications, such as Renevey and Drygajlo [18], Ramirez et al. [17], Sohn et al. [20], and Tan et al. [21]. Due to its scalability, the proposed method enables us to effectively handle large-scale data, exhibiting superior generalization performance compared to previous machine learning techniques. VAD is also motivated by this trend to incorporate hybrid acoustic features as input to neural networks in Drugman et al. [5] and Meier and Kellermann [15], or by modeling contextual information using popular neural architectures such as Deep Neural Networks (DNN) in Zhang and Wang [29], convolutional neural network (CNN) in Thomas et al. [22], Zazo Candil et al. [28], and Obuchi [16], recurrent neural network (RNN) in Gelly and Gauvain [7], Eyben et al. [6], and Hughes and Mierle [9], and convolutional recurrent neural network (CRNN) in Dinkel et al. [4].

The extraction of salient features in context using convolutional neural networks (CNNs) or deep neural networks (DNNs) has been demonstrated to be effective. Nevertheless, the inherent input length constraint of these models limits their ability to process long messages, which can be crucial for VAD tasks. Previous attempts to model contextual information using recurrent neural networks (RNNs) have not been promising. In this regard, the attention mechanism in Kim and Hahn [12], Jo et al. [11], and Sofer and Chazan [19] demonstrates practical high performance in various context-involving settings. ACAM in Kim and Hahn [12] uses LSTM and attention mechanism to generate VAD, which is essential for capturing the correlation between the hidden vectors of the encoder and decoder. In Jo et al. [11], VAD is based on the Self Attention (SA) framework. The idea behind this architecture is to model contextual information between acoustic input frames. Unfortunately, the SA mechanism is only used for short context frames. Also, without using a CNN on the input, the spectral information of the speech may not be fully exploited. Sofer and Chazan [19] utilizes a Convolutional Neural Network (CNN) to exploit the spatial information of the noisy input spectrum to extract a sequence of frame-wise embeddings, followed by a self-attention encoder designed to find contextual information from the embedding sequence. Unlike Jo et al. [11] which is used separately for each frame (with context frames), this method is able to process the entire waveform at once, allowing for long receptive fields.

In this study, we present a voice activity detection (VAD) system that combines CRNN modules with an attention mechanism and feature aggregation. First, the CRNN module is designed to capture frame-wise features and extend the effective receptive field. Second, the attention mechanism focuses on essential features while suppressing irrelevant ones. To this end, we introduce two novel attention mechanisms. On one hand, we employ a method that emphasizes meaningful features along the channel and spatial axes. This approach leverages the fact that convolution operations integrate cross-channel and spatial information to refine feature representation. On the other hand, we address less context-sensitive issues using the multi-head self-attention (MHSA) module. Finally, refined features from backend LSTMs or GRUs are aggregated based on trainable scalar weights or vector-based attention weights across different scales. Our results demonstrate that the combined architecture outperforms individual components. Furthermore, we demonstrate that the proposed method not only achieves state-of-the-art performance on various benchmarks but is also computationally efficient, processing the entire waveform in a single step.

The VAD systems proposed in this paper take an acoustic feature vector extracted from the audio as an input, and should produce frame-level class labels for two categories: non-speech (ns) and speech (s). We implemented the first six different architectures to built VAD systems, as illustrated by Figure 1. The last two diverse methods are built in Figure 2. Table 1 shows different configurations for eight algorithms.

To mimic the nonlinear response to the human ear’s acoustic spectrum, we choose a logarithmic mel-scale filter bank energy (fbank). In order to prepare for input into the neural network, the adjacent 11 acoustic feature frames are concatenated as input to the convolution operation.

Our first approach to implementing a VAD system utilizes a convolutional recurrent neural network (CRNN) backend. The architecture consists of a two-layer CNN (utilizing 5 × 5 and 3 × 3 convolutions), each followed by 2 × 2 max pooling. A single layer of long short-term memory (LSTM) is appended after the last CNN output, enhancing the temporal consistency of our model. The final two layer is fully-connected layers, which outputs whether speech exists in each frame. The framework and specific parameters can be seen in Figure 1a.

The second method is depicted as Figure 1b. Compared with Figure 1a, The operation of batch normalization is excluded. A single layer long short term memory (LSTM) will be replaced with a two-layer long short term memory (LSTM). Others remain unchanged. We also tried larger networks but saw no performance gain, possibly due to the limited diversity of the training data.

As shown in Figure 1c, a channel attention block (ca-block) is inserted after the single-layer long short term memory (LSTM). The idea of channel-wise attention is borrowed from SENets in Hu et al. [8]. Specifically, the ca-block in Figure 3 consists of two operations: squeeze and excitation. The squeeze operation employs global average pooling of feature maps to generate channel statistics. Formally, the statistics z ∈ ℝ^C are generated by shrinking feature maps U = [u_ı, u₂, ·, u_C] through its spatial dimensions H×W, so that the c-th element of z is calculated by:

Therefore, C×H×W feature maps are reduced to C×1×1 channel-wise statistics where H is the input height, W is the input width, and C is the number of channels. The excitation operation then takes the channel-wise statistics as inputs and computes the scaled values with a range of [0,1] through two fully-connected (FC) layers forming a bottleneck. A simple gating mechanism with sigmoid activation is formulated as follows:

where σ refers to the sigmoid function, δ means the ReLU function, $W1∈ℝCr×C,W2∈ℝC×Cr$ and r is a reduction ratio. The final output of the ca-block is obtained by rescaling U with the activations s:

where X = [x₁, x₂, ·, x_C], s = [s₁, s₂, ·, s_C], and ⊗ refers to channel-wise multiplication.

A spatial attention block (sa-block) can be seen in Figure 1d. According to Woo et al. [26], we exploit the inter-spatial relationships of features to generate spatial attention maps. To compute the spatial attention, we first apply average-pooling and max-pooling operations along the channel axis and concatenate them to generate an efficient feature descriptor. On the concatenated feature descriptor, we apply a convolution layer to generate a spatial attention map M(U) ∈ ℝ^H×W which encodes where to emphasize or suppress.

We aggregate channel information of a feature map by using two pooling operations, generating two 2D maps: U^avg ∈ ℝ^1×H×W and U^max ∈ ℝ^1×H×W. Each denotes average-pooled features and max-pooled features across the channel. Those are then concatenated and convolved by a standard convolution layer, producing our 2D spatial attention map. In short, the spatial attention is computed as:

where δ is the sigmoid function and ƒ^3×3 represents a convolution operation with the filter size of 3 × 3. The spatial attention process can be summarized as:

where Y is the final refined output. This detailed operation is described in Figure 4.

Figure 1e shows a hybrid attention block (ha-block) applied. The hybrid attention module is bred from the ca-block and sa-block. Given an intermediate feature map U ∈ ℝ^C×H×W as input, the ha-block infers a 1D channel attention map s ∈ ℝ^C×1×1 and a 2D spatial attention map M ∈ ℝ^1×H×W. The overall attention process can be summarized as:

where $X^$ is the final refined output. In our hybrid attention block (Figure 5), we exploit both spatial and channel-wise attention based on an efficient architecture and empirically verify that exploiting both is superior to using only the channel-wise attention as Hu et al. [8].

A feature aggregation module is shown in Figure 1f. The feature aggregation module is based on scalar weights and aims to capture more information for important estimated features. Given a feature matrix, this method assigns weights to each element of the matrix through a trainable layer. The matrix is a parameter which can be learned by model optimization, where i is the feature dimension index, n is the total dimension, j is the RNN category index, and m is the total number of RNN categories used. The RNN categories are GRU and LSTM. m is set to 2 here. The associated scalar weights for each element of a feature row can be defined as follows:

where t is the softmax activation function calculated as follows:

Given the output feature map f^lstm∈ ℝ^c×H×w from ha-block after the LSTM branch and the output feature map f^gru∈ ℝ^c×H×w from ha-block after the GRU branch, the two feature maps are reshaped to 1D feature vectors $f^lstm=[f^1lstm,f^2lstm,⋯,f^ilstm,⋯,f^nlstm]$ and $f^gru=[f^1gru,f^2gru,⋯,f^igru,⋯,f^ngru]$⁠, respectively. Here, n = C × H × W is the feature dimension. It is also the hidden state number in LSTM or GRU. The feature vectors are aggregated by the following equations:

The weighted feature vector is $f^=[f^1,f^2,⋯,f^i,⋯,f^n]$⁠. Then, we feed them into a LayerNorm layer.

A multi-head self-attention block (mhsa-block) can be described in Figure 2a. The mhsa-block in Vaswani et al. [24] is the core of prominent architectures in multiple Machine Learning domains such as Natural Language Processing (NLP) and Computer Vision (CV). The main goals of the mhsa encoder is mapping a query and a set of key-value pairs to an output. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed using the given query with the corresponding key. An intermediate feature vector v_t is converted into an embedding vector e_t to be fed as an input of the self-attention network. In this process, the corresponding embeddings provided to the model are computed as follows:

where t is the desired position in an input sequence, and the positional embedding p_t as a vector containing pairs of sines and cosines for each frequency. It is defined as:

where $ωk=1100002k/d$ and d = 160 is the encoding dimension. Given the frame-wise embedding sequence E = [e₁, e₂, ·, e_t, e_L] ∈ ℝ^d×L where L is the input sequence length, a single layer attention of the multi-head attention with H = 4 attention heads is computed as:

where

The terms $(WiQ,WiK,WiV,WO)$ are the learned parameter sets of the multi-head attention layer. Then, we feed them into a LayerNorm layer.

Figure 2b depicts a feature aggregation module using vector-based attention weights methods. The vector-based attention was proposed in Chen et al. [3] for sentence embedding and in Wu et al. [27] for text-independent speaker verification. Such attention provides a vectorial attention weight for each feature sequence estimated by algorithm, collecting more discriminative information.

Here, we ignore the frame index t. Given the output feature vector f^lstm∈ ℝⁿ from mhsa-block after the LSTM branch and the output feature vector f^gru∈ ℝⁿ from mhsa-block after the GRU branch, the two feature vectors are $flstm=[f1lstm,f2lstm,⋯,filstm,⋯,fnlstm]$ and $fgru=[f1gru,f2gru,⋯,figru,⋯,fngru]$⁠, respectively. The input vector of vector-based attention methods is defined as $hi=[filstmfigru]$⁠. The corresponding attention weight a_i= [ω_i1, …,ω_ij,…, ω_im] for each element of the vector can be computed as:

where W₁ ∈ ℝ^d×2 and W₂ ∈ ℝ^2×d are weight matrices, b₁ ∈ ℝ^d and b₂ ∈ ℝ² are bias items, d =16 is a hyper-parameter, m = 2 is the total number of used RNN categories, ƒ(·) is a non-linear activation function, e.g. ReLU. The softmax function ensures that the sum of all elements is 1 in the weight vector a_i. Each element ω_ij of the vector a_i is the attention weight for the element of the h_i. The final enhanced feature vector is formulated as follows:

where $fi=ωi1filstm+ωi2figru$⁠. Then, we feed them into a LayerNorm layer.

To evaluate the effectiveness of the proposed methods, we resort to the AISHELL-1 corpus set by Bu et al. [1]. We choose it because it is open and available. 340 speakers’ data is employed for training (120098 utterances, 340 hours). Test set contains 7176 utterances from 20 speakers. The corpus is sampled at 16 kHz with 16 bit quantization. For all the datasets, to produce the frame-level ground truth personal VAD labels used in training and evaluation, we run forced alignment with a pretrained speech recognition model.

To generate noisy speech for evaluating the robustness of the proposed algorithm, white noise, babble noise, factory1 noise, factory2 noise, and pink noise from Noisex-92 in Varga and Steeneken [23] are added to the clean speech. a total of 25 different conditions are considered: 5 types of noise with 5 signal-to-noise ratio (SNR) each (-10, -5, 0, 5, and 10 dB). To make the train set, we randomly select 30000 utterances from the AISHELL-1 training set. There is a noisy condition of random selection for each utterance in training. To make the test set, we use the whole AISHELL-1 testing set (7176 utterances) and consider that the test incorporates the type of noise (f16 and leopard) that is absent in the train set.

Kaggle VAD datasets (kaggle-speech) in lazyrac00n [14] collect 719 audio from three different databases (TIMIT, PTDB-TUG and Noizeus). one of those creates an annotation file which write, in a txt format, where the silent and sounding intervals are in the signal. Noise signal (from AURORA database) is artificially added to the speech signal, in particular the database contains audio corrupted with babble (crowd of preople), street, train, train station, car and restaurant noise at SNRs of 5dB, and the original ones. To make the train set, we randomly select 519 utterances. To make the test set, we select the remaining 200 utterances.

AVA-Speech is a publicly available dataset of movies densely labeled with speech activity in Chaudhuri et al. [2]. At time of writing, AVA-Speech consisted of 160 segments from movies hosted on YouTube, each 15 minutes in duration, totalling 40 hours of labelled data. The segments are densely labelled for speech activity using the following mutually exclusive labels: “NoSpeech”, “CleanSpeech”, “Speech+Music” and “Speech+Noise”. We select 39 segments from it and make 3988 utterances ranging from 3s to 15s. To make the train set, we randomly select 3188 utterances. To make the test set, we select the remaining 800 utterances.

In order to train the models, 6 epochs were utilized for the synthetic VAD datasets, while 100 epochs were allocated for both the Kaggle VAD datasets and AVA-speech datasets. The model is trained with Adam optimizer in Kingma and Ba [13] and a learning rate of 0.00005, using an effective batch size of 16. We use a LSTM or GRU network with 160 cells, followed by a fully-connected layer with 160 neurons.

To evaluate the performance of the proposed approach, we calculated the Average Precision (AP) in Zhu [30] for each class and the mean Average Precision (mAP) over all the classes.

With an input frame x and the corresponding ground truth label y ∈ {ns, s}, The VAD task can be thought of as a binary classification problem. The network outputs the unnormalized distribution of x over the two classes, denoted as z = ƒ_VAD(x). We use z^k to denote the unnormalized probability of the k-th class. To train the model, we minimize the cross entropy loss as:

where k ∈ {ns, s}.

Results of the first six different methods for synthetic VAD datasets are listed in Table 2. It suggest that each of the components contribute to overall performance, with hybrid attention mechanism and feature aggregation based on scalar weights yielding the largest performance boost. The proposed model in Figure 1f shows decent performance for white and babble noises, which are present in the train set. A similar result for comparably f16 scenario, which is absent in the train set, displays the robustness of the proposed algorithm. The proposed method in Figure 1f outperforms the other methods for all the noisy conditions, except the leopard.

This states the benefit of this approach. It is further evident that our model, which combines a CRNN embedder before the hybrid attention encoder and applies the feature aggregation based on scalar weights, gains even better results. The parameter of the proposed model is 389K which is much smaller than 440K of the algorithm with two-layer LSTM structures in Figure 1b, with better performance (Table 2). This suggests that the proposed model is robust enough to maintain the performance in a harsh environment while lessening the parameter size.

In Table 3, the results for synthetic VAD datasets prove the strength of the proposed model, which merges a CRNN embedder before the mhsa block and employs the feature aggregation based on vector-based attention weights, with similar or better performance. We show that the fusion of mhsa-block and vector-based attention weights outperforms methods based solely on mhsa-block. Figure 6 further confirms this conclusion.

Figure 7 shows the receiver operating characteristic (ROC) curves for all systems tested on kaggle-speech. We see that the Figure 2b systems we propose outperform the CNN-biLSTM VAD in Wilkinson and Niesler [25] and the MARBLENET VAD in Jia et al. [10] across all operating points. When threshold=0.5, the CNN-biLSTM VAD, the MARBLENET VAD, and the Figure 2b method obtain accuracy rates of 91.49%, 91.56% and 90.99%, respectively. The feature scales of the CNN-biLSTM VAD and the Figure 2b method are the same. Figure 8 shows the receiver operating characteristic (ROC) curves for all systems tested on ava-speech. We also see that the Figure 2b systems we propose outperform the CNN-biLSTM VAD and the MARBLENET VAD across all operating points. When threshold=0.5, the CNN-biLSTM VAD, the MARBLENET VAD, and the Figure 2b method obtain accuracy rates of 86.03%, 84.21% and 86.65%, respectively.

In this paper, we propose a heterogeneous CRNN network framework with attention mechanism and feature aggregation for voice activity detection, which introduces hybrid attention block (ha-block) or multi-head self-attention (mhsa) block into the rear of LSTM layer in the network architecture. The attention module can improve representation power of CRNN networks. In the ha-block, we apply attention-based feature refinement with two distinctive modules, channel and spatial, and achieve considerable performance improvement while keeping the overhead small. In the mhsa-block, it is applied on the frame-wise embedding sequence to gain contextual information. Finally, a novel feature aggregation method has been implemented based on scalar weights or vector-based attention weights. It can extract more robust embeddings from the CRNN network structure.

We evaluate the proposed method with the baseline CRNN system. Experiments conducted on synthetic VAD datasets, kaggle VAD datasets and AVA-speech datasets. The mean Average Precision (mAP) and receiver operating characteristic (ROC) curves demonstrate the effectiveness of the proposed method. The algorithm with mhsa modules and feature aggregation based on vector-based attention weights achieves the best mAP, on average. In the future, we plan to incorporate the new attention mechanism and feature aggregation methods into more CRNN-based network architectures and evaluate the effect of different configurations.