A Convolutional Recurrent Neural Network for Real-Time ...

1 Interspeech 2018. 2-6 September 2018, Hyderabad A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement Ke Tan1 , DeLiang Wang1,2. 1. Department of Computer Science and Engineering, The Ohio State University, USA. 2. Center for Cognitive and Brain Sciences, The Ohio State University, USA. Abstract speech [6] [7]. In this study, we use the magnitude spectra of target speech as the training target. Many real-world applications of speech enhancement, such as For supervised speech enhancement, noise generalization hearing aids and cochlear implants, desire Real-Time processing, and speaker generalization are both crucial. A simple yet ef- with no or low latency. In this paper, we propose a novel convo- fective method to deal with noise generalization is to train with lutional Recurrent Network (CRN) to address Real-Time monaural different noise types [8]. Analogously, to address speaker gen- speech enhancement.

2 We incorporate a Convolutional encoder- eralization would include a large number of speakers in a train- decoder (CED) and long short-term memory (LSTM) into the ing set. However, it has been found that a feedforward DNN is CRN architecture, which leads to a causal system that is natu- unable to track a target speaker in the presence of many training rally suitable for Real-Time processing. Moreover, the proposed speakers [9] [10] [11]. Typically, a DNN independently predicts model is noise- and speaker-independent, noise types and a label for each time frame from a small context window around speakers can be different between training and test. Our exper- the frame. An interpretation is that such DNNs cannot lever- iments suggest that the CRN leads to consistently better objec- age long-term contexts, which would be essential for tracking a tive intelligibility and perceptual quality than an existing LSTM target speaker. Recent studies [9] [10] suggest that it would be based model.

3 Moreover, the CRN has much fewer trainable pa- better to formulate speech separation as a sequence-to-sequence rameters. mapping in order to leverage long-term contexts. Index Terms: noise- and speaker-independent speech enhance- With such a formulation, Recurrent Neural networks (RNNs). ment, Real-Time applications, Convolutional encoder-decoder, and Convolutional Neural networks (CNNs) have been used for long short-term memory, Convolutional Recurrent networks noise- and speaker-independent speech enhancement, where noise types and speakers can be different between training and 1. Introduction test. Chen et al. [10] proposed an RNN with four hidden LSTM. layers to deal with speaker generalization of noise-independent Speech separation aims to separate target speech from a back- models. Their experimental results show that the LSTM model ground interference, which may include nonspeech noise, inter- generalizes well to untrained speakers, and substantially out- fering speech and room reverberation [1].

4 Speech enhancement performs a DNN based model in terms of short-time objective refers to the separation of speech and nonspeech noise. It has intelligibility (STOI) [12]. A more recent study [13] developed various real-world applications such as robust automatic speech a gated residual Network (GRN) based on dilated convolution- recognition and mobile speech communication. For many such s. Compared with the LSTM model in [10], the GRN exhibits applications, Real-Time processing is required. In other words, higher parameter efficiency and better generalization capabili- speech enhancement is performed with low computational com- ty for untrained speakers at different SNR levels. On the other plexity, providing near-instantaneous output. hand, the GRN requires a large amount of future information In this study, we focus on monaural (single-microphone) for mask estimation or spectral mapping at each time frame. speech enhancement that can operate in Real-Time applications.

5 Hence, it cannot be used for Real-Time speech enhancement. In digital hearing aids, for example, it has been found that a Motivated by recent works [14] [15] on CRNs, we develop delay as low as 3 milliseconds is noticeable to listeners and a a novel CRN architecture for noise- and speaker-independent delay of longer than 10 milliseconds is objectionable [2]. For speech enhancement in real time. The CRN incorporates a con- such applications, causal speech enhancement systems, where volutional encoder-decoder and long short-term memory. We no future information is allowed, are often required. find that the proposed CRN leads to consistently better objective Inspired by the concept of time-frequency (T-F) masking speech intelligibility and quality than the LSTM model in [10]. in computational auditory scene analysis (CASA) [3], speech Moreover, the CRN has much fewer trainable parameters. separation has been formulated as supervised learning in recent The rest of this paper is organized as follows.

6 We give a years, where a deep Neural Network (DNN) is employed to learn detailed description of our proposed model in Section 2. The a mapping from noisy acoustic features to a T-F mask [4]. The experimental setup and results are presented in Section 3. We ideal binary mask, which classifies T-F units as either speech- conclude this paper in Section 4. dominant or noise-dominant, is the first training target used in supervised speech separation. More recent training target- s include the ideal ratio mask [5] and mapping-based target- 2. System description s corresponding to the magnitude or power spectra of target Encoder-decoder with causal convolutions This research was supported in part by an NIDCD grant (R01 D- Badrinarayanan et al. first proposed a Convolutional encoder- C012048) and the Ohio Supercomputer Center. decoder Network for pixel-wise image labelling [16]. It com- 3229 Output Note that the input can be treated as a sequence of feature vec- tors, while only the time dimension is illustrated in Fig.

7 1. In Hidden Layer causal convolutions, the output does not depend on future input- s. With causal convolutions instead of noncausal convolutions, Hidden Layer the encoder-decoder architecture leads to a causal system. Note that we can easily apply causal deconvolutions to the decoder, Hidden Layer since the deconvolution is intrinsically a convolution operation. Input Temporal modeling via LSTM. Past Future Time In order to track a target speaker, it may be important to leverage long-term contexts, which cannot be utilized by the aforemen- Figure 1: An example of causal convolutions. The convolution tioned Convolutional encoder-decoder. The LSTM [21], a spe- output does not depend on future inputs. cific type of RNN which incorporates a memory cell, has been successful in temporal modeling in various applications such as acoustic modeling and video classification. To account for tem- prises a Convolutional encoder followed by a corresponding poral dynamics of speech, we insert two stacked LSTM layers decoder which feeds into a softmax classification layer.

8 The between the encoder and the decoder. In this study, we use the encoder is a stack of Convolutional layers and pooling layers, LSTM defined by the following equations: which serves to extract high-level features from a raw input im- age. With essentially the same structure as the encoder in the it = (Wii xt + bii + Whi ht 1 + bhi ) (1). reverse order, the decoder maps low-resolution feature maps at ft = (Wif xt + bif + Whf ht 1 + bhf ) (2). the output of the encoder to feature maps of the full input image gt = tanh(Wig xt + big + Whg ht 1 + bhg ) (3). size. The symmetric encoder-decoder architecture ensures that the output has the same shape as the input. With such an at- ot = (Wio xt + bio + Who ht 1 + bho ) (4). tractive property, the encoder-decoder architecture is naturally ct = ft ct 1 + it gt (5). suitable for any pixel-wise dense prediction task, which aims to ht = ot tanh(ct ) (6). predict a label for each pixel in the input image.

9 For speech enhancement, one approach is to employ a CED where xt , gt , ct and ht represent input, block input, memory to map from the magnitude spectrogram of noisy speech to that cell and hidden activation at time t, respectively. W 's and b's of clean speech, where the magnitude spectrograms are sim- denote weights and biases, respectively. represents sigmoid ply treated as images. To our knowledge, Park et al. [17] first nonlinearity and represents element-wise multiplication. introduced CED for speech enhancement. They proposed a re- To fit the input shape required by the LSTM, we flatten the dundant CED Network (R-CED), which consists of repetition- frequency dimension and the depth dimension of the encoder s of a convolution, batch normalization (BN) [18], and a Re- output to produce a sequence of feature vectors before feeding LU activation [19] layer. The R-CED architecture additionally it into the LSTM layers. The output sequence of the LSTM.

10 Incorporates skip connections to facilitate optimization, which layers is subsequently reshaped back to fit the decoder. It is connect each layer in the encoder to its corresponding layer in worth noting that the inclusion of the LSTM layers does not the decoder. change the causality of the system. In our proposed Network , the encoder comprises five convo- lutional layers while the decoder has five deconvolutional layer- Network architecture s. We apply exponential linear units (ELUs) [20] to all convolu- tional and deconvolutional layers except the output layer. ELUs Table 1: Architecture of our proposed CRN. Here T denotes the have been demonstrated to lead to faster convergence and bet- number of time frames in the STFT magnitude spectrum. ter generalization than ReLUs. In the output layer, we utilize softplus activation [19] which is a smooth approximation to the layer name input size hyperparameters output size ReLU function and can constrain the Network output to always reshape 1 T 161 - 1 T 161.