Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude ...

1 1 Conv-TasNet: Surpassing Ideal time -FrequencyMagnitude Masking for Speech SeparationYi Luo, Nima MesgaraniAbstract Single-channel, speaker-independent speech sepa-ration methods have recently seen great progress. However,the accuracy, latency, and computational cost of such methodsremain insufficient. The majority of the previous methods haveformulated the separation problem through the time -frequencyrepresentation of the mixed signal, which has several drawbacks,including the decoupling of the phase and Magnitude of the signal,the suboptimality of Time-Frequency representation for speechseparation, and the long latency in calculating the address these shortcomings, we propose a fully-convolutionaltime-domain audio separation network (Conv-TasNet)

2 , a deeplearning framework for end-to-end time -domain speech separa-tion. Conv-TasNet uses a linear encoder to generate a representa-tion of the speech waveform optimized for separating individualspeakers. Speaker separation is achieved by applying a set ofweighting functions (masks) to the encoder output. The modifiedencoder representations are then inverted back to the waveformsusing a linear decoder. The masks are found using a temporalconvolutional network (TCN) consisting of stacked 1-D dilatedconvolutional blocks, which allows the network to model thelong-term dependencies of the speech signal while maintaininga small model size.

3 The proposed Conv-TasNet system signifi-cantly outperforms previous Time-Frequency masking methods inseparating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several Ideal Time-Frequency Magnitude masksin two-speaker speech separation as evaluated by both objectivedistortion measures and subjective quality assessment by humanlisteners. Finally, Conv-TasNet has a significantly smaller modelsize and a shorter minimum latency, making it a suitable solutionfor both offline and real- time speech separation applications. Thisstudy therefore represents a major step toward the realizationof speech separation systems for real-world speech Terms Source separation, single-channel, time -domain,deep learning, real-timeI.

4 INTRODUCTIONR obust speech processing in real-world acoustic environ-ments often requires automatic speech separation. Because ofthe importance of this research topic for speech processingtechnologies, numerous methods have been proposed for solv-ing this problem. However, the accuracy of speech separation,particularly for new speakers, remains previous speech separation approaches have beenformulated in the Time-Frequency (T-F, or spectrogram) rep-resentation of the mixture signal, which is estimated from thewaveform using the short- time Fourier transform (STFT) [1].

5 Speech separation methods in the T-F domain aim to approx-imate the clean spectrogram of the individual sources fromthe mixture spectrogram. This process can be performed bydirectly approximating the spectrogram representation of eachsource from the mixture using nonlinear regression techniques,where the clean source spectrograms are used as the trainingtarget [2] [4]. Alternatively, a weighting function (mask) canbe estimated for each source to multiply each T-F bin inthe mixture spectrogram to recover the individual sources.

6 Inrecent years, deep learning has greatly advanced the perfor-mance of Time-Frequency masking methods by increasing theaccuracy of the mask estimation [5] [12]. In both the directmethod and the mask estimation method, the waveform ofeach source is calculated using the inverse short- time Fouriertransform (iSTFT) of the estimated Magnitude spectrogram ofeach source together with either the original or the modifiedphase of the mixture Time-Frequency masking remains the most commonlyused method for speech separation, this method has severalshortcomings.

7 First, STFT is a generic signal transformationthat is not necessarily optimal for speech separation. Second,accurate reconstruction of the phase of the clean sources isa nontrivial problem, and the erroneous estimation of thephase introduces an upper bound on the accuracy of thereconstructed audio. This issue is evident by the imperfectreconstruction accuracy of the sources even when the idealclean Magnitude spectrograms are applied to the methods for phase reconstruction can be applied toalleviate this issue [11], [13], [14], the performance of themethod remains suboptimal.

8 Third, successful separation fromthe Time-Frequency representation requires a high-resolutionfrequency decomposition of the mixture signal, which requiresa long temporal window for the calculation of STFT. Thisrequirement increases the minimum latency of the system,which limits its applicability in real- time , low-latency appli-cations such as in telecommunication and hearable example, the window length of STFT in most speechseparation systems is at least 32 ms [5], [7], [8] and is evengreater in music separation applications, which require an evenhigher resolution spectrogram (higher than 90 ms) [15], [16].

9 Because these issues arise from formulating the separationproblem in the Time-Frequency domain, a logical approach isto avoid decoupling the Magnitude and the phase of the soundby directly formulating the separation in the time studies have explored the feasibility of time -domainspeech separation through methods such as independent com-ponent analysis (ICA) [17] and time -domain non-negativematrix factorization (NMF) [18]. However, the performance ofthese systems has not been comparable with the performanceof Time-Frequency approaches, particularly in terms of theirability to scale and generalize to large data.

10 On the otherhand, a few recent studies have explored deep learning fortime-domain audio separation [19] [21]. The shared idea in [ ] 15 May 20192these systems is to replace the STFT step for feature extractionwith a data-driven representation that is jointly optimized withan end-to-end training paradigm. These representations andtheir inverse transforms can be explicitly designed to replaceSTFT and iSTFT. Alternatively, feature extraction togetherwith separation can be implicitly incorporated into the networkarchitecture, for example by using an end-to-end convolutionalneural network (CNN) [22], [23].