IEEE SIGNAL PROCESSING LETTERS, ACCEPTED NOVEMBER …

1 IEEE SIGNAL PROCESSING LETTERS, ACCEPTED NOVEMBER 20161 Deep Convolutional Neural Networks and DataAugmentation for environmental SoundClassificationJustin Salamon and Juan Pablo BelloAbstract The ability of deep convolutional neural networks(CNN) to learn discriminative spectro-temporal patterns makesthem well suited to environmental sound classification. However,the relative scarcity of labeled data has impeded the exploitationof this family of high-capacity models. This study has twoprimary contributions: first, we propose a deep convolutionalneural network architecture for environmental sound classifica-tion. Second, we propose the use of audio data augmentation forovercoming the problem of data scarcity and explore the influenceof different augmentations on the performance of the proposedCNN architecture. Combined with data augmentation , the pro-posed model produces state-of-the-art results for environmentalsound classification.

2 We show that the improved performancestems from the combination of a deep, high-capacity model andan augmented training set: this combination outperforms boththe proposed CNN without augmentation and a shallow dic-tionary learning model with augmentation . Finally, we examinethe influence of each augmentation on the model s classificationaccuracy for each class, and observe that the accuracy for eachclass is influenced differently by each augmentation , suggestingthat the performance of the model could be improved further byapplying class-conditional data Terms environmental sound classification, deep convo-lutional neural networks, deep learning, urban sound INTRODUCTIONTHE problem of automatic environmental sound classifi-cation has received increasing attention from the researchcommunity in recent years. Its applications range from contextaware computing [1] and surveillance [2] to noise mitigationenabled by smart acoustic sensor networks [3].

3 To date, a variety of SIGNAL PROCESSING and machine learningtechniques have been applied to the problem, including matrixfactorization [4] [6], dictionary learning [7], [8], wavelet fil-terbanks [8], [9] and most recently deep neural networks [10],[11]. See [12] [14] for further reviews of existing particular, deep convolutional neural networks (CNN) [15]are, in principle, very well suited to the problem of environ-mental sound classification: first, they are capable of capturingenergy modulation patterns across time and frequency whenapplied to spectrogram-like inputs, which has been shownto be an important trait for distinguishing between different,often noise-like, sounds such as engines and jackhammers [8].Second, by using convolutional kernels (filters) with a smallJ. Salamon is with the Music and Audio Re-search Laboratory (MARL) and the Center for Urban Science and Progress(CUSP) at New York University, USA.

4 J. P. Bello is withthe Music and Audio Research Laboratory at New York University, field, the network should, in principle, be able tosuccessfully learn and later identify spectro-temporal patternsthat are representative of different sound classes even if partof the sound is masked (in time/frequency) by other sources(noise), which is where traditional audio features such as Mel-Frequency Cepstral Coefficients (MFCC) fail [16]. Yet theapplication of CNNs to environmental sound classification hasbeen limited to date. For instance, the CNN proposed in [11]obtained comparable results to those yielded by a dictionarylearning approach [7] (which can be considered an instance of shallow feature learning), but did not improve upon neural networks, which have a high model capacity,are particularly dependent on the availability of large quanti-ties of training data in order to learn a non-linear functionfrom input to output that generalizes well and yields highclassification accuracy on unseen data.

5 A possible explanationfor the limited exploration of CNNs and the difficulty toimprove on simpler models is the relative scarcity of labeleddata for environmental sound classification. While several newdatasets have been released in recent years ( , [17] [19]),they are still considerably smaller than the datasets availablefor research on, for example, image classification [20].An elegant solution to this problem isdata augmentation ,that is, the application of one or more deformations to acollection of annotated training samples which result in new,additional training data [20] [22]. A key concept of data aug-mentation is that the deformations applied to the labeled datado not change the semantic meaning of the labels. Taking anexample from computer vision, a rotated, translated, mirroredor scaled image of a car would still be a coherent image of acar, and thus it is possible to apply these deformations to pro-duce additional training data while maintaining the semanticvalidity of the label.

6 By training the network on the additionaldeformed data, the hope is that the network becomes invariantto these deformations and generalizes better to unseen deformations have also been proposedfor the audio domain, and have been shown to increase modelaccuracy for music classification tasks [22]. However, in thecase of environmental sound classification the application ofdata augmentation has been relatively limited ( , [11], [23]),with the author of [11] (which used random combinations oftime shifting, pitch shifting and time stretching for dataaugmentation) reporting that simple augmentation techniquesproved to be unsatisfactory for the UrbanSound8K datasetgiven the considerable increase in training time they generatedand negligible impact on model accuracy . [ ] 28 Nov 2016 IEEE SIGNAL PROCESSING LETTERS, ACCEPTED NOVEMBER 20162In this paper we present a deep convolutional neural networkarchitecture with localized (small) kernels for environmentalsound classification.

7 Furthermore, we propose the use of dataaugmentation to overcome the problem of data scarcity and ex-plore different types of audio deformations and their influenceon the model s performance. We show that the proposed CNNarchitecture, in combination with audio data augmentation ,yields state-of-the-art performance for environmental METHODA. Deep Convolutional Neural NetworkThe deep convolutional neural network (CNN) architectureproposed in this study is comprised of 3 convolutional layersinterleaved with 2 pooling operations, followed by 2 fully con-nected (dense) layers. Similar to previously proposed featurelearning approaches applied to environmental sound classifi-cation ( , [7]), the input to the network consists of time-frequency patches (TF-patches) taken from the log-scaled mel-spectrogram representation of the audio SIGNAL . Specifically,we use Essentia [24] to extract log-scaled mel-spectrogramswith 128 components (bands) covering the audible frequencyrange (0-22050 Hz), using a window size of 23 ms (1024samples at kHz) and a hop size of the same the excerpts in our evaluation dataset (described below)are of varying duration (up to 4 s), we fix the size of the inputTF-patchXto 3 seconds (128 frames), R128 128.

8 TF-patches are extracted randomly (in time) from the full log-mel-spectrogram of each audio excerpt during training as describedfurther our inputX, the network is trained to learn theparameters of a composite nonlinear functionF( | )whichmapsXto the output (prediction)Z:Z=F(X| ) =fL( f2(f1(X| 1)| 2)| L),(1)where each operationf`( | `)is referred to as alayerof thenetwork, withL= 5layers in our proposed architecture. Thefirst three layers,` {1,2,3}, are convolutional, expressedas:Z`=f`(X`| `) =h(W X`+b), l= [W,b](2)whereX`is a 3-dimensional input tensor consisting ofNfeature maps,Wis a collection ofM3-dimensional kernels(also referred to as filters), represents a valid convolution,bis a vector bias term, andh( )is a point-wise activationfunction. Thus, the shapes ofX`,W, andZ`are(N,d0,d1),(M,N,m0,m1)and(M,d0 m0+1,d1 m1+1) that for the first layer of our networkd0=d1= 128, , the dimensions of the input TF-patch.

9 We apply stridedmax-pooling after the first two convolutional layers` {1,2}using a stride size equal to the pooling dimensions (providedbelow), which reduces the dimensions of the output featuremaps and consequently speeds up training and builds somescale invariance into the network. The final two layers,` {4,5}, are fully-connected (dense) and consist of a matrixproduct rather than a convolution:Z`=f`(X`| `) =h(WX`+b), `= [W,b](3)whereX`is flattened to a column vector of lengthN,Whas shape(M,N),bis a vector of lengthMandh( )is apoint-wise activation proposed CNN architecture is parameterized as follows: `1: 24 filters with a receptive field of (5,5), ,Whas theshape (24,1,5,5). This is followed by (4,2) strided max-pooling over the last two dimensions (time and frequencyrespectively) and a rectified linear unit (ReLU) activationfunctionh(x) = max(x,0). `2: 48 filters with a receptive field of (5,5), ,Whasthe shape (48, 24, 5, 5).

10 Like`1, this is followed by (4,2)strided max-pooling and a ReLU activation function. `3: 48 filters with a receptive field of (5,5), ,Whasthe shape (48, 48, 5, 5). This is followed by a ReLUactivation function (no pooling). `4: 64 hidden units, ,Whas the shape (2400, 64),followed by a ReLU activation function. `5: 10 output units, ,Whas the shape (64,10),followed by a softmax activation that our use of a small receptive field(5,5)in`1compared to the input dimensions(128,128)is designed toallow the network to learn small, localized patterns that canbe fused at subsequent layers to gather evidence in supportof larger time-frequency signatures that are indicative of thepresence/absence of different sound classes, even when thereis spectro-temporal masking by interfering training, the model optimizes cross-entropy loss viamini-batch stochastic gradient descent [25].