Time-Aware Multi-Scale RNNs for Time Series Modeling

1 Time-Aware Multi-Scale RNNs for time Series ModelingZipeng Chen1,Qianli Ma1;2 andZhenxi Lin11 School of Computer Science and Engineering,South China University of Technology, Guangzhou, China2 Key Laboratory of Big Data and Intelligent Robot(South China University of Technology), Ministry of information is crucial for modelingtime Series . Although most existing methods con-sider multiple scales in the time - Series data, theyassume all kinds of scales are equally importantfor each sample, making them unable to capturethe dynamic temporal patterns of time Series . Tothis end, we propose Time-Aware Multi-Scale Re-current Neural Networks (TAMS-RNNs), whichdisentangle representations of different scales andadaptively select the most important scale for eachsample at each time step. First, the hidden stateof the RNN is disentangled into multiple inde-pendently updated small hidden states, which usedifferent update frequencies to model time -seriesmulti-scale information.

2 Then, at each time step,the temporal context information is used to mod-ulate the features of different scales, selecting themost important time - Series scale. Therefore, theproposed model can capture the Multi-Scale in-formation for each time Series at each time stepadaptively. Extensive experiments demonstrate thatthe model outperforms state-of-the-art methods onmultivariate time Series classification and humanmotion prediction tasks. Furthermore, visualizedanalysis on music genre recognition verifies the ef-fectiveness of the IntroductionTime Series is a set of values sequentially recorded over of time Series can provide meaningful knowledge,which is beneficial for decision-making in a variety of fields,such as human motion prediction[Martinezet al., 2017]andEEG/ECG data analysis[Bagnallet al., 2018]. Multi-Scale information is crucial to the Modeling of timeseries[Mozer, 1992; Koutniket al.]

3 , 2014]. Currently, mostmethods use Multi-Scale convolution[Cuiet al., 2016]or skipconnections to model multiple scales of time Series [Koutniket al., 2014; Changet al., 2017; Chiuet al., 2019; Cartaet al.,2020]. Multi-Scale convolution uses different downsamplingfrequencies or convolution kernels of different sizes, while Qianli Ma is the corresponding 1: Music time Series of two different with skip connections capture multiple scales throughdirect connections spanning different lengths. These methodsuse pre-fixed multiple scales, assuming that all the scales areequally important for each , it is difficult for these fixed-scale methods tocapture the dynamic temporal patterns of time Series . Forexample, Fig. 1 shows two music clips of different genres. Hip-Hop is a style of music with a strong beat, while Ex-perimental is relatively chaotic. Intuitively, Hip-Hop re-quires a larger scale to capture long-term dependencies due toits regularity, while Experimental needs a smaller scale tocapture short-term dependencies due to its sharp , we need to adaptively select a suitable scale for eachsample rather than a fixed one.

4 Meanwhile, the recognitionof genres requires Modeling the emotional changes in music,which are controlled by note duration. Therefore, differentscales are also needed at different time steps as the notes havedifferent durations at different times[Huet al., 2019].Recently, some methods have been proposed to select ap-propriate scales corresponding to each sample dynamically[Neilet al., 2016; Jerniteet al., 2016; Camposet al., 2018;Huet al., 2019]. These methods select a specific scale adap-tively or decide whether to update the hidden state at eachtime step of the Recurrent Neural Network (RNN). However,these methods attempt to use uniform feature representationsto model multiple scales, entangling representations of differ-ent scales. It is hard to explore the impact of different scalesusing the entangled representation, which is harmful to theinterpretability of the model[Tamkinet al., 2020].

5 Hence,disentangling the scales helps learning the representations ofdifferent scales better and is conducive to address the above issues, we propose time -AwareMulti-Scale RNNs (TAMS-RNNs) that can adaptively modelProceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)2285the Multi-Scale information for each time Series at each timestep. First, we design a mechanism called Multi-Scale Fea-ture Disentanglement (MSFD) to obtain decoupled featurerepresentations of different scales. Concretely, the hiddenstate of the RNN is disentangled into multiple independentlyupdated small hidden states, which use different update fre-quencies to model the Multi-Scale , a mechanism called Time-Aware Feature Modulation(TAFM) is designed to modulate the features of differentscales, adaptively selecting the most important scale at eachtime step.

6 We conduct comparative experiments on Multi-variate time Series (MTS) classification, human motion pre-diction, and music genre recognition to verify the superiorityof TAMS-RNNs. The contributions of our work are: We propose Time-Aware Multi-Scale RNNs (TAMS-RNNs) to adaptively model the Multi-Scale informationfor time Series Modeling . It uses a Multi-Scale Fea-ture Disentanglement (MSFD) mechanism to disentan-gle feature representations of different scales. We design a simple but effective mechanism we callTime-Aware Feature Modulation (TAFM) that uses tem-poral context information to modulate the features ofdifferent scales, adaptively selecting the most importantscale for each sample at each time step. We conduct extensive experiments showing that ourmodel outperforms state-of-the-art methods on MTSclassification and human motion prediction tasks. Fur-thermore, we visualize the network s behavior on musicgenre recognition, verifying the model s Related WorkMany methods have been proposed that use RNNs to modelmulti-scale time Series dynamics.

7 They can be roughly di-vided into two categories:Pre-fixed RNN (CW-RNN)[Kout-niket al., 2014]divides one layer of the RNN into separatemodules, each processing inputs at its own temporal granular-ity. Dilated RNN[Changet al., 2017]extracts the representa-tions of multiple scales by stacking multiple layers of RNNsand using skip connections of different lengths. MS-LMN[Cartaet al., 2020]separates the RNN into different moduleswith different sampling rates, using an incremental trainingalgorithm to learn long-term dependencies. However, thesemethods use fixed, predefined scales, making it difficult toadapt to different time Series are also methods that selectsuitable scales dynamically. Phased LSTM[Neilet al., 2016]and Skip RNN[Camposet al., 2018]decide whether to up-date the hidden state at each time step of the RNN to learnrepresentations at multiple scales. VCRNN[Jerniteet al.]

8 ,2016]adaptively determines the number of neurons to be up-dated at each time step. ASRNN[Huet al., 2019]presetsmultiple scales and selects a specific scale at each time , these methods attempt to use uniform feature rep-resentations to model multiple scales, and the information ofvarious scales are entangled together. Hence, it is difficult toexplore the impact of different scales on the most relevant model to ours is CW-RNN[Koutniketal., 2014]. However, in CW-RNN, representations of differ-ent scales are entangled together, and it is difficult for themodel to capture the dynamic temporal patterns of time se- ries . We update the small hidden states independently to learnthe representations of different scales better. Meanwhile, thetemporal context information is used to select the most impor-tant scale at each time step adaptively, capturing more com-plicated temporal patterns. A detailed comparison of theirupdate process to ours is shown in Fig.

9 Time-Aware Multi-Scale RNNsWe propose Time-Aware Multi-Scale RNNs (TAMS-RNNs)and design two mechanisms (named Multi-Scale Feature Dis-entanglement and Time-Aware Feature Modulation, respec-tively) for TAMS-RNNs to adaptively model the multi-scaleinformation for time Series Modeling . The whole architectureof TAMS-RNNs is shown in Fig. Multi-Scale Feature DisentanglementGiven time seriesX=fx1;x2; ;xTgwherext2Rd0,d0is the dimension of input data andTdenotes the lengthof time Series , the hidden stateht2 Rdof RNN cell can beexpressed as follows:ht=f(Wxt+Uht 1+b);(1)wherextandhtare the input and hidden state at time stept, d0,U2Rd d, andb2 Rdarethe learnable the hyperbolic tangentactivation capture the feature representations of different scales,the mechanism of MSFD is designed. First, the hidden stateof RNN is disentangled into multiple small hidden states thatare updated independently. Specifically, the hidden state attime steptis disentangled intoKsmall hidden states asht= [h1t; ;hKt], wherehkt2Rp,p=d=Kand[ ]is theconcatenation operation.

10 The corresponding recurrent matrixis defined aseU= [eU1; ;eUK], whereeUk2Rp p. Eachsmall hidden statehktis independently updated by an individ-ual recurrent matrixeUkand then merged via concatenationto constitute the hidden stateht. Meanwhile, The learnableparameterWis defined asW= [W1; ;WK], whereWk2Rp d0. The update equation ofhtis defined as:hkt=f(Wkxt+eUkhkt 1+eb);(2)ht= [h1t; ;hKt];(3)wherefdenotes the hyperbolic tangent activation functiontanhandeb2 Rpis the learnable parameter. Specifically, themodel is equivalent to RNN whenK= the disentangling process, each small hidden statewould use a specific update frequency to capture informationof a particular scale. Suppose the scale setSisfs1; ;sKg,for simplicity,skis usually set to a power of 2 in our exper-imentation. For scalesk, the small hidden statehktwill beupdated everysktime steps. As shown in Fig. 2, the en-tire hidden state of RNN is disentangled into3small hiddenstates, and the corresponding scale set isf1;2;4g.