AdaRNN: Adaptive Learning and Forecasting for Time Series

1 AdaRNN: Adaptive Learning and Forecasting for Time Series Yuntao Du1, Jindong Wang2, Wenjie Feng3, Sinno Pan4, Tao Qin2, Renjun Xu5, Chongjun Wang11 Nanjing University, Nanjing, China2 Microsoft Research Asia, Beijing, China3 Institute of Data Science, National University of Singapore4 Nanyang Technological University5 Zhejiang time Series Forecasting has a wide range of real-world appli-cations, it is a very challenging task. This is because the statisticalproperties of a time Series can vary with time, causing the distri-bution to change temporally, which is known as the distributionshift problem in the machine Learning community. By far, it stillremains unexplored to model time Series from the distribution-shiftperspective. In this paper, we formulate the Temporal CovariateShift (TCS) problem for the time Series Forecasting . We proposeAdaptive RNNs (AdaRNN) to tackle the TCS composed of two modules. The first module is referredto as Temporal Distribution Characterization, which aims to bettercharacterize the distribution information in a time Series .

2 The sec-ond module is termed as Temporal Distribution Matching, whichaims to reduce the distribution mismatch in the time Series to learnan RNN-based Adaptive time Series prediction ageneral framework with flexible distribution distances on human activity recognition, air quality prediction,household power consumption, and financial analysis show thatAdaRNNoutperforms some state-of-the-art methods of accuracy on classification tasks and terms ofthe mean squared error on regression tasks. We also show thatthe temporal distribution matching module can be extended to theTransformer architecture to further boost its CONCEPTS Computing methodologies Transfer Learning ; Series , multi-task Learning , transfer learningACM Reference Format:Yuntao Du1, Jindong Wang2, Wenjie Feng3, Sinno Pan4, Tao Qin2, RenjunXu5, Chongjun Wang1. : Adaptive Learning and Forecastingfor Time Series . InProceedings of the 30th ACM Int l Conf. on Informationand Knowledge Management (CIKM 21), November 1 5, 2021, Virtual Event, , New York, NY, USA, 11 pages.

3 Corresponding author: J. Wang and W. Feng. This work is done when Y. Du was anintern at to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from 21, November 1 5, 2021, Virtual Event, Australia. 2021 Association for Computing ISBN 978-1-4503-8446-9/21/11.. $ dataProbability distributionB Temporal Covariate Shift:?A Unseen testC Figure 1: Thetemporal covariate shiftproblem in non-stationary time Series . The raw time Series data ( ) is mul-tivariate in reality.

4 At time intervals , , and the unseentest data, the distributions are different: ( ) ( ) ( ) ( ). With the distribution changing over time,how to build an accurate and Adaptive model?1 INTRODUCTIONTime Series (TS) data occur naturally in countless domains includingfinancial analysis [55], medical analysis [29], weather conditionprediction [46], and renewable energy production [5]. Forecastingis one of the most sought-after tasks on analyzing time Series data(arguably the most difficult one as well) due to its importance inindustrial, society and scientific applications. For instance, giventhe historical air quality data of a city for the last five days, how topredict the air quality in the future more accurately?In real applications, it is natural that the statistical propertiesof TS are changing over time, , the non-stationary TS. Overthe years, various research efforts have been made for buildingreliable and accurate models for the non-stationary TS.

5 Tradi-tional approaches such as hidden Markov models (HMMs) [43],dynamic Bayesian networks [35], Kalman filters [9], and otherstatistical models ( ARIMA [19]), have made great progress. Re-cently, better performance is achieved by the recurrent neural net-works (RNNs) [37,46]. RNNs make no assumptions on the temporalstructure and can find highly non-linear and complex dependencerelation in non-stationary property of time Series implies thedata dis-tribution changes over time. Given the example in Figure 1, datadistributions ( )vary for different intervals , , and where , are samples and predictions respectively; Especially for the test datawhich is unseen during training, its distribution is also differentfrom the training data and makes the prediction more conditional distribution ( | ), however, is usually consideredto be unchanged for this scenario, which is reasonable in manyreal applications. For instance, in stock prediction, it is naturalthat the market is fluctuating which changes the financial [ ] 11 Aug 2021 CIKM 21, November 1 5, 2021, Virtual Event, , et al.

6 ( ( )), while the economic laws remain unchanged ( ( | )). It isan inevitable result that the afore-mentioned methods have inferiorperformance and poor generalization [22] since the distributionshift issue violates their basic , it remains unexplored to model the time seriesfrom the distribution perspective. The main challenges of the prob-lem lies in two aspects. First, how to characterize the distributionin the data to maximally harness the common knowledge in thesevaried distributions? Second, how to invent an RNN-based distri-bution matching algorithm to maximally reduce their distributiondivergence while capturing the temporal dependency?In this paper, we formally define theTemporal Covariate Shift(TCS)in Figure 1, which is a more practical and challenging set-ting to model time Series data. Based on our analysis on TCS, weproposeAdaRNN, a novel framework to learn an accurate andadaptive prediction composed of two , to better characterize the distribution information in TS,we propose atemporal distribution characterization (TDC)algo-rithm to split the training data into most diverse periods that arewith large distribution gap inspired by the principle of maximumentropy.

7 After that, we propose atemporal distribution matching(TDM)algorithm to dynamically reduce distribution divergenceusing a RNN-based model. Experiments on activity recognition, airquality prediction, household power consumption and stock priceprediction show that ourAdaRNNoutperforms the state-of-the-artbaselines terms of an accuracy on classification tasksand terms of RMSE on regression to both RNN structures ( , RNNs, LSTMs, and GRUs) anddistribution matching distances ( , cosine distance, MMD [3], oradversarial discrepancy [12]).AdaRNNcan also be extended to theTransformer architecture to further boost its sum up, our main contributions are as follows, Novel problem:For the first time, we propose to modelthe time Series from the distribution perspective, then wepostulate and formulate the Temporal Covariate Shift (TCS)problem in non-stationary time Series , which is more realisticand challenging. General framework:To solve TCS, we propose a generalframeworkAdaRNNthat learns an accurate and adaptivemodel by proposing the temporal distribution characteriza-tion and Temporal Distribution matching algorithms.

8 Effectiveness:We conduct extensive experiments on hu-man activity recognition, air quality prediction, householdpower consumption, and stock price prediction datasets. Re-sults show thatAdaRNNoutperforms the state-of-the-artbaselines on both classification and regression RELATED Time Series analysisTraditional methods for time Series classification or regressioninclude distance-based [13,32], feature-based [38], and ensemblemethods [27]. The distance-based methods measure the distanceand similarity of (segments of ) raw Series data with some metric,like the Euclidean distance or Dynamic time wrapping (DTW) [20].The feature-based methods capture the global/local patterns in thetime Series rely on a set of manually extracted or learned ensemble methods combine multiple weak classifiers to boostthe performance of the , involving heavy crafting on data pre-processing andlabor-intensive feature engineering makes those methods strug-gling to cope with large-scale data and the performance is alsolimited for more complex patterns.

9 Recurrent Neural Networks(RNNs), such as Gated Recurrent Unit (GRU) and Long Short TermMemory (LSTM) have been popular due to their ability to extracthigh quality features automatically and handle the long term depen-dencies. These deep methods tackle TS classification or forecastingby leveraging attention mechanisms [23,33] or tensor factoriza-tion [40] for capturing shared information among Series . Anothertrend is to model uncertainty by combining deep Learning andState space models [37]. Furthermore, some methods [24,37,46]adopt seq2seq models for multi-step predictions. Unlike those meth-ods based on statistical views, ourAdaRNNmodels TS from theperspective of the distributions. Recently, the Transformers struc-ture [45] is proposed for sequence Learning with the self-attentionmechanism. The vanilla Transformer can be easily modified fortime Series prediction, while it still fails for distribution Series segmentation and clustering are two similar topicsto our temporal distribution characterization.

10 Time Series segmen-tation [6,28] aim to separate time Series into several pieces that canbe used to discover the underlying patterns; it mainly uses change-point detection algorithms, include sliding windows, bottom-up,and top-down methods. The methods in segmentation almost don tutilize the distribution matching scheme and can t adapt to ourproblem. Time- Series clustering [16,34], on the other hand, aimsat finding different groups consisting of similar (segments of ) timeseries, which is clearly different from our Distribution matchingWhen the training and test data are coming from different distri-butions, it is common practice to adapt some domain adaptation(DA) algorithms to bridge their distribution gap, such that domain-invariant representations can be learned. DA often performs in-stance re-weighting or feature transfer to reduce the distributiondivergence in training and test data [12,44,47,48,56]. Similar toDA in general methodology, domain generalization (DG) [49] alsolearns a domain-invariant model on multiple source domains in thehope that it can generalize well to the target domain [2,26,30].