AdaRNN: Adaptive Learning and Forecasting for Time Series

1 AdaRNN: Adaptive Learning and Forecasting for time Series Yuntao Du1, Jindong Wang2, Wenjie Feng3, Sinno Pan4, Tao Qin2, Renjun Xu5, Chongjun Wang11 Nanjing University, Nanjing, China2 Microsoft Research Asia, Beijing, China3 Institute of Data Science, National University of Singapore4 Nanyang Technological University5 Zhejiang time Series Forecasting has a wide range of real-world appli-cations, it is a very challenging task. This is because the statisticalproperties of a time Series can vary with time , causing the distri-bution to change temporally, which is known as the distributionshift problem in the machine Learning community.

2 By far, it stillremains unexplored to model time Series from the distribution-shiftperspective. In this paper, we formulate the Temporal CovariateShift (TCS) problem for the time Series Forecasting . We proposeAdaptive RNNs (AdaRNN) to tackle the TCS composed of two modules. The first module is referredto as Temporal Distribution Characterization, which aims to bettercharacterize the distribution information in a time Series . The sec-ond module is termed as Temporal Distribution Matching, whichaims to reduce the distribution mismatch in the time Series to learnan RNN-based Adaptive time Series prediction ageneral framework with flexible distribution distances on human activity recognition, air quality prediction,household power consumption, and financial analysis show thatAdaRNNoutperforms some state-of-the-art methods of accuracy on classification tasks and terms ofthe mean squared error on regression tasks.

3 We also show thatthe temporal distribution matching module can be extended to theTransformer architecture to further boost its CONCEPTS Computing methodologies Transfer Learning ; Series , multi-task Learning , transfer learningACM Reference Format:Yuntao Du1, Jindong Wang2, Wenjie Feng3, Sinno Pan4, Tao Qin2, RenjunXu5, Chongjun Wang1. : Adaptive Learning and Forecastingfor time Series . InProceedings of the 30th ACM Int l Conf. on Informationand Knowledge Management (CIKM 21), November 1 5, 2021, Virtual Event, , New York, NY, USA, 11 pages. Corresponding author: J.

4 Wang and W. Feng. This work is done when Y. Du was anintern at to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee.

5 Request permissions from 21, November 1 5, 2021, Virtual Event, Australia. 2021 Association for Computing ISBN 978-1-4503-8446-9/21/11.. $ dataProbability distributionB Temporal Covariate Shift:?A Unseen testC Figure 1: Thetemporal covariate shiftproblem in non-stationary time Series . The raw time Series data ( ) is mul-tivariate in reality. At time intervals , , and the unseentest data, the distributions are different: ( ) ( ) ( ) ( ). With the distribution changing over time ,how to build an accurate and Adaptive model?

6 1 INTRODUCTIONTime Series (TS) data occur naturally in countless domains includingfinancial analysis [55], medical analysis [29], weather conditionprediction [46], and renewable energy production [5]. Forecastingis one of the most sought-after tasks on analyzing time Series data(arguably the most difficult one as well) due to its importance inindustrial, society and scientific applications. For instance, giventhe historical air quality data of a city for the last five days, how topredict the air quality in the future more accurately?In real applications, it is natural that the statistical propertiesof TS are changing over time , , the non-stationary TS.

7 Overthe years, various research efforts have been made for buildingreliable and accurate models for the non-stationary TS. Tradi-tional approaches such as hidden Markov models (HMMs) [43],dynamic Bayesian networks [35], Kalman filters [9], and otherstatistical models ( ARIMA [19]), have made great progress. Re-cently, better performance is achieved by the recurrent neural net-works (RNNs) [37,46]. RNNs make no assumptions on the temporalstructure and can find highly non-linear and complex dependencerelation in non-stationary property of time Series implies thedata dis-tribution changes over time .

8 Given the example in Figure 1, datadistributions ( )vary for different intervals , , and where , are samples and predictions respectively; Especially for the test datawhich is unseen during training, its distribution is also differentfrom the training data and makes the prediction more conditional distribution ( | ), however, is usually consideredto be unchanged for this scenario, which is reasonable in manyreal applications. For instance, in stock prediction, it is naturalthat the market is fluctuating which changes the financial [ ] 11 Aug 2021 CIKM 21, November 1 5, 2021, Virtual Event, , et al.

9 ( ( )), while the economic laws remain unchanged ( ( | )). It isan inevitable result that the afore-mentioned methods have inferiorperformance and poor generalization [22] since the distributionshift issue violates their basic , it remains unexplored to model the time seriesfrom the distribution perspective. The main challenges of the prob-lem lies in two aspects. First, how to characterize the distributionin the data to maximally harness the common knowledge in thesevaried distributions? Second, how to invent an RNN-based distri-bution matching algorithm to maximally reduce their distributiondivergence while capturing the temporal dependency?

10 In this paper, we formally define theTemporal Covariate Shift(TCS)in Figure 1, which is a more practical and challenging set-ting to model time Series data. Based on our analysis on TCS, weproposeAdaRNN, a novel framework to learn an accurate andadaptive prediction composed of two , to better characterize the distribution information in TS,we propose atemporal distribution characterization (TDC)algo-rithm to split the training data into most diverse periods that arewith large distribution gap inspired by the principle of maximumentropy. After that, we propose atemporal distribution matching(TDM)algorithm to dynamically reduce distribution divergenceusing a RNN-based model.