Machine Learning Strategies for Time Series …

1 Machine Learning Strategies for TimeSeries PredictionMachine Learning Summer School(Hammamet, 2013)Gianluca BontempiMachine Learning Group, Computer Science DepartmentBoulevard de Triomphe - CP 212 Learning Strategies for Prediction p. 1/128 Introducing myself 1992: Computer science engineer (Politecnico di Milano, Italy), 1994: Researcher in robotics in IRST, Trento, Italy, 1995: Researcher in IRIDIA, ULB Artificial Intelligence Lab, Brussels, 1996-97: Researcher in IDSIA, Artificial Intelligence Lab,Lugano,Switzerland, 1998-2000: Marie Curie fellowship in IRIDIA, ULB ArtificialIntelligenceLab, Brussels, 2000-2001: Scientist in Philips Research, Eindhoven, The Netherlands, 2001-2002: Scientist in IMEC, Microelectronics Institute, Leuven,Belgium, since 2002: professor in Machine Learning , Modeling and Simulation,Bioinformatics in ULB Computer Science Dept.

2 , Brussels, since 2004: head of the ULB Machine Learning Group (MLG). since 2013: director of the Interuniversity Institute of Bioinformatics inBrussels (IB)2, Learning Strategies for Prediction p. 2 and in terms of distancesAccording to MathSciNet Distance from Erdos= 5 Distance from Zaiane= 6 Distance from Deisenroth= 8 Machine Learning Strategies for Prediction p. 3/128 ULB Machine Learning Group (MLG) 3 professors, 10 PhD students, 5 postdocs. Research topics: Knowledge discovery from data, Classification, Computationalstatistics, Data mining, Regression, time Series prediction, Sensor networks,Bioinformatics, Network inference. Computing facilities: high-performing cluster for analysis of massive datasets,Wireless Sensor Lab. Scientific collaborations in ULB: Hopital Jules Bordet, Laboratoire de M decineexperimentale, Laboratoire d Anatomie, Biom canique et Organog n se (LABO),Service d Anesthesie (ERASME).

3 Scientific collaborations outside ULB: Harvard Dana Farber(US), UCL MachineLearning Group (B), Politecnico di Milano (I), Universit del Sannio (I), Inst RechCliniques Montreal (CAN). Machine Learning Strategies for Prediction p. 4/128 ULB-MLG: recent projects1. Machine Learning for Question Answering (2013-2014).2. Adaptive real- time Machine Learning for credit card fraud detection (2012-2013).3. Epigenomic and Transcriptomic Analysis of Breast Cancer(2012-2015).4. Discovery of the molecular pathways regulating pancreatic beta cell dysfunctionand apoptosis in diabetes using functional genomics and bioinformatics: ARC(2010-2015)5. ICT4 REHAB - Advanced ICT Platform for Rehabilitation (2011-2013)6. Integrating experimental and theoretical approaches todecipher the molecularnetworks of nitrogen utilisation in yeast: ARC (2006-2010).

4 7. TANIA - Syst me d aide la conduite de l anesth sie. WALEO II project fundedby the R gion Wallonne (2006-2010)8. "COMP2 SYS" (COMP utational intelligence methods for COMPlex SYStems)MARIE CURIE Early Stage Research Training funded by the EU (2004-2008). Machine Learning Strategies for Prediction p. 5/128 What you are supposed to know Basic notions of probability and statistics Random variable Expectation, variance, covariance Least-squaresWhat you are expected to get acquainted with Foundations of statistical Machine Learning How to build a predictive model from data Strategies for forecastingWhat will remain Interest, curiosity for Machine Learning taste for prediction Contacts Companion Learning Strategies for Prediction p. 6/128 Outline Notions of time Series (30 mins) conditional probability Machine Learning for prediction (45 mins) bias/variance parametric and structural identification validation model selection feature selection COFFEE BREAK Local Learning (15 mins) forecasting : one-step and multi-step-ahed (30 mins) Some applications (15 mins) time Series competitions wireless sensor biomedical Future directions and perspectives (15 mins) Machine Learning Strategies for Prediction p.

5 7/128 What is Machine Learning ? Machine Learning is that domain of computational intelligence which isconcerned with the question of how to construct computer programs thatautomatically improve with experience.[16]Reductionist attitude:ML is just a buzzword which equates to statistics plusmarketingPositive attitude:ML paved the way to the treatment of real problems relatedto data analysis, sometimes overlooked by statisticians (nonlinearity,classification, pattern recognition, missing variables, adaptivity,optimization, massive datasets, data management, causality,representation of knowledge, parallelisation)Interdisciplinary attitude:MLshouldhave its roots on statistics andcomplements it by focusing on: algorithmic issues, computationalefficiency, data Learning Strategies for Prediction p. 8/128 Why study Machine Learning ?

6 Machine Learning is cool. Practical way to understand:All models are wrong but some are The fastest way to become a data scientist .. the sexiest jobin the 21stcentury Someone who knows statistics better than a computer scientists andprograms better than a Learning Strategies for Prediction p. 9/128 Notion of time seriesMachine Learning Strategies for Prediction p. 10/128 time seriesDefinitionA time Series is a sequence of observationsst R, usuallyordered in of time Series in every scientific and applied domain: Meteorology: weather variables, like temperature, pressure, wind. Economy and finance: economic factors (GNP), financial indexes,exchange rate, spread. Marketing: activity of business, sales. Industry: electric load, power consumption, voltage, sensors. Biomedicine: physiological signals (EEG), heart-rate, patienttemperature.

7 Web: clicks, logs. Genomics: time Series of gene expression during cell Learning Strategies for Prediction p. 11/128 Why studying time Series ?There are various reasons:Predictionof the future based on the the process producing the the mechanism generating the the salient features of the Learning Strategies for Prediction p. 12/128 Univariate discrete time Series Quantities, like temperature and voltage, change in a continuous way. In practice, however, the digital recording is made discretely in time . We shall confine ourselves todiscretetime Series (which however takecontinuous values). Moreover we will considerunivariatetime Series , where one type ofmeasurement is made repeatedly on the same object or individual. Multivariate time Series are out of the scope of this presentation butrepresent an important topic in the Learning Strategies for Prediction p.

8 13/128A general modelLet an observed discrete univariate time Series bes1, .. , sT. This meansthat we haveTnumbers which are observations on some variable made atTequally distant time points, which for convenience we label1,2, .. , fairly general model for the time Series can be writtenst=g(t) + tt= 1, .. , TThe observed Series is made of two componentsSystematic part:g(t), also calledsignalortrend, which is a determisticfunction of timeStochastic sequence:a residual term t, also callednoise, which follows aprobability Learning Strategies for Prediction p. 14/128 Types of variationTraditional methods of time - Series analysis are mainly concerned withdecomposing the variation of a seriesstinto:Trend: this is a long-term change in the mean level, eg. an increasing effect: many time Series (sale figures, temperature readings) exhibitvariation which is seasonal ( annual) in period.

9 The measure and theremoval of such variation brings fluctuations: after trend and cyclic variations have been removedfrom a set of data, we are left with a Series of residuals, which may ormay not becompletely will assume here that once we have detrended and deseasonalized theseries, we can still extract information about the dependency between thepast and the future. Henceforth twill denote the detrended anddeseasonalized Learning Strategies for Prediction p. 15/128320340360observed320340360trend 3 11 2 3seasonal of additive time seriesDecomposition returned by the R Learning Strategies for Prediction p. 16/128 Probability and dependency forecasting a time Series is possible sincefuture depends on the pastoranalogously because there is a relationship between the future and thepast. However this relation is not deterministic and can hardly be writtenin an analytical form.

10 An effective way to describe a nondeterministic relation between twovariables is provided by the probability formalism. Consider two continuous random variables 1and 2representing forinstance the temperature today (timet1) and tomorrow (t2). We tend tobelieve that 1could be used as a predictor of 2with some degree ofuncertainty. The stochastic dependency between 1and 2is resumed by the jointdensityp( 1, 2)or equivalently by theconditional probabilityp( 2| 1) =p( 1, 2)p( 1) Ifp( 2| 1)6=p( 2)then 1and 2are not independent or equivalentlythe knowledge of the value of 1reduces the uncertainty about Learning Strategies for Prediction p. 17/128 Stochastic processes The stochastic approach to time Series makes the assumptionthat atime Series is a realization of a stochastic process (like tossing anunbiased coin is the realization of a discrete random variable with equalhead/tail probability).