Forecasting Time Series by SOFNN with …

1 Forecasting time Series by SOFNN with reinforcement learning Takashi Kuremoto, Masanao Obayashi, and Kunikazu Kobayashi Abstract A self-organized fuzzy neural network ( SOFNN ) to a probability policy which to determinate actions in with a reinforcement learning algorithm called Stochastic Gra- the procedure of reinforcement learning and the error of dient Ascent (SGA) is proposed to forecast a set of 11 time Series . Forecasting is as reward value (Subsection ). The proposed system is confirmed to predict chaotic time Series before, and is applied to predict each/every time Series in NN3 A. Embedding Forecasting competition modifying parameters of threshold of fuzzy neurons only. The training results are obviously effective According to the Takens embedding theorem, the inputs and results of long-term prediction give convincible trend values of prediction system on time t, can be reconstructed as a n in the future of time Series .

2 Dimensions vector space X(t), which includes n observed points with same intervals on a time Series y(t). I. I NTRODUCTION. Though many artificial neural networks (ANN) are suitable X(t) = (x1 (t), x2 (t), , xn (t)) (1). to forecast time Series , radial basis function network (RBFN) = (y(t), y(t ), , y(t (n 1) ) (2). is still recommend to be applied on chaotic time Series [1], [2], [3] and financial time Series [4]. Meanwhile, how to de- where is time delay (interval of sampling) may be an sign the structure of hidden-layer in RBFN is a puzzling and arbitrary value in theory but it effects prediction accuracy confusing problem in practice. Leung and Wang proposed in practice, n is the embedding dimension, n > 1. a technique called the cross-validated subspace method to B.)

3 Self-organized Fuzzy Neural Network ( SOFNN ). estimate the optimum number of hidden units, and applied Fig. 2 shows an architecture of self-organized fuzzy neu- the method to prediction of noisy chaotic time Series [3]. ral network ( SOFNN ) we proposed. The initial number of but the method may fall its efficiency when the training membership function and fuzzy rule is only 1, respectively. sample data is not enough. To overcome the problem of 1) Membership Function: To each element xi (t) of the RBFN, we proposed a self-organization fuzzy network and input X(t), membership function Bij (xi (t)) is represented its effectiveness on prediction of chaotic time Series was as . investigated [5], [6]. (xi (t) mij )2. Furthermore, the learning algorithm is so important to any Bij (xi (t)) = exp 2 (3).

4 2 ij artificial neural network. reinforcement learning (RL), a kind of goal-directed learning , is well known for an agent adapting where mij and ij are the parameters of mean and standard unknown environments [7], [8]. We have proposed to apply deviation of the Gaussian membership function in jth node, a kind of RL called stochastic gradient ascent (SGA) on respectively. Initially, j = 1, and with increasing of input nonlinear predations [5], [6], [9]. The accurate of Forecasting patterns, the new membership function will be added. in experiments using Lorenz chaotic time Series shown its good efficiency whatever the type of ANN is either multi- time Series layer perception (MLP) or SOFNN . y(t). In this paper, we intend to use SOFNN and SGA to forecast a set of 11 time Series given by neural Forecasting Reconstructed Inputs reinforcement learning competition (NN3) [10].

5 The prediction system is introduced in Section 1 in detail, and all of train Forecasting results and X(t). Forecasting results are shown in Section 2. Self-organized Fuzzy Neural Network II. F ORECASTING S YSTEM. r: reward Flow of Forecasting is shown in Fig. 1 and self-organized Prediction under Stochastice Policy fuzzy neural network, ANN predictor, is a RBF-like neural network (Fig. 2). The input layer is given by data in history ^. y(t+1). of time Series (Subsection ). The hidden layer consists Gaussian membership functions which number is decided Evaluation of Prediction by a threshold and the rule layer realizes fuzzy inference (Subsection and ). Forecasting is executed according Authors are with the Graduate School of Science and Systems Engineer- ing, Yamaguchi University , Tokiwadai 2-16-1, Ube, Yamaguchi 755-8611, Fig.

6 1. Flow chart of training and Forecasting . Japan, Tel: +81-836-859520, Fax: +81-836-859501, Email: {wu, , 2) Fuzzy Inference: The fitness k (X(t)) is an algebraic C. reinforcement learning Algorithm (SGA). product of membership functions which connects to rule k. reinforcement learning has recently been well-known as n Y a kind of intelligent machine learning [7], [8]. It needs not k (X(t)) = Bio (xi ) (4) any model of operator but learns to approach to its goal i=1 by observing sensing rewards from environments. Kimura where o is the number of membership function, connects and Kobayashi suggested an algorithm called stochastic with kth rule. o {1, 2, , li }. gradient ascent (SGA), which respect to continuous action and confirmed in control systems [8]. The error of Forecasting Inputs Fuzzy Membership Functions Fuzzy Rules Distributions of prediction Predicted Values is as reward to adjust probability policy in training process.}

7 SGA algorithm is shown under. R1. R. x (t)= y(t). 1. w 1. 1) Accept an observation X(t) from environment. w 1. Average 2) Predict a future data y (t + 1) under a probability R. w 2. R2. ( . y (t + 1), W, X(t)). Stochastic Function x2(t)= y(t- ). w 2 3) Collate training samples of times Series , take the error . as reward ri . R. Deviation w . 4) Calculate the degree of adaption ei (t), and its history R. x n(t)= y(t-(n-1) ). for all elements i of internal variable W . where is RK w . a discount (0 < 1). R.. ei (t) = ln ( . y (t + 1), W, X(t)) (9). i R. ij m ij Di (t) = ei (t) + Di (t 1) (10). Fig. 2. A structure of self-organized fuzzy neural network ( SOFNN ). 5) Calculate i (t) by under equation. 3) Self-organization of Neural Network : A new member- i (t) = (ri b)Di (t) (11).

8 Ship function will be add if where b is a constant. Bij (xi (s)) < F (F : threshold). n (5). 6) Improvement of policy: renew W by under equation. corresponding to the input,li li + 1. W (t) = ( 1 (t), 2 (t), , i (t), ) (12). A new fuzzy rule will be added also adapting to the new membership function. 4) Prediction Policy from Defuzzification: Integrate fuzzy W W + (1 ) W (t) (13). rules with connection weight, fuzzy inference can be ob- tained. The output of signal can be considered as a new where is a learning constant,non-negative. Gaussian distribution either, , 7) Advance time step t to t + 1,return to(1). PK. k=1 k k (X(t), k ) = P K. (6) 6000. k=1 k "101". "train_forecast". 5800 "forecast". PK 5600. k=1 k k (X(t), k ) = PK (7) 5400. k=1 k 5200. tonaru. where is the mean of output, is its standard 5000.

9 Deviation. Weight k and k are parameters concerning 4800. with inputs setX(t), and will be renew during training. 4600. 4400. 1 y (t + 1) )2. ( . y (t + 1), W, X(t)) = . ( exp 4200. 2 2 2. (8) 4000. 0 20 40 60 80 100 120 140 160. where y (t + 1) is the value of one-step ahead prediction, produce by regular random numbers. W means weights k Fig. 3. Forecasting results of time Series 101. and k . " ". vincible trends of the time Series in future. The Forecasting results for all 11 time Series are shown in Table 1, where SMAPE means the mean Symmetric Mean Absolute Pn Percent Error across a time Series (SM AP Es =. 1 |xt . yt |. n t=1 (xt + yt )/2 100)[10]. 8. 1. 7. 6. 5. 0 5000 10000 15000 20000 25000 30000. 4 " ". " ". Fig. 4. Change of Forecasting error in training.

10 " ". 3. 2. III. F ORECASTING NN3 T IME S ERIES. 1. We applied the proposed Forecasting system on a set of 11 time Series provided by Neural Forecasting Competition 0. 0 500000 1e+06 +06 2e+06 +06 3e+06 +06 4e+06. (NN3)[10]. The procedure is : Fig. 5. Change of numbers of membership functions during training. 1) Obtain all data (from 1 to 126, or 123, or 115 steps). of one time Series given by NN3 and normalize their value to around using a linear transformation;. 2) Let the input dimension be 3, and the delay of time be 16. " ". 1; 14. 3) Tune parameters , n F (threshold of membership function Bij (xi (s))), initial value of ( , only ij because 12. mij was set to ), value of reward r, to appropriate 10. /satisfied values by observing change of training errors, then train Forecasting is obtained when the training error (Mean 8.