Stochastic Processes I - MIT OpenCourseWare

1 Lecture 5 : Stochastic Processes I1 Stochastic processA Stochastic process is a collection of random variables indexed by alternate view is that it is a probability distribution over a spaceof paths; this path often describes the evolution of some random value, orsystem, over time. In a deterministic process, there is a fixed trajectory(path) that the process follows, but in a Stochastic process, we do not knowapriori which path we will be given. One should not regard this as havingno information of the path since the information on the path is given bythe probability distribution. For example, if the probability distribution isgiven as one path having probability one, then this is equivalent to having adeterministic process. Also, it is often interpreted that the process evolvesover time. However, from the formal mathematical point of view, a betterpicture to have in mind is that we have some underlying (unknown) path,and are observing only the initial segment of this example, the functionf:R 0 Rgivenbyf(t) =tis a determin-isticprocess, but a random function f:RR 0 givenbyf(t) =twithprobability1/2 andf(t) = twith probability 1/2 is a Stochastic is a rather degernerate example and we will later see more examples ofstochastic are still dealing with a single basic experiment that involves outcomesgoverened by a probability law.

2 However, the newly introduced time variableallows us to ask many new interesting questions. We emphasize on thefollowing topics:(a) We tend to focus on the dependencies in the sequence of valuesgenerated by the process. For example, how do future prices of a stockdepend on past values?(b) We are interested in long-term averages involving the entire sequenceof generated values. For example, what is the fraction of time that a machineis idle?1(c) We are interested in boundary events. For example, what is theprobability that within a given hour all circuits of some telephone systembecome simultaneously busy?A Stochastic process has discrete-time if the time variable takes positiveinteger values, and continuous-time if the time variable takes postivie realvalues. We start by studying discrete time Stochastic Processes .

3 Theseprocesses can be expressed explicitly, and thus are more tangible , or easyto visualize . Later we address continuous time Simple random walkLetY1,Y2, be random variables such thatYi= 1 with equalprobability. LetX0= 0 andXk=Y1+ +Yk,for allk 1. This gives a probability distribution over the sequences{X0,X1, ,}, and thus defines a discrete time Stochastic process. Thisprocess isknown as theone-dimensional simple random walk, which weconveniently refer to asrandom walkfrom now the central limit theorem, we know that for large enoughn,thedistribution of1 nXnconverges to the normal distribution with mean 0 andvariance 1. This already tells us some information about the random state some further properties of the random (i)E[Xk] = 0for allk.(ii) (Independent increment) For all0 =k0 k1 kr, the randomvariablesXki+1 Xkifor0 i r 1are mutually independent.

4 (iii) (Stationary) For allh 1andk 0, the distribution ofXk+h Xkis the same as the distribution proofs are straightforward and are left as an exercise. Notethat these properties hold as long as the incrementsYiare identical andindependent and have mean 5 Example (i) Suppose that a gambler plays the following game. Ateach turn the dealer throws an un-biased coin, and if the outcome is headthe gambler wins $1, while if it is head she loses $1. If each coin toss isindependent, then the balance of the gambler has the distribution of thesimple random walk.(ii) random walk can also be used as a (rather inaccurate) model ofstock two positive integersAandB, what is the probability that therandom walk reachesAbefore it reaches B? Let be the first time atwhich the random walk reaches eitherAor B.

5 ThenX =Aor (k) =P(X =A|X0=k),and note that our goal is to computef(0). The recursive formulaf(k) =12f(k+ 1) +1f2(k 1) follows from considering the outcome of the first coin-toss. We also have the boundary conditionsf(A) = 1,f( B) = 0. If weletf( B+ 1) = , then it follows thatf(1 B+r) = rforallr A+ , =A+B, and it follows thatf(0) =BLecture +B3 Markov ChainOne important property of the simple random walk is that the effect of thepast on the future is summarized only by the current state, rather thanthe whole history. In other words, the distribution ofXk+1depended onlyon the value ofXk, not on the whole set of values ofX0,X1, ,Xk. Astochastic process with such property is called aMarkov formally, letX0,X1, be a discrete-time Stochastic process whereeachXitakes value in some discrete setS(note that this is not the case inthe simple random walk).

6 The setSis called thestate say thatthe Stochastic process has theMarkov propertyifP(Xn+1=i|Xn,Xn1, ,X0) =P(Xn+1=i |Xn)for alln 0andi S. We will discuss the case whenSis a finite set. Inthis case, we letS= [m] for some positive Stochastic process with the Markov property is called a Markov that a finite Markov chain can be described in terms of thetransitionprobabilitiespij=P(Xn+1=j|X n=i)i,j can easily see that pij= 1 i S3 All the elements of a Markov chain model can be encoded in atransitionprobability matrix p11p21 pA= m1 . pm2 ..p1mp2m pmmNotethat the sum of each column is equal to one. Example (i) A machine can be either working or broken on a givenday. If it is working, it will break down in the next day with , and will continue working with probability If it breaks downon agiven day, it will be repaired and be working in the next day withprobability , and will continue to be broken down with probability canmodel this machine by a Markov chain with two states: working,and broken down.

7 The transition probability matrix is given by[ ](ii) A simplerandom walk is an example of a Markov chain. However,there is no transition probability matrix associated with the simple randomwalk since the sample space is of infinite (n) =P(Xn=j|X0=i) be then-th step transition probabilities satisfy the recurrence relationmrij(n) = rik(n 1)pkjforn >1,k=1whererij(1) =pij. Hence then-step transition probability matrix can easilybe shown to distributionof a Markov chain is a probability distributionover the state spaceS(whereP(X0=j) = j) such that m j= k pkj( j S).k=1 Example 0. Consider the Markov chainX0,Xn+11,X2, such thatXn+1=X1 with proability2andXn+1=Xn 1 with probability12. Then the stationary distribution of this Markovchain is i=1 Lecture 5infor all .4 Note that the vector ( 1, 2, , m) is an eigenvector ofAwith eigen-value 1.

8 Hence the following theorem can be deduced from the Perron-Frobenius >0for alli,j S, then there exists a unique station-ary distribuion of the system. Moreover,limrij(n) = j, i,j A corresponding theorem is not true if we consider infinite state MartingaleDefinition discrete-time Stochastic process{X0,X1, }is amar-tingaleifXt=E[Xt+1|Ft],forallt 0, whereFt={X0, ,Xt}(hencewe are conditioning on theinitial segment of the process).This says that our expectated gain in the process is zero at all can also view this definition as a Mathematical formalization of a gameof chance being allt s, we haveXs=E[Xt|Fs]. follows from (i) random walk is a martingale.(ii) The balance of a roulette player is not a martingale (we always haveXk>E[Xk+1|Fk]).(iii) LetY1,Y2, be random variables such thatYi= 2 withprobability13andYi=12with probability2 Lecture 5.

9 LetX0= 0,Xk=ki= {X0,X1, }forms a martingale. 5 Optional stopping theoremDefinition (Stopping time) Given a Stochastic process{X0,X1, },anon-negative integer-valued random variable is called astopping timeif for every integerk 0, the event kdepends only on the eventsX0,X1, , (i) In the coin toss game, consider a gambler who bets $1all the time. Let be the first time at which the balance of the gamblerbecomes $100. Then is a stopping time.(ii) Consider the same gambler as in (i). Let be the time of the firstpeak (local maximum) of the balance of the gambler. Then is not astopping (Doob s optional stopping time theorem, weak form) SupposethatX0,X1,X2, is a martingale sequence and is a stopping time suchthat Tfor some constantT. ThenE[X ] =E[X0]. thatT 1X =X0+ (Xi+1i=0 Xi) 1{ i+1.}(we usedthe fact T).

10 SinceTis a constant, by linear of expectation wehaveT 1E[X ] =E[X0]+ E[(Xi+1 Xi){=0 1 i+1}i].The main observation is that i+ 1 is determined byX0,X1, , [(Xi+1 Xi) 1{ i+1}]=E[E[(Xi+1 Xi) 1{ i+1}|Fi=E[(E[X[i+1| Fi]] Xi) 1{ i+1]}=E0] 1{ i+1}]= [X ] =E[X0].Lecture 5 The condition can be further weakened (see [4]). The lesson to learn isthat a mortal being has no winning strategy (when the game is fair) . Onthe other hand, if one has some advantage over an opponent in some game,then no matter how small that advantage is, he/she will win in the long the coin toss game, consider the following strategy. Thegambler stops playing the first time at which the balance becomes $ that by definition, we haveE[X ] = 100. Does this contradict theoptional stopping theorem?6 Lecture 5 Optional stopping time theorem can be used to deduce interesting two positive integersaandb, consider the followingstrategy for the coin toss game.