Lecture 4: Continuous-time Markov Chains

1 Miranda Holmes-CerfonApplied Stochastic Analysis, Spring 2019 Lecture 4: Continuous-time Markov ChainsReadings Grimmett and Stirzaker (2001) , : Grimmett and Stirzaker (2001) (a survey of the issues one needs to address to make the discussionbelow rigorous) Norris (1997) Chapter 2,3 (rigorous, though readable; this is the classic text on Markov Chains , bothdiscrete and continuous ) Durrett (2012) Chapter 4 (straightforward introduction with lots of examples)Many random processes have a discrete state space, but can change their values at any instant of timerather than at fixed intervals: radioactive atoms decaying, the number of molecules in a chemical reaction,populations with birth/death/immigration/emigration, the number of emails in an inbox, etc.

2 The process ispiecewise constant, with jumps that occur at continuous times, as in this example showing the number ofpeople in a lineup, as a function of time (from Dobrow (2016)):The dynamics may still satisfy a continuous version of the Markov property, but they evolve continuously intime. Rather than simply discretize time and apply the tools we learned before, a more elegant model comesfrom considering a Continuous-time Markov chain (ctMC.)In this class we ll introduce a set of tools to describe Continuous-time Markov Chains . We ll make the linkwith discrete- time Chains , and highlight an important example called the Poisson process. If time permits,we ll show two applications of Markov Chains (discrete or continuous ): first, an application to clustering anddata science, and then, the connection between MCs, electrical networks, and flows in porous Definition and Transition (Xt)t 0be a family of random variables taking values in a finite or countable state spaceS, which we can take to be a subset of the acontinuous- time Markov chain (ctMC) if it satisfiesMiranda Holmes-CerfonApplied Stochastic Analysis, Spring 2019theMarkov property:1P(Xtn=in|Xt1=i1.)

3 ,Xtn 1=in 1) =P(Xtn=in|Xtn 1=in 1)(1)for alli1,..,in Sand any sequence 0 t1<t2< <tnof times. The process istime-homogeneousifthe conditional probability does not depend on the current time , so that:P(Xt+s=j|Xs=i) =P(Xt=j|X0=i),s 0.(2)We will consider only time -homogeneous processes in this Lecture . Additionally, we will consider onlyprocesses which Markov property (1) says that the distribution of the chain at some time in the future, only depends onthe current state of the chain , and not its history. The difference from the previous version of the Markovproperty that we learned in Lecture 2, is that now the set of timestis continuous the chain can jumpbetween states at any time , not just at integer is no exact analogue of the transition matrixP, since there is no natural unit of time .

4 Therefore weconsier the transition probabilities as a function of probabilityfor a time -homogeneous chain isPi j(t) =P(Xt+s=j|Xs=i),s,t 0.(3)WriteP(t) = (Pi j(t))for the matrix of transition probabilities at ,P(t)is a stochastic a time -inhomogeneous chain , the transition probability would be a function of two times, asPi j(s,t) =P(X(t+s)=j|Xs=i).In a similar way to the discrete case, we can show the Chapman-Kolmogorov equations hold forP(t):Chapman-Kolmogorov Equation.( time -homogeneous)P(t+s) =P(t)P(s) Pi j(t+s) = k SPik(t)Pk j(s).(4)1 The Markov property in continuous time can be formulated more rigorously in terms of -algebras. Let( ,F,P)a the probabilityspace and let{Ft}t 0be a filtration: an increasing sequence of -algebras such thatFt Ffor eacht, andt1 t2 Ft1 suppose the processXtisadaptedto the filtration{Ft}t 0: eachXtis measurable with respect toFt.

5 For example, this will betrue automatically if we letFtbe the -algebra generated by(Xs)0 s t, generated by the pre-imagesX 1s(B)for Borel setsB the Markov property ifE(f(Xt)|Fs) =E(f(Xt)| (Xs))for all 0 s tand bounded, measurable functionsf. Another way to say this isP(Xt A|Fs) =P(Xt A| (Xs)), whereP( | )is aregular conditional probability(see Koralov and Sinai (2010), )2 Miranda Holmes-CerfonApplied Stochastic Analysis, Spring j(s+t) =P(Xs+t=j|X0=i)= kP(Xs+t=j|Xt=k,X0=i)P(Xt=k|X0=i)(LoTP)= kP(Xs+t=j|Xt=k)P(Xt=k|X0=i)( Markov property)= kPik(t)Pk j(s)( time -homogeneity.) set of operators(T(t))t 0such thatT(t+s) =T(s)T(t)is asemigroup. The transition probabil-itiesP(t)form a semigroup (whereP(t)is an operator in the sense that it maps vectors to vectors.)

6 Studyingabstract semigroups of operators, sometimes with additional properties like continuity att=0, or uniformcontinuity, is another approach to studying Markov processes, that is often used in the probability Grimmett and Stirzaker (2001), , for more details related to Continuous-time MCs, and seeKoralov and Sinai (2010); Pavliotis (2014) for a discussion of general Markov transition probability can be used to completely characterize the evolution of probability for a Continuous-time Markov chain , but it gives too much information. We don t need to knowP(t)for all timestin orderto characterize the dynamics of the chain . We will consider two different ways of completely characterizingthe dynamics:(i) By the times at which the chain jumps, and the states that it jumps to.

7 (ii) Through thegenerator Q, which is like an infinitesimal version will consider both approaches. Let s start with the second one, since it gives the most fundamentalcharacterization of a Markov Infinitesimal generator, and forward and backward equationsFor the rest of the Lecture , we will assume that|S|< . Most of the results we derive will also be true when|S|= under certain additional, not-very-restrictive assumptions. Because worrying about these details istechnical and beyond the scope of the course, we leave them out and focus on the easier case for now. SeeNorris (1997) for a rigorous discussion that includes the case|S|= . Infinitesimal generatorThe fundamental way to characterize a ctMC is by its generator, which is like its infinitesimal transitionrates.

8 Let s go back to the transition probabilityP(t). By definition, we have thatP(0) =I, the identitymatrix. Let sassumethatP(t)is right-differentiable att=0. This does not have to be the case for anyctMC, but it will be true in applications and it makes the subsequent theory much easier to , the weaker assumptionP(t) Iuniformly ast 0 is sufficient to derive the results; see Grimmett and Stirzaker (2001),section or Norris (1997), Chapter Holmes-CerfonApplied Stochastic Analysis, Spring 2019assumption also implies thatP(t)is differentiable for allt>0. You can show this (if you are interested)from the Chapman-Kolmogorov (Xt)t 0be a ctMC with transition probabilitiesP(t). Thegeneratororinfinitesimalgeneratorof the Markov chain is the matrixQ=limh 0+P(h) Ih.

9 (5)Write its entries asQi j=qi properties of the generator that follow immediately from its definition are:(i) Its rows sum to 0: jqi j=0.(ii)qi j 0 fori6=j.(iii)qii<0 Proof.(i) jPi j(h) =1, sinceP(h)is a transition matrix, and jIi j=1. Pass to the limit.(ii)Pi j(h) Ii j, so this is true in the limit.(iii) Follows from (i) and (ii), sinceqii= jqi |S|= , then most of what follows will be true under the assumption that supi|qii|< . Thiscondition ensures the chain cannot blow up to in finite time . Therefore, we will consider examples with|S|= , even though we won t prove here that the theory works in this case. See Norris (1997), Chapters should the entries of the generator be interpreted? Each entryqi jis like arateof jumping fromitoj.

10 Recall that a rate measures the average number of events (in the colloquial sense), per unit time . In thiscase, an event is a transition fromitoj. The diagonal entries qiiare the overall rates of leaving each an experiment where we start in statei, and every time we jump out of it we return ,qi jwould be approximated by the total number of jumps to statej, divided by the total time overwhich we perform the experiment. Indeed, the instantaneous transition rate of hittingj6=iislimh 0+E[number of transitions tojin (t,t+h]|Xt=i]h=limh 0+P(Xt+h=j|Xt=i)h=qi j.(Since, in a small interval of time , the number of transitions is either 0 or 1.) For a similar reason the overallrate of leaving is qii= jqi j, because the overall rate of leaving is the rate at whichanytransitionhappens, so it is the sum of the individual justification of the word rate comes from considering the transition probabilities after a smallamount of timehhas elapsed:Pi j(h) =qi jh+o(h)(j6=i)Pii(h) =1+qiih+o(h)(j=i)(6)4 Miranda Holmes-CerfonApplied Stochastic Analysis, Spring 2019 Thenqi jgives the rate at which probability accumulates in statej(given we are in statei), at least instanta-neously, and qiiis the rate at which probability are some examples of ctMCs and their corresponding generators.)