1 Introduction to Stochastic Processes

1 MA636: Introduction to Stochastic processes1 11 Introduction to Stochastic IntroductionStochastic modelling is an interesting and challenging area of proba-bility and statistics. Our aims in this introductory section of the notesare to explain what astochastic processis and what is meant by theMarkov property, give examples and discuss some of the objectivesthat we might have in studying Stochastic DefinitionsWe begin with a formal definition, Astochastic processis a familyof random variables{X }, indexed by a parameter , where belongsto some index set .In almost all of the examples that we shall look at in this module, will represent time. If is a set of integers, representing specifictime points, we have a Stochastic process indiscrete timeand weshall replace the general subscript byn. So we shall talk about thediscrete time process{Xn}. Sections 2 4 of the module are aboutprocesses in discrete is the real line (or some interval of the real line) we have a sto-chastic process incontinuous timeand we shall replace the generalsubscript bytand change the notation slightly, writingX(t) ratherthanXt.

2 Sections 5 6 of the module are about Processes in reason that we introduce the rather abstract notion of an indexset , rather than just working with time, is that we sometimes wantto study spatial Processes as well as temporal Processes . In a spatialprocess, would be avector, representing location in space ratherthan time. For example, we might have a process{X(u,v)}, repre-senting a random variable that varies across two-dimensional : Introduction to Stochastic processes1 2 Here,X(u,v)represents the value of the process at position (u,v). Wecan even have Processes that evolve in both time and space, so calledspatio-temporal Processes . However, apart from occasional exam-ples, spatial and spatio-temporal Processes are beyond the scope ofthis Processes in time, a less formal definition is that astochasticprocessis simply a process that develops in time according to prob-abilistic rules. We shall be particularly concerned withstationaryprocesses, in which the probabilistic rules do not change with general, for a discrete time process, the random variableXnwilldepend on earlier values of the process,Xn 1,Xn 2.

3 Similarly, incontinuous time,X(t) will generally depend on valuesX(u) foru< , we are often interested in conditional distributions of theformPr(Xtk|Xtk 1,Xtk 2,..,Xt1)for some set of timestk>tk 1>..>t1. In general, this condi-tional distribution will depend upon values ofXtk 1,Xtk 2,.., , we shall focus particularly in this module on Processes thatsatisfy theMarkovproperty, which says thatPr(Xtk|Xtk 1,Xtk 2,..,Xt1)= Pr(Xtk|Xtk 1).The Markov property is named after the Russian probabilist AndreiAndreyevich Markov (1856-1922). An informal mnemonic for remem-bering the Markov property is this. Given the present (Xk 1), thefuture (Xk) is independent of the past (Xk 2,Xk 3,..,X1). TheMarkov property is sometimes referred to as the lack of memory Processes that satisfy the Markov property are typicallymuch simpler to analyse than general Processes , and most of theprocesses that we shall study in this module are Markov course, in attempting to model any real system it will be impor-tant to consider whether the Markov property is likely to well as classifying time as discrete or continuous, we can also clas-sify the random variableXas discrete or continuous.

4 We shall seeMA636: Introduction to Stochastic processes1 3examples of all four combinations (discrete/continuous time in con-junction with discrete/continuous random variable) in this end this section with a few more definitions related to stochasticprocesses: Acounting processis a processX(t) in discrete or continuoustime for which the possible values ofX(t) are the natural num-bers (0,1,2,..) with the property thatX(t) is a non-decreasingfunction oft. Often,X(t) can be thought of as counting thenumber of events of some type that have occurred by basic example of a counting process is the Poisson process,which we shall study in some detail. Asample pathof a Stochastic process is a particularrealisa-tionof the process, a particular set of valuesX(t) for allt(which may be discrete of continuous), generated according tothe ( Stochastic ) rules of the process. Theincrementsof a process are the changesX(t) X(s) be-tween time pointssandt(s<t).

5 Processes in which the in-crements for non-overlapping time intervals are independent andstationary ( dependent only on the lengths of the time inter-vals, not the actual times) are of particular importance. Randomwalks, which we study extensively in Chapter 2, are a good ex-ample. General Processes of this type are called L evy Processes ,and include the Poisson process (Chapter 5) and Brownian mo-tion (Chapter 6). Stochastic and deterministic modelsStochastic models can be contrasted withdeterministicmodels. Adeterministic model is specified by a set of equations that describeexactlyhow the system will evolve over time. In a Stochastic model ,the evolution is at least partially random and if the process is runseveral times, it willnotgive identical results. Different runs of astochastic process are often calledrealisationsof the : Introduction to Stochastic processes1 4 Deterministic models are generally easier to analyse than stochasticmodels.

6 However, in many cases Stochastic models are more realistic,particulary for problems that involve small numbers . For example,suppose we are trying to model the management of a rare species,looking at how different strategies affect the survival of the models will not be very helpful here, because they willpredict that the species either definitely becomes extinct or definitelysurvives. In a Stochastic model , however, there will be a probability ofextinction, and we study how this is affected by management recent years, the distinction between deterministic and stochas-tic models has been blurred slightly by the development of chaoticmodels. A chaotic model is a deterministic model that is extremelysensitive to the values of some of the parameters in the model . Makinga very small change to the values of these parameters can make theoutcome of the model completely different. Some people have arguedthat systems that are normally regarded as Stochastic Processes arebetter regarded as chaotic deterministic systems, as exemplified bythis quote:A mountain stream, a beating heart, a smallpox epidemic , and a col-umn of rising smoke are all examples of dynamic phenomena thatsometimes seem to behave randomly.

7 In actuality, such Processes ex-hibit a special order that scientists and engineers are only just begin-ning to understand. This special order is deterministic chaos , orchaos, for own, doubtless biased, view is that Stochastic models are, in gen-eral, much more useful than models based on deterministic chaos. Butin any event, chaos theory uses quite different mathematical techniquesand is outside the scope of this Examples of Stochastic processesIn this section, we offer an eclectic collection of examples of stochasticprocesses, to give you some idea of the wide range of application : Introduction to Stochastic processes1 Exchange ratesFigure 1:Exchange rate between British pound and Japanese yen overthe period 1/1/02-31/7/04. Upper graph shows true exchange rate,lower graph shows a simulation of a random walk upper panel of Figure 1 shows the exchange rate between theBritish pound and the Japanese yen from 1st January 2002 to 31stJuly 2004.

8 Over this period, the average exchange rate was yento one pound. The lower panel shows a simulation of a type of sto-chastic process called arandom walk. We will be studying randomwalks in Section 2 of this module. In the random walk model , thedaily changes in exchange rate are independent normal random vari-ables with zero mean and standard deviation of (matching theMA636: Introduction to Stochastic processes1 6standard deviation in the observed data). Whilst the detailed patternsare of course different, the two series have a similar structure. Notethat in the random walk model , upward and downward movements inthe exchange rate are equally likely, and there is no scope for makingmoney through currency speculation except by Photon emissionPhotons are minute particles of light. Under some circumstances, alight source will emit photons at random (according to what is called aPoisson process, which we will study in section 5).

9 Individual photonsare too faint to be detected by the human eye, but electronic photondetectorscandetect single photons. However, there is a problem withthese machines. Immediately after they have detected a photon, thereis a short time period, known as the dead time, during which no newphotons can be number of photons detected by the machine therefore underesti-mates the actual number of photons emitted by the light source andwe need to correct the observed number of photons to get an estimateof the true number. It s clear that a relatively complicated correctionwill be needed because when the light source is emitting photons ata low rate, the chances of two photons being emitted close enoughtogether to be affected by dead time is quite small, whereas when theemission rate is high, many photons will be missed. To tackle thisproblem it is necessary to set up a Stochastic model that models boththe emission of photons and the dead time effect and then to studythe distribution of the observed number of photons and see how thisis related to the number actually epidemic modelsFigure 2, which is from a World Health Organisation report, showsthe progress of the SARS epidemic in 2002-3.

10 The data plotted arethe number of new cases reported each week worldwide. The dataexclude 2527 probable cases of SARS (mostly from Beijing) in whichMA636: Introduction to Stochastic processes1 7the data of onset is unknown. This is an example of a discrete timeFigure 2:Daily number of new cases of SARS worldwide during theperiod 1/11/02 10/7/03. each daystochastic process. The variable of interest (number of cases) is alsodiscrete. Many sophisticated mathematical models of epidemics havebeen developed. These incorporate factors such as the number of con-tacts that an infected person makes with non-infected people, and thechances that the infection will be passed on during one of these con-tacts. These models help us to understand the factors that determinewhether an epidemic is likely to get a hold and affect a large propor-tion of the population (like the BSE epidemic in cattle in the UK), orwhether it is more likely to affect just a few people and then die models can also be used to predict the effects of interventions -how will it affect the epidemic if we vaccinate 50% of the population?