1 Introduction to Stochastic Processes

1 MA636: Introduction to Stochastic processes1 11 Introduction to Stochastic IntroductionStochastic modelling is an interesting and challenging area of proba-bility and statistics. Our aims in this introductory section of the notesare to explain what astochastic processis and what is meant by theMarkov property, give examples and discuss some of the objectivesthat we might have in studying Stochastic DefinitionsWe begin with a formal definition, Astochastic processis a familyof random variables{X }, indexed by a parameter , where belongsto some index set.

2 In almost all of the examples that we shall look at in this module, will represent time. If is a set of integers, representing specifictime points, we have a Stochastic process indiscrete timeand weshall replace the general subscript byn. So we shall talk about thediscrete time process{Xn}. Sections 2 4 of the module are aboutprocesses in discrete is the real line (or some interval of the real line) we have a sto-chastic process incontinuous timeand we shall replace the generalsubscript bytand change the notation slightly, writingX(t) ratherthanXt.

3 Sections 5 6 of the module are about Processes in reason that we introduce the rather abstract notion of an indexset , rather than just working with time, is that we sometimes wantto study spatial Processes as well as temporal Processes . In a spatialprocess, would be avector, representing location in space ratherthan time. For example, we might have a process{X(u,v)}, repre-senting a random variable that varies across two-dimensional : Introduction to Stochastic processes1 2 Here,X(u,v)represents the value of the process at position (u,v).

4 Wecan even have Processes that evolve in both time and space, so calledspatio-temporal Processes . However, apart from occasional exam-ples, spatial and spatio-temporal Processes are beyond the scope ofthis Processes in time, a less formal definition is that astochasticprocessis simply a process that develops in time according to prob-abilistic rules. We shall be particularly concerned withstationaryprocesses, in which the probabilistic rules do not change with general, for a discrete time process, the random variableXnwilldepend on earlier values of the process,Xn 1,Xn 2.

5 Similarly, incontinuous time,X(t) will generally depend on valuesX(u) foru< , we are often interested in conditional distributions of theformPr(Xtk|Xtk 1,Xtk 2,..,Xt1)for some set of timestk>tk 1>..>t1. In general, this condi-tional distribution will depend upon values ofXtk 1,Xtk 2,.., , we shall focus particularly in this module on Processes thatsatisfy theMarkovproperty, which says thatPr(Xtk|Xtk 1,Xtk 2,..,Xt1)= Pr(Xtk|Xtk 1).The Markov property is named after the Russian probabilist AndreiAndreyevich Markov (1856-1922).

6 An informal mnemonic for remem-bering the Markov property is this. Given the present (Xk 1), thefuture (Xk) is independent of the past (Xk 2,Xk 3,..,X1). TheMarkov property is sometimes referred to as the lack of memory Processes that satisfy the Markov property are typicallymuch simpler to analyse than general Processes , and most of theprocesses that we shall study in this module are Markov course, in attempting to model any real system it will be impor-tant to consider whether the Markov property is likely to well as classifying time as discrete or continuous, we can also clas-sify the random variableXas discrete or continuous.

7 We shall seeMA636: Introduction to Stochastic processes1 3examples of all four combinations (discrete/continuous time in con-junction with discrete/continuous random variable) in this end this section with a few more definitions related to stochasticprocesses: Acounting processis a processX(t) in discrete or continuoustime for which the possible values ofX(t) are the natural num-bers (0,1,2,..) with the property thatX(t) is a non-decreasingfunction oft. Often,X(t) can be thought of as counting thenumber of events of some type that have occurred by basic example of a counting process is the Poisson process,which we shall study in some detail.

8 Asample pathof a Stochastic process is a particularrealisa-tionof the process, a particular set of valuesX(t) for allt(which may be discrete of continuous), generated according tothe ( Stochastic ) rules of the process. Theincrementsof a process are the changesX(t) X(s) be-tween time pointssandt(s<t). Processes in which the in-crements for non-overlapping time intervals are independent andstationary ( dependent only on the lengths of the time inter-vals, not the actual times) are of particular importance. Randomwalks, which we study extensively in Chapter 2, are a good ex-ample.

9 General Processes of this type are called L evy Processes ,and include the Poisson process (Chapter 5) and Brownian mo-tion (Chapter 6). Stochastic and deterministic modelsStochastic models can be contrasted withdeterministicmodels. Adeterministic model is specified by a set of equations that describeexactlyhow the system will evolve over time. In a Stochastic model,the evolution is at least partially random and if the process is runseveral times, it willnotgive identical results. Different runs of astochastic process are often calledrealisationsof the : Introduction to Stochastic processes1 4 Deterministic models are generally easier to analyse than stochasticmodels.

10 However, in many cases Stochastic models are more realistic,particulary for problems that involve small numbers . For example,suppose we are trying to model the management of a rare species,looking at how different strategies affect the survival of the models will not be very helpful here, because they willpredict that the species either definitely becomes extinct or definitelysurvives. In a Stochastic model, however, there will be a probability ofextinction, and we study how this is affected by management recent years, the distinction between deterministic and stochas-tic models has been blurred slightly by the development of chaoticmodels.