Stochastic Calculus, Filtering, and Stochastic Control

1 StochasticCalculus, Filtering, andStochast icControlLecture Notes(Thisversion:May29,2007)RamonvanHan delSpring2007 PrefaceTheselecturenoteswerewrittenforth ecourseACM217:AdvancedTopicsin Stochas-ticAnalysisat Caltech;thisyear(2007), anintroductorycourseonthesubject,andasth ereareonlysomany weeksina term, suf cientdodevelopa largeclassofinterestingmodels,andto developsomestochasticcontroland ,andmany otherfascinating(anduseful)topics,arelef tfora , thestochasticcontrolportionofthesenotesc oncentratesonveri- cationtheorems, hope,however, thattheinterestedreaderwillbeencouragedt oprobea littledeeperandultimatelytomove have noillusionsaboutthestateofthesenotes they werewrittenratherquickly,sometimesatther ateofa chaptera have nodoubtthatmany errorsremaininthetext;at theveryleastmany oftheproofsareextremelycompact,andshould bemadea littleclearerasis be ttingofa pedagogical(?)

2 I have anotheropportunitytoteachsucha course,I willgoover thenotesagainindetailandattemptthenecess arymodi ,however, youhave any commentsat allaboutthesenotes questions,suggestions,omis-sions,general comments,andparticularlymistakes Iwouldlove assumethatthereaderhashada basiccourseinprobabil-itytheoryat thelevel of,say, GrimmettandStirzaker [GS01] orhigher(ACM116/216shouldbesuf cient).Someelementarybackgroundinanalysi sis bitofanexperiment,butappearstohave , .. ,independence,andabsolutecontinuity.

3 Technicaltool:Dynkin's -systemlemma..442 Conditioning,Martingales, :a trialrun.. :a multiscaleconstruction..854 TheIt o wrongwiththeStieltjesintegral?.. o integral.. o calculus .. 's theorem.. :existenceanduniqueness.. propertyandKolmogorov's equations.. therelifebeyondtheLipschitzcondition?.. cation: nitetimehorizon.. cation:inde nitetimehorizon.. cation:in nitetimehorizon.. chainapproximation.. lteringforstochasticdifferentialequation s.. lter.. lter.. messageovera noisychannel.

4 :themodi cationproblem.. ,stochasticcalculusprovidesa mathematicalfoundationforthetreatmentofe quationsthatinvolve ,bothmathematicalandpractical, , ,tracking,and nanceBrownianmotionandtheWienerprocessIn 1827,the(thenalready)famousScottishbotan istRobertBrownobserveda rathercuriousphenomenon[Bro28]. Brownwasinterestedinthetiny particlesfoundinsidegrainsofpollen, ,it appearedthattheparticleswerecon-stantlyj itteringaroundin the rstBrownthoughtthattheparticleswerealive ,buthewasabletoruleoutthishypothesisafte rheobservedthesamephenomenonwhenusinggla sspowder, anda largenumberofotherinorganicsubstances, 's observationwasnotprovideduntilthepublica tionofEinstein's famous1905paper[Ein05].

5 Einstein's argumentreliesonthefactthatthe uid,inwhichthepollenparti-clesaresuspend ed,consistsofa giganticnumberofwatermolecules(thoughthi sisnow undisputed,theatomichypothesiswashighlyc ontroversialatthetime).Asthe uidis at a nitetemperature,kinetictheorysuggeststha tthevelocityofeverywatermoleculeis randomlydistributedwithzeromeanvalue(the lattermustbethecase,asthetotal uidhasnonetvelocity)andis weplacea pollenparticleinthe uid,thenineverytimeinter-valtheparticlew illbebombardedbya largenumberofwatermolecules,givingita shouldwegoaboutmodellingthisphenomenon?

6 Thefollowingprocedure,whichis a somewhatmodernizedversionofEinstein's argument,is bombardedbyNwatermoleculesperunittime,an dthateverywatermoleculecontributesaninde pen-dent,identicallydistributed( )randomdisplacement ntotheparticle(where (N)fortheBrownianmotionmodelinthetext,wi th(fromlefttoright)N= 20;50; narechosentoberandomvariableswhichtake thevalues N 1= ).Thenat timet, thepositionxt(N)ofthepollenparticleis givenbyxt(N) =x0+bN tcXn=1 n:We wanttoconsiderthelimitwherethenumberofbo mbardmentsNis verylarge,butwhereeveryindividualwatermo leculeonlycontributesa tiny displacementtothepollenparticle thisis a reasonableassumption,asthepollenparticle ,whilebeingsmall,is extremelylargecomparedtoa beconcrete,letusde nea constant byvar( n) = N 1.

7 Notethat is preciselythemean-squaredisplacementofthe pollenparticleperunittime:E(x1(N) x0)2= var NXn=1 n!=Nvar( n) = :Thephysicalregimein whichweareinterestednow correspondsto thelimitN!1, ,wherethenumberofcollisionsNis largebutthemean-squaredisplacementperuni ttime remains (N) =x0+p tPbN tcn=1 npNt;where n= npN= ,weseethatthelimitingbehaviorofxt(N)asN! 1is describedbythecentrallimittheorem:we ndthatthelaw ofxt(N)convergestoa Gaussiandistributionwithzeromeanandvaria nce t. Thisis indeedtheresultofEinstein's !

8 1is knownasBrownianmo-tion. Youcangetsomeideaofwhatxt(N)lookslike forincreasinglylargeNbyhavinga lookat Butnowwecometoour rstsigni cantmathematicalproblem:doesthelimitofth estochasticprocesst7!xt(N)asN! 1evenexistina suitablesense?Thisis notatallobvious(wehave onlyshownconvergenceinIntroduction3distr ibutionfor xedtimet), noris wecanmake nosenseofthislimit,therewouldbenomathema ticalmodelofBrownianmotion(aswehave de nedit);andinthiscase, mathematicalsenseofBrownianmotion(chapte r3), whichwas rstdoneinthefundamentalworkofNorbertWien er[Wie23].

9 Thelimitingstochasticprocessxt(with = 1) is knownastheWienerprocess, andplaysa diffusingparticleUsingonlythenotionofa Wienerprocess, ,like RobertBrown, ,wewouldlike tozoominononeoftheparticles ,wewouldlike toincreasethemagni cationofthemicroscopeuntilonepollenparti cle llsa largepartofthe eldofview. Whenwedothis,however, theBrownianmotionbecomesa bitofa nuisance;therandommotionofthepollenparti clecausesit torapidlyleave our eldofview. If wewanttokeeplookingat thepollenparticlefora reasonableamountoftime,wehave to dealwiththisproblem,weattachanelectricmo tortothemicroscopeslidewhichallowsustomo ve ; thenwecanwritedztdt= ut;whereutis thevoltageappliedto themotorand >0is a totheslideis modelledbya Wienerprocessxt, sothatthepositionoftheparticlerelative tothemicroscopefocusis givenbyxt+zt.

10 Wewouldlike tocontroltheslidepositiontokeepthepartic leinfocus, ,it is ourgoaltochooseutinorderthatxt+ formalizethisproblem,wecouldintroducethe followingcostfunctional:JT[u] =pE"1 TZT0(xt+zt)2dt#+qE"1 TZT0u2tdt#;wherepandqaresomepositive rstterminthisexpressionisthetime-average (onsometimeinterval[0;T]) meansquaredistanceoftheparticlefromthefo cusofthemicroscope:clearlywewouldlike ,ontheotherhand,is theaveragepowerinthecontrolsignal,whichs houldalsonotbetoolargeinany realisticapplication(ourelectricmotorwil lonlytake somuch).