Stochastic Calculus, Filtering, and Stochastic Control

1 StochasticCalculus, Filtering, andStochast icControlLecture Notes(Thisversion:May29,2007)RamonvanHan delSpring2007 PrefaceTheselecturenoteswerewrittenforth ecourseACM217:AdvancedTopicsin Stochas-ticAnalysisat Caltech;thisyear(2007), anintroductorycourseonthesubject,andasth ereareonlysomany weeksina term, suf cientdodevelopa largeclassofinterestingmodels,andto developsomestochasticcontroland ,andmany otherfascinating(anduseful)topics,arelef tfora , thestochasticcontrolportionofthesenotesc oncentratesonveri- cationtheorems, hope,however, thattheinterestedreaderwillbeencouragedt oprobea littledeeperandultimatelytomove have noillusionsaboutthestateofthesenotes they werewrittenratherquickly,sometimesatther ateofa chaptera have nodoubtthatmany errorsremaininthetext;at theveryleastmany oftheproofsareextremelycompact,andshould bemadea littleclearerasis be ttingofa pedagogical(?)

2 I have anotheropportunitytoteachsucha course,I willgoover thenotesagainindetailandattemptthenecess arymodi ,however, youhave any commentsat allaboutthesenotes questions,suggestions,omis-sions,general comments,andparticularlymistakes Iwouldlove assumethatthereaderhashada basiccourseinprobabil-itytheoryat thelevel of,say, GrimmettandStirzaker [GS01] orhigher(ACM116/216shouldbesuf cient).Someelementarybackgroundinanalysi sis bitofanexperiment,butappearstohave , .. ,independence,andabsolutecontinuity.. technicaltool:Dynkin's -systemlemma..442 Conditioning,Martingales, :a trialrun.. :a multiscaleconstruction..854 TheIt o wrongwiththeStieltjesintegral?.. o integral.. o calculus .. 's theorem.

3 :existenceanduniqueness.. propertyandKolmogorov's equations.. therelifebeyondtheLipschitzcondition?.. cation: nitetimehorizon.. cation:inde nitetimehorizon.. cation:in nitetimehorizon.. chainapproximation.. lteringforstochasticdifferentialequation s.. lter.. lter.. messageovera noisychannel.. :themodi cationproblem.. ,stochasticcalculusprovidesa mathematicalfoundationforthetreatmentofe quationsthatinvolve ,bothmathematicalandpractical, , ,tracking,and nanceBrownianmotionandtheWienerprocessIn 1827,the(thenalready)famousScottishbotan istRobertBrownobserveda rathercuriousphenomenon[Bro28]. Brownwasinterestedinthetiny particlesfoundinsidegrainsofpollen, ,it appearedthattheparticleswerecon-stantlyj itteringaroundin the rstBrownthoughtthattheparticleswerealive ,buthewasabletoruleoutthishypothesisafte rheobservedthesamephenomenonwhenusinggla sspowder, anda largenumberofotherinorganicsubstances, 's observationwasnotprovideduntilthepublica tionofEinstein's famous1905paper[Ein05].

4 Einstein's argumentreliesonthefactthatthe uid,inwhichthepollenparti-clesaresuspend ed,consistsofa giganticnumberofwatermolecules(thoughthi sisnow undisputed,theatomichypothesiswashighlyc ontroversialatthetime).Asthe uidis at a nitetemperature,kinetictheorysuggeststha tthevelocityofeverywatermoleculeis randomlydistributedwithzeromeanvalue(the lattermustbethecase,asthetotal uidhasnonetvelocity)andis weplacea pollenparticleinthe uid,thenineverytimeinter-valtheparticlew illbebombardedbya largenumberofwatermolecules,givingita shouldwegoaboutmodellingthisphenomenon?T hefollowingprocedure,whichis a somewhatmodernizedversionofEinstein's argument,is bombardedbyNwatermoleculesperunittime,an dthateverywatermoleculecontributesaninde pen-dent,identicallydistributed( )randomdisplacement ntotheparticle(where (N)fortheBrownianmotionmodelinthetext,wi th(fromlefttoright)N= 20;50; narechosentoberandomvariableswhichtake thevalues N 1= ).

5 Thenat timet, thepositionxt(N)ofthepollenparticleis givenbyxt(N) =x0+bN tcXn=1 n:We wanttoconsiderthelimitwherethenumberofbo mbardmentsNis verylarge,butwhereeveryindividualwatermo leculeonlycontributesa tiny displacementtothepollenparticle thisis a reasonableassumption,asthepollenparticle ,whilebeingsmall,is extremelylargecomparedtoa beconcrete,letusde nea constant byvar( n) = N 1. Notethat is preciselythemean-squaredisplacementofthe pollenparticleperunittime:E(x1(N) x0)2= var NXn=1 n!=Nvar( n) = :Thephysicalregimein whichweareinterestednow correspondsto thelimitN!1, ,wherethenumberofcollisionsNis largebutthemean-squaredisplacementperuni ttime remains (N) =x0+p tPbN tcn=1 npNt;where n= npN= ,weseethatthelimitingbehaviorofxt(N)asN!

6 1is describedbythecentrallimittheorem:we ndthatthelaw ofxt(N)convergestoa Gaussiandistributionwithzeromeanandvaria nce t. Thisis indeedtheresultofEinstein's ! 1is knownasBrownianmo-tion. Youcangetsomeideaofwhatxt(N)lookslike forincreasinglylargeNbyhavinga lookat Butnowwecometoour rstsigni cantmathematicalproblem:doesthelimitofth estochasticprocesst7!xt(N)asN! 1evenexistina suitablesense?Thisis notatallobvious(wehave onlyshownconvergenceinIntroduction3distr ibutionfor xedtimet), noris wecanmake nosenseofthislimit,therewouldbenomathema ticalmodelofBrownianmotion(aswehave de nedit);andinthiscase, mathematicalsenseofBrownianmotion(chapte r3), whichwas rstdoneinthefundamentalworkofNorbertWien er[Wie23].

7 Thelimitingstochasticprocessxt(with = 1) is knownastheWienerprocess, andplaysa diffusingparticleUsingonlythenotionofa Wienerprocess, ,like RobertBrown, ,wewouldlike tozoominononeoftheparticles ,wewouldlike toincreasethemagni cationofthemicroscopeuntilonepollenparti cle llsa largepartofthe eldofview. Whenwedothis,however, theBrownianmotionbecomesa bitofa nuisance;therandommotionofthepollenparti clecausesit torapidlyleave our eldofview. If wewanttokeeplookingat thepollenparticlefora reasonableamountoftime,wehave to dealwiththisproblem,weattachanelectricmo tortothemicroscopeslidewhichallowsustomo ve ; thenwecanwritedztdt= ut;whereutis thevoltageappliedto themotorand >0is a totheslideis modelledbya Wienerprocessxt, sothatthepositionoftheparticlerelative tothemicroscopefocusis givenbyxt+zt.

8 Wewouldlike tocontroltheslidepositiontokeepthepartic leinfocus, ,it is ourgoaltochooseutinorderthatxt+ formalizethisproblem,wecouldintroducethe followingcostfunctional:JT[u] =pE"1 TZT0(xt+zt)2dt#+qE"1 TZT0u2tdt#;wherepandqaresomepositive rstterminthisexpressionisthetime-average (onsometimeinterval[0;T]) meansquaredistanceoftheparticlefromthefo cusofthemicroscope:clearlywewouldlike ,ontheotherhand,is theaveragepowerinthecontrolsignal,whichs houldalsonotbetoolargeinany realisticapplication(ourelectricmotorwil lonlytake somuch).Thegoaloftheoptimalcontrolproble misto ndthefeedbackstrategyutwhichminimizesthe costJT[u]. Many variationsonthisproblemarepossible;forex ample,if wearenotinterestedina particulartimehorizon[0;T], wecouldtrytominimizeJ1[u] =plimsupT!

9 1E"1 TZT0(xt+zt)2dt#+qlimsupT!1E"1 TZT0u2tdt#:Introduction4 Thetradeoff betweenthecon ictinggoalsofminimizingthedistanceofthep articletothefocusofthemicroscopeandminim izingthefeedbackpowercanbeselectedbymodi fyingtheconstantsp;q. Theoptimalcontroltheoryfurtherallowsusto studythistradeoff explicitly:forexample,wecancalculatetheq uantityC(U) = inf(limsupT!1E"1 TZT0(xt+zt)2dt#: limsupT!1E"1 TZT0u2tdt# U); ,C(U)isthesmallesttime-averagetrackinger rorthatisachievableusingcon-trolswhoseti me-averagepoweris atmostU. Thisgivesa ,ina muchmoregeneralcontext,is :how toinvestyourmoneyThoughthetheoreticalide asbehindBrownianmotionareoftenattributed toEinstein,thesamemodelwasdevelopedsever alyearsearlierina completelydifferentcon-textbytheFrenchma thematicianLouisBachelier[Bac00].

10 1 Bachelierwasinterestedinspeculationonren tes(Frenchgovernmentbonds),andintroduced theBrownianmotiontomodelthe 's workformsthefoun-dationformuchofthemoder ntheoryofmathematical nance,thoughhisworkwasvirtuallyunknownto economistsformorethanhalfa century. Thesedaysmathe-matical nanceis animportantapplicationareaforstochastica nalysisandstochasticcontrol,andprovidesa nancetheirop-erations;themoney madefromthesaleofstockcanthenbeusedbythe company to nanceproduction,specialprojects, ,a certainportionofthepro tmadebythecompany is periodicallypaidouttotheshareholders(peo plewhoownstockinthecompany). thecompany is doingwell( ,if salesaresoaring),thenowningstockinthecom pany is likelytobepro onlythebeginningofthestory, however.