Stochastic Calculus, Filtering, and Stochastic Control

1 StochasticCalculus, Filtering, andStochast icControlLecture notes (Thisversion:May29,2007)RamonvanHan delSpring2007 PrefaceTheselecturenoteswerewrittenforth ecourseACM217:AdvancedTopicsin Stochas-ticAnalysisat Caltech;thisyear(2007), anintroductorycourseonthesubject,andasth ereareonlysomany weeksina term, suf cientdodevelopa largeclassofinterestingmodels,andto developsomestochasticcontroland ,andmany otherfascinating(anduseful)topics,arelef tfora , thestochasticcontrolportionofthesenotesc oncentratesonveri- cationtheorems, hope,however, thattheinterestedreaderwillbeencouragedt oprobea littledeeperandultimatelytomove have noillusionsaboutthestateofthesenotes they werewrittenratherquickly,sometimesatther ateofa chaptera have nodoubtthatmany errorsremaininthetext;at theveryleastmany oftheproofsareextremelycompact,andshould bemadea littleclearerasis be ttingofa pedagogical(?) I have anotheropportunitytoteachsucha course,I willgoover thenotesagainindetailandattemptthenecess arymodi ,however, youhave any commentsat allaboutthesenotes questions,suggestions,omis-sions,general comments,andparticularlymistakes Iwouldlove assumethatthereaderhashada basiccourseinprobabil-itytheoryat thelevel of,say, GrimmettandStirzaker [GS01] orhigher(ACM116/216shouldbesuf cient).

2 Someelementarybackgroundinanalysisis bitofanexperiment,butappearstohave , .. ,independence,andabsolutecontinuity.. technicaltool:Dynkin's -systemlemma..442 Conditioning,Martingales, :a trialrun.. :a multiscaleconstruction..854 TheIt o wrongwiththeStieltjesintegral?.. o integral.. o calculus .. 's theorem.. :existenceanduniqueness.. propertyandKolmogorov's equations.. therelifebeyondtheLipschitzcondition?.. cation: nitetimehorizon.. cation:inde nitetimehorizon.. cation:in nitetimehorizon.. chainapproximation.. lteringforstochasticdifferentialequation s.. lter.. lter.. messageovera noisychannel.. :themodi cationproblem.. ,stochasticcalculusprovidesa mathematicalfoundationforthetreatmentofe quationsthatinvolve ,bothmathematicalandpractical, , ,tracking,and nanceBrownianmotionandtheWienerprocessIn 1827,the(thenalready)famousScottishbotan istRobertBrownobserveda rathercuriousphenomenon[Bro28]. Brownwasinterestedinthetiny particlesfoundinsidegrainsofpollen, ,it appearedthattheparticleswerecon-stantlyj itteringaroundin the rstBrownthoughtthattheparticleswerealive ,buthewasabletoruleoutthishypothesisafte rheobservedthesamephenomenonwhenusinggla sspowder, anda largenumberofotherinorganicsubstances, 's observationwasnotprovideduntilthepublica tionofEinstein's famous1905paper[Ein05].

3 Einstein's argumentreliesonthefactthatthe uid,inwhichthepollenparti-clesaresuspend ed,consistsofa giganticnumberofwatermolecules(thoughthi sisnow undisputed,theatomichypothesiswashighlyc ontroversialatthetime).Asthe uidis at a nitetemperature,kinetictheorysuggeststha tthevelocityofeverywatermoleculeis randomlydistributedwithzeromeanvalue(the lattermustbethecase,asthetotal uidhasnonetvelocity)andis weplacea pollenparticleinthe uid,thenineverytimeinter-valtheparticlew illbebombardedbya largenumberofwatermolecules,givingita shouldwegoaboutmodellingthisphenomenon?T hefollowingprocedure,whichis a somewhatmodernizedversionofEinstein's argument,is bombardedbyNwatermoleculesperunittime,an dthateverywatermoleculecontributesaninde pen-dent,identicallydistributed( )randomdisplacement ntotheparticle(where (N)fortheBrownianmotionmodelinthetext,wi th(fromlefttoright)N= 20;50; narechosentoberandomvariableswhichtake thevalues N 1= ).Thenat timet, thepositionxt(N)ofthepollenparticleis givenbyxt(N) =x0+bN tcXn=1 n:We wanttoconsiderthelimitwherethenumberofbo mbardmentsNis verylarge,butwhereeveryindividualwatermo leculeonlycontributesa tiny displacementtothepollenparticle thisis a reasonableassumption,asthepollenparticle ,whilebeingsmall,is extremelylargecomparedtoa beconcrete,letusde nea constant byvar( n) = N 1.

4 Notethat is preciselythemean-squaredisplacementofthe pollenparticleperunittime:E(x1(N) x0)2= var NXn=1 n!=Nvar( n) = :Thephysicalregimein whichweareinterestednow correspondsto thelimitN!1, ,wherethenumberofcollisionsNis largebutthemean-squaredisplacementperuni ttime remains (N) =x0+p tPbN tcn=1 npNt;where n= npN= ,weseethatthelimitingbehaviorofxt(N)asN! 1is describedbythecentrallimittheorem:we ndthatthelaw ofxt(N)convergestoa Gaussiandistributionwithzeromeanandvaria nce t. Thisis indeedtheresultofEinstein's ! 1is knownasBrownianmo-tion. Youcangetsomeideaofwhatxt(N)lookslike forincreasinglylargeNbyhavinga lookat Butnowwecometoour rstsigni cantmathematicalproblem:doesthelimitofth estochasticprocesst7!xt(N)asN! 1evenexistina suitablesense?Thisis notatallobvious(wehave onlyshownconvergenceinIntroduction3distr ibutionfor xedtimet), noris wecanmake nosenseofthislimit,therewouldbenomathema ticalmodelofBrownianmotion(aswehave de nedit);andinthiscase, mathematicalsenseofBrownianmotion(chapte r3), whichwas rstdoneinthefundamentalworkofNorbertWien er[Wie23].

5 Thelimitingstochasticprocessxt(with = 1) is knownastheWienerprocess, andplaysa diffusingparticleUsingonlythenotionofa Wienerprocess, ,like RobertBrown, ,wewouldlike tozoominononeoftheparticles ,wewouldlike toincreasethemagni cationofthemicroscopeuntilonepollenparti cle llsa largepartofthe eldofview. Whenwedothis,however, theBrownianmotionbecomesa bitofa nuisance;therandommotionofthepollenparti clecausesit torapidlyleave our eldofview. If wewanttokeeplookingat thepollenparticlefora reasonableamountoftime,wehave to dealwiththisproblem,weattachanelectricmo tortothemicroscopeslidewhichallowsustomo ve ; thenwecanwritedztdt= ut;whereutis thevoltageappliedto themotorand >0is a totheslideis modelledbya Wienerprocessxt, sothatthepositionoftheparticlerelative tothemicroscopefocusis givenbyxt+zt. Wewouldlike tocontroltheslidepositiontokeepthepartic leinfocus, ,it is ourgoaltochooseutinorderthatxt+ formalizethisproblem,wecouldintroducethe followingcostfunctional:JT[u] =pE"1 TZT0(xt+zt)2dt#+qE"1 TZT0u2tdt#;wherepandqaresomepositive rstterminthisexpressionisthetime-average (onsometimeinterval[0;T]) meansquaredistanceoftheparticlefromthefo cusofthemicroscope:clearlywewouldlike ,ontheotherhand,is theaveragepowerinthecontrolsignal,whichs houldalsonotbetoolargeinany realisticapplication(ourelectricmotorwil lonlytake somuch).

6 Thegoaloftheoptimalcontrolproblemisto ndthefeedbackstrategyutwhichminimizesthe costJT[u]. Many variationsonthisproblemarepossible;forex ample,if wearenotinterestedina particulartimehorizon[0;T], wecouldtrytominimizeJ1[u] =plimsupT!1E"1 TZT0(xt+zt)2dt#+qlimsupT!1E"1 TZT0u2tdt#:Introduction4 Thetradeoff betweenthecon ictinggoalsofminimizingthedistanceofthep articletothefocusofthemicroscopeandminim izingthefeedbackpowercanbeselectedbymodi fyingtheconstantsp;q. Theoptimalcontroltheoryfurtherallowsusto studythistradeoff explicitly:forexample,wecancalculatetheq uantityC(U) = inf(limsupT!1E"1 TZT0(xt+zt)2dt#: limsupT!1E"1 TZT0u2tdt# U); ,C(U)isthesmallesttime-averagetrackinger rorthatisachievableusingcon-trolswhoseti me-averagepoweris atmostU. Thisgivesa ,ina muchmoregeneralcontext,is :how toinvestyourmoneyThoughthetheoreticalide asbehindBrownianmotionareoftenattributed toEinstein,thesamemodelwasdevelopedsever alyearsearlierina completelydifferentcon-textbytheFrenchma thematicianLouisBachelier[Bac00].

7 1 Bachelierwasinterestedinspeculationonren tes(Frenchgovernmentbonds),andintroduced theBrownianmotiontomodelthe 's workformsthefoun-dationformuchofthemoder ntheoryofmathematical nance,thoughhisworkwasvirtuallyunknownto economistsformorethanhalfa century. Thesedaysmathe-matical nanceis animportantapplicationareaforstochastica nalysisandstochasticcontrol,andprovidesa nancetheirop-erations;themoney madefromthesaleofstockcanthenbeusedbythe company to nanceproduction,specialprojects, ,a certainportionofthepro tmadebythecompany is periodicallypaidouttotheshareholders(peo plewhoownstockinthecompany). thecompany is doingwell( ,if salesaresoaring),thenowningstockinthecom pany is likelytobepro onlythebeginningofthestory, however. Any individualwhoownsstockina company candecidetosellhisstockona ,thegoingratefora particularstockdependsonhow wellthecompany is doing(orisexpectedtodointhefuture).Whent hecompany is doingwell,many peoplewouldlike toownstock(afterall,thereis a prospectoflargedividends) is notdoingwell,however, it is likelythatmoreshareholdersarewillingtose llthanthereis demandforthestock,sothatthemarket priceofthestockis these market forces ,thestockpricestendto uctuaterandomlyinthecourseoftime;see,for example, Evenif weignoredividends(whichwewilldotosimplif ymatters),wecanstilltrytomake money onthestockmarketbybuyingstockwhenthepric eis low andsellingwhenthepriceis ,how shouldweinvestourmoney in thestockmarket to maximizeourpro t?

8 1 Anexcellentannotatedtranslationhasrecent lyappearedin[DE06].Introduction520052000 19951990198519801975197001020304050 Stock Price ($)2007200620052004200320022001200019991 0203040 Stock Price ($)McDonald s Corporation (MCD) Stock Prices 1970-2007 Figure priceofMcDonald's stockontheNew YorkStockExchange(NYSE)overtheperiod1999 2007(upperplot)and1970 2007(lowerplot).Thestockpricedataforthis gurewasobtainedfromYahoo! essentiallya stochasticcontrolproblem,whichwastackled ina famouspaperbyMerton[Mer71]. A differentclassofquestionsconcernsthepric ingofoptions asortof insurance issuedonstock andsimilar [BS73] andis variationsontheseandothertopics,butthey have at ,letusconsiderhow to builda mathematicalmodelforthestockprices a thatthestockpricesarenotguaranteedtobepo sitive, whichis unrealistic;afterall,nobodypaysmoney intoaccountis thateventhoughthisis notasvisibleonshortertimescales,stockpri cestendtogrowexponentiallyonthelongrun:s ee inthebank,whichis a goodreasontoinvestinstock(investinginsto ckis notthesameasgamblingat a casino!)

9 Thissuggeststhefollowingmodelforstockpri ces,whichis widelyused:thepriceStat timetofa singleunitofstockis givenbySt=S0exp 22 t+ Wt ;whereWtisa Wienerprocess,andS0>0(theinitialprice), >0(thereturnrate),and >0(thevolatility) NotethatStisalwayspositive,andmoreoverE( St) =S0e t(exercise:youshouldalreadybeabletoverif ythis!)Henceevidently,Introduction6onave rage,thestockmakesmoney atrate . Inpractice,however, thepricemay uctuatequitefarawayfromtheaverageprice,a sdeterminedbymagnitudeofthevolatility . Thismeansthatthereisa probabilitythatwewillmake money ataratemuchfasterthan , butwecanalsomake money slowerorevenlosemoney. Inessence,a stockwithlargevolatilityis a risky investment,whereasa stockwithsmallvolatilityis a nanceareappliedtoreal-worldtrading,param eterssuchas and areoftenestimatedfromrealstockmarket data(like thedatashownin ).BesideinvestinginthestockSt, wewillalsosupposethatwehave theoptionofputtingourmoney xedinterestrater >0onourmoney:thatis,if weinitiallyputR0dollarsinthebank,thenat timetwewillhaveRt=R0exp(rt) willbethecasethatr < ; thatis,investinginthestockwillmake usmoremoney, onaverage,thanif weputourmoney ,investinginstockis risky:thereis some niteprobabilitythatwewillmake lessmoney thanif a modelforthestockpricesandforthebank,wene edtobeabletocalculatehowmuchmoney wemake usinga aregoingto investsomefraction 0ofthismoney instock,andtherestinthebank( ,weput(1 0)X0dollarsinthebank,andwebuy 0X0=S0unitsofstock).

10 Thenattimet, ourtotalwealthXt(indollars)amountstoXt= 0X0e( 2=2)t+ Wt+ (1 0)X0ert:Now supposethatat thistimewedecidetoreinvestourcapital; ,wenow investafraction tofournewlyaccumulatedwealthXtinthestock (wemightneedtoeitherbuyorsellsomeofourst ocktoensurethis), sometimet0> t, ourtotalwealthbecomesXt0= tXte( 2=2)(t0 t)+ (Wt0 Wt)+ (1 t)Xter(t0 t):Similarly, if wechoosetoreinvestat thetimes0 =t0< t1< < tN 1< tN=t,thenwe ndthatourwealthat timetis givenbyXt=X0 NYn=1h tn 1e( 2=2)(tn tn 1)+ (Wtn Wtn 1)+ (1 tn 1)er(tn tn 1)i:Inprinciple,however, weshouldbeabletodecideatany pointintimewhatfractionofourmoney toinvestinstock; ,toallowforthemostgeneraltradingstrategi esweneedtogeneralizetheseexpressionstoth ecasewhere :wearemissinga key mathematicalingredient,thestochasticcalc ulus(chapters4and5).Oncewehave developedthelatter, ,corruptedsignals,andnoise-drivensystems WhitenoiseandtheWienerprocessDespitethef actthatitsmathematicaldescriptionis somewhatelusive, thenotionofwhitenoiseis ,youhave willrevisitit now andexhibitsomeoftheassociateddif somediscretetimemessagefangwhichwewouldl ike totransmittoa receiver.