PARTICLE FILTER WITH MODE TRACKER(PF-MT) …

1 PARTICLEFILTERWITHMODETRACKER(PF-MT)FORV ISUALTRACKINGACROSSILLUMINATIONCHANGEAmi tKale*,NamrataVaswani**andChristopherJay nes** Ctr. StateUniversity, Lighting,Tracking,MonteCarlomethodsABSTR ACTI nrecentwork,theauthorsintroduceda multiplicative, lowdimen-sionalmodelofilluminationthatis computedasa linearcombina-tionofa cientsdescribingilluminationchange,areca nbecom-binedwiththe shape vectortode nea joint`shape Inthispaper, weutilizetherecentlyproposedPF-MTalgorit hmtoestimatetheilluminationvector. Thisis moti-vatedbythefactthat,exceptincaseofoc clusions,multimodalityofthestateposterio ris usuallyduetomultimodalityinthe shape vector( ).Inotherwords,giventhe shape vectorattimet, theposterioroftheillumination(probabilit ydistributionofilluminationconditionedon the shape andilluminationatprevi-oustime)is ,it is cientstobesolvedinclosedformasa solutionofa , thisgoalis ,changesinposeoftheobjectorillumina-tion cancausea assumptionshold,shapechangeforrigidobjec tscanbecapturedbya low dimensional shape vector(here shape referstolocationandscalechange,ingeneral canalsobeaf ne).

2 Trackingis theproblemofcausallyesti-matinga hiddenstatesequencecorrespondingto this shape vector,fXtg(thatis Markovianwithstatetransitionpdfp(XtjXt 1), froma sequenceofobservations,fYtg, thatsatisfytheHiddenMarkovModel(HMM)assu mption(Xt!Ytis a Markov chainforeacht,withobservationlikelihoodd enotedp(YtjXt)).ThisinterpretationThiswo rkwasfundedbyNSFCAREERA wardIIS-0092874andbyDepartmentofHomeland Securityformsthebasisofseveraltrackingal gorithmsincludingthewell-knownCondensati onalgorithm[6] similarlycon-cisemodelisrequiredifwearet orobustlyestimateilluminationchangesina functionofilluminationis a widelystudiedareaincomputervision[2,1]. surveillancewherea 3-Dmodelora largenumberofimagesofeveryobjecttobetrac kedunderdif-ferentilluminationconditions is unavailable[5].Examplesofsuchtasksthatin volve trackingobjectsthroughsimultaneousillumi nationand shape [2] [7], theauthorsintroduceda multiplicative, lowdimensionalmodelofilluminationthatisc omputedasa linearcombinationofa multi-plicative modelcanbeinterpretedasanapproximationof theillumi-nationimageasdiscussedinWeiss[ 12].)

3 Thebasiscoef cientsde-scribingilluminationchangecanbe combinedwiththe shape vec-tor( neorsimilaritygroup)to de nea joint shape-illumination shapespace ofdimensionNu= 3correspondingtox;ytranslationandscalean dthenumberofilluminationcoef cients,N = 7is suf cienttocapturea sig-ni cantvariabilityfromtheinitialtemplatetoi tsrepositionedandre-litcounterpartsinsuc cessive frames,weneedtosamplea is a wellknownfactthatasstatedimensionin-crea ses,theeffective particlesizereducesandhencemoreparticles areneededfora certainaccuracy. Thequestionis canwedobetterthanbruteforcePFona 10dimspace?We canutilizethefactthat,exceptincaseofoccl usions,multimodalityofthestateposteriori susuallyduetomultimodalityinthe shape vector( ).Orin otherwords,giventhe shape vectorat timet, theposterioroftheillumination(probabilit ydistributionofilluminationconditionedon the shape ,theimageandilluminationat theprevioustimeinstantis ,it is alsotruethatthisposterioris usuallyquitenarrow assumptions,wecanutilizethePF-MTalgo-rit hmproposedin[11, 10].)

4 Themainideais tosplittheentirestatevectorinto effective basis and residualspace .We runtheSIRPF(samplefromstatetransitionpdf )[3]ontheeffective basis, ,werunSIRPFon1 shape andwecomputethemodeoftheposteriorofillum inationconditionedonthe shape , (RB-PF)[9], butis moregeneralsinceit onlyrequiresthesub-systemtohave a unimodalposterior(neednotbelinearGaussia n).We wouldlike topointoutthoughthatforthespeci cobservationmodelconsideredinthispaper, ,if theobservationnoiseis non-Gaussianortheilluminationmodelisnonl inear, ,wealsoeffectivelyusetheAuxPF[8] toimprove resamplingef :Tt(x;y) =Lt(x;y)R(x;y)(1)whereLt(x;y)denotesthei lluminationimagein frametandR(x;y)denotesa xedre ectanceimage[12].Thusif theRisknown,trackingbecomestheproblemofe stimatingtheilluminationimageanda shape -vector. Ofcourse,Ris typicallyunavailableandtheilluminationim agecanonlybecomputedmodulotheilluminatio ncontainedintheimagetemplateT0,Lt(x;y) =~Lt(x;y)L0(x;y)R(x;y) =~Lt(x;y)T0(x;y)(2)whereL0istheinitialil luminationimageand~ [7],then,is theprod-uctofT0withanapproximationofLtwh ichis constructedusingalinearcombinationofa setofN Legendrebasisfunctionsde nedoverthetemplateofsize,M.

5 Letpk(x) ,forN = 2k+ 1, = [ 0; ; N ]T, thescaledintensityvalueat a pixel ofthetemplateTtis computedas:^Tt(x;y) = (1N ( 0+ 1p1(x) + + kpk(x) +(3) k+1p1(y) + + N pk(y)) + 1)T0(x;y)sothatwhen 0^Tt=T0. Forpurposesofnotation,wewilldenotetheeff ectof onT0as T0 T0 P +T0(4)whereP=26412n+1p0 12n+1pn(y1)..12n+1p0 12n+1pn(yM)375:(5)We de ne asanoperatorthatscalestherowsofPwiththec orre-spondingelementofT0writtenasa vector. Givena proposalimageregionGandT0, theLegendrecoef cientsthatrelightT0toresem-bleGcanbecomp utedbysolvingtheleastsquaresproblem:T0 P =AT0 T0 G(6)whereAT0,T0 (0thorderLegendrecoef cient)aspartoftheeffective illuminationmodelcanbecombinedwith shape tode nea joint shape-illumination vectorXt= ut t (7)whereut= [s txty]0correspondstoa threedimensional shape spaceencompassingscale(s) andtranslationstxandtyand 2RN correspondstocoef assumedtobea randomwalkmodelonobject shape ,utandonilluminationcoef cients, +1=ut+ ut; ut h(:)(8) t+1= t+ t; t N(0; )(9)where N N isa diagonalcovariancematrix(varianceofindi- vidualcomponentsof ) andh(:)denotesthepdfof utwhichisdescribedinSection( ).

6 Timet,Ytis theimageat timet. We assumethefollowingimageformationprocess: theimageintensitiesoftheimageregionthatc ontainstheobjectareillumina-tionscaledve rsionsoftheintensitiesoftheoriginaltempl ate,T0, ,theproposalsoftheimageregionthatcontain stheobjectareobtainedbyapplyingthedynami csoftheob-ject's shape ,therestoftheimage(whichdoesnotcontainth eobject)is independentoftheob-jectintensityor shape (andhencecanbethrownaway).Thuswehave thefollowingobservationmodel:Yt Jut+ X0Y0 = tT0+ t(10)where t N(0;V)whereVM Mis a diagonalcovariancematrix(varianceofindiv idualpixel noise)andJisJ= X0 x0110Y0 y0101 :(11)whereX0andY0denotethexandycoordinat esofeachpointonthetemplateand x0and edfortheaf necaseasdescribedin[6].Forbrevitywewilld enotetheimageregioninYtindicatedbya vectoruas:Gut=Yt Ju+ X0Y0 : (12)Thustheobservationlikelihoodcanbewri ttenas:p(YtjXt) =p(Ytjut; t) = exp[ jjGutt tT0jj2v](13)where(V)i;i= (PF-MT)ALGORITHMA naive approachwouldbetosimplyapplytheSIRPF[4] tothesystem(8),(9)andobservationmodel(10 ).

7 In(9)weuseak= 3or-derLegendrebasisandhenceN = 2k+1 = 7. Nt 1to Nt(Xt)=PNi=1w(i)n (Xt Xit),Xit= [ui; i] :Computegitusing(17)andresampleXit (wit 1)newde nedin(18). (IS)oneffectivebasis:8i, sample ut h(u)andcomputeuit=uit 1+ (MT)inresidualspace:8i, computemitusing(15)andset it= :Computewitusing(16).spacedimensionis is a wellknownfactthatthenumberofpar-ticlesre quiredfora certainaccuracy increaseswithstatedimension[4],makingthe PFveryexpensive , theposteriorof tis ,weobservedinexptsthatcovarianceofchange of [10] (IS) shape ,uitfromitsstatetran-sitionmodel(8),butr eplaceISbyposteriorModeTracking(MT)foril lumination, t, (denoteit asmit) ofp( tjuit; it 1;Yt)andset it=mit. InexactPF, onewouldcom-putemitandusea GaussianaboutmitastheISdensity. ReplacingISbyMTis a validapproximationwhenthethecovarianceis small[10]whichis , it is easytoseethatp( tjuit; it 1;Yt)/p(Ytjuit; t)p( tj it 1)(14)wherethe rsttermis de nedin(13)andthesecondtermis givenby(10).

8 Thusmitcanbecomputedastheminimizerofthe log[:]of(14)andthisturnsouttobea niceregularizedleastsquaresprob-lem(regu larizationtermis theweighteddistancefrom t 1) withaclosedformsolutiongivenbymit= it 1+ ( 1+ATT0V 1AT0) 1 ATT0V 1(Guitt tT0)(15)Noteallthemultiplierscanbepre-co mputed,makingthisa importancesamplingstrategy, theweightingtermwillbe[10]wit/wit 1p(Ytjuit; it)p( itj it 1); it=mit(16)Usingthismethodgreatlyreducest heweightvariance,thusreducingthenumberof particlesrequiredfora certainaccuracy (orimprovingtrackingaccuracy whennumberofparticlesavailableis small).Wehave shownthecomparisonofPF-MTwithotherexisti ngmethods- SIRPF(calledFULLPF),AuxiliaryPF(calledFU LLPFWAP)andPFwithouttrackingillumination (calledNOILLUM) improve resamplingef ciency, weusedthelook-aheadre-samplingideaofAuxi liaryPF[8].Thisperformsresamplingofthepa stparticleswhenthecurrentobservation,Ytc omesin,andusesthelikelihoodofXit 1generatingYttoresample, resamplesaccordingtogit=wit 1p(YtjXit 1)(17)Afterresampling,theweightsoftheres ampledparticlesaresetto(wit 1)new=wit 1 Ngit=p(YtjXit 1)N(18)01020304050020406080100120 FramesLocation Tracking Error PFMTNOILLUMFULLPFWAPFULLPF01020304050020 406080100120 FramesLocation Tracking Error PFMTNOILLUMFULLPFWAPFULLPFFig.

9 Ltersusing300particles(top)and100par-tic les(bottom).Thecompletealgorithmis ,fortheparticularformoftheobservationmod elthatweuse(additive Gaussiannoise),onecanalsouseRao-Blackwel lizedPF[9].Butiftheobservationnoiseweren on-Gaussian( ), ,incaseofocclusions,onemaywanttousetheme anintensity( 0)alsoaspartoftheeffective basis(importancesampleforit) needtolearnthenoisemodelsfor shape (h(:)),ofillumi-nation( ) andtheobservationnoisecovariance(V).We assumethatwehave a staticcameraacquiringimagesandthattheill umina-tionconditions,althoughvariablewit hinthescene,donotchangesigni startingtemplateT0anditslocationandshape insubsequentframes,Gt;t= 1; ;Nfareusedtocomputestate-vectorsXt=[ut t]T;t= 1; ;Nfusing(12)and(6)forthismotionwiththeco rrespondingapproximationsf^Gt= tT0;t= 1; ;NfgWe considerthe shape differencevectorsdut=ut ut 1fort= 1; ;Nf. Assumingthattheindividualcomponentsofdut areindependent,webuilda shape samplingdistributionh(u) ;t= 1:::Nthedynamicmodelwasesti-matedforthes hapevector.

10 Intoaccountthenatureofhumangaitthevertic aldis-placementdtywasmodeledasa mixtureoftwo shape samplingdistri-butionis givenbyh(u) = [N( s; 2s)N( tx; 2tx)2Xi=1 iN( i; i)]A thirdorderLegendrepolynomial(N = 2 3 + 1 = 7) t= t t 1fort= 1; ;Nfweestimate as =1Nf 1 NfXt=2( t t 1)( t t 1)T(19)Theper-pixel observationnoiseVis estimatedbyaveragingtheSSDbetweenthecorr espondingpixelsof^GtandGtasV=1 NfNfXt=1(^Gt Gt) (^Gt Gt)](20) (seeFigure2)includingoverhead,side-litan dpartiallyshadedregionsasthey approacha ~u=1 NPNi=1ui. Figure1 showsthelocationerrorfromthegroundtruthf ordifferentparticle tisim-portancesampledfromN( it 1; )andSIRis NOIL-LUMrepresentsthecasewherenoillumina tionmodelis ,theestimated shape remainsintrackevenwithjust100particles(b ottomrow ofFigure1). joint shape-illumination spaceintroducedin[7].We usedthePF-MTideatoexploitthefactthat,exc eptincaseofocclusions,multimodalityofthe stateposterioris usuallyduetomultimodalityinthe motion vectorandthatgiventhe motion vectorattimet, theposterioroftheillumination(probabilit ydistri-butionofilluminationconditionedo nthe motion ,theimageandilluminationattimet 1) is [1] , 25(2):218 233,2003.