Statistical Models of Appearance

1 BiomedicalEngineering,University of 8, 20041 Thisis an ongoingdraftof a report describingour work on Active Shape Models and Active will be expandedto becomemorecomprehensive, whenI get the the missingand incompletesectionsand the inconsistanciesin Overview52 Introduction63 Background94 StatisticalShape .. TrainingSet.. Variation.. Number of Modes .. Shape Models .. PDF.. NearestPlausibleShape .. model to NewPoints .. Well the model Generalises.. (shape) .. Models .. Models .. FEMM odes.. Prioron Covariance.. 285 StatisticalModelsof Texture.. Shape ParameterWeights .. :FacialAppearanceModel.. NewExample.. 321 CONTENTS26 .. Fit Function.. model Fit.. 357 Active Shape .. Structure.. Shape Models .. Search .. 418 Active .. AAMS earch .. CorrectModel Parameters.

2 TheFace model .. model .. model Re nement .. Active AppearanceModel Search .. Results.. Failure.. 529 .. Matching.. Point Positions.. Errors.. Constraints .. the Parameters.. of Points .. 6110 Variationson the .. UsingShape Parameters.. Experiments .. 6811 Alternatives and Extensionsto .. 70 CONTENTS312 Comparisonbetween ASMsand .. 7713 2D .. 3D .. 8114 .. Matching.. 9215 Applicationsand .. 9616 Discussion98A Applyinga PCAwhenthereare fewer samplesthandimensions101B AligningTwo 2D Case.. neCase.. 103C AligningShapes (II) (2D).. Scaling(2D).. (2D).. ne(2D).. 107D Representing2D TransformationCase.. 108E RepresentingTextureTransformations109 CONTENTS4F ne.. Dimensions.. 113G Density KernelMethod .. PDFfroma KernelEstimate.. edEM Algorithm.. 115H Implementation116 Chapter1 OverviewThe ultimategoalof machinevisionis imageunderstanding- the ability not onlyto recover imagestructurebut alsoto know whatit represents.

3 By de nition,thisinvolves the use of modelswhich describe andlabel the expectedstructureof the the pastdecade, model -basedvisionhas been appliedsuccessfullyto imagesof has proved muchmoredi cultto developmodel-basedapproaches to the interpretationof imagesof complexand variablestructuressuch as facesor the internalorgansof the humanbody (as visualisedinmedicalimages).In such casesit has even been problematicto recover imagestructurereliably,withouta model to organisethe oftennoisyand problemis thatof variability. To be useful,a model needsto be speci c- thatis, to be capableofrepresentingonly'legal'examples of the modelledobject(s).It has proved di cultto achievethiswhilstallowingfor naturalvariability. Recent developments have overcomethisproblem;it has been shown thatspeci cpatternsof variability inshape and grey-level appearancecan becapturedby statisticalmodelsthatcan be useddirectlyin describes methods of buildingmodelsof shape andappearance,andhowsuch modelscan be usedto majority of tasksto which machinevisionmight usefullybe appliedare 'hard'.

4 The exampleswe use in this work are frommedicalimageinterpretationand face recognition,thoughthe sameconsiderationsapplyto many mostobviousreasonfor the degreeof di cultyis thatmostnon-trivialapplicationsinvolve the needfor an automatedsystemto 'understand'the imageswithwhich it is presented - thatis, to recover imagestructureandknow whatit the use of modelswhich describe andlabel the expectedstructureof the alsotypicallycharacterisedby the needto dealwithcomplexand variablestructure- facesare a good example- and withimagesthatprovidenoisyand possiblyincompleteevidence- medicalimagesare a good example,whereit is oftenimpossibleto interpreta given imagewithoutpriorknowledgeof o erpotential solutionsto all thesedi problemcan,in principle,be usedto resolve the potential confusioncausedby structuralcomplexity, providetoleranceto noisyor missingdata,andprovidea meansof wouldlike to applyknowledgeof the expectedshapes of structures,theirspatialrelationships,and theirgrey-level appearanceto restrictour automatedsystemto 'plausible' particularinterestare generative Models - thatis, modelssu ciently completethattheyare ableto generaterealisticimagesof examplewouldbe a face model capableof generatingconvincingimagesof any individual,changingtheirexpressionand so on.

5 Usingsuch a model , imageinterpretationcan be formulatedas a matchingproblem:given an imageto interpret,structurescan be locatedand labelledby adjustingthemodel'sparametersin such a way thatit generatesan 'imaginedimage'which is as similaraspossibleto the real applicationsofteninvolve dealingwithclassesof objectswhich are not identical-for examplefaces- we needto dealwithvariability. Thisleadsnaturallyto the ideaof deformablemodels - Models which maintain the essential characteristicsof the classof objectstheyrepresent,but which can deformto t a rangeof two maincharacteristicswe wouldlike such modelsto ,theyshouldbe general- thatis, theyshouldbe capableofgeneratingany plausibleexampleof the classtheyrepresent. Second,and crucially, theyshouldbe speci c- thatis, theyshouldonlybe capableof generating'legal'examples- because,as wenotedearlier,the wholepoint of usinga model -basedapproach is to limitthe attentionof our67systemto orderto obtainspeci cmodelsof variableobjects,weneedto acquireknowledgeof how make use of a priormodel of whatis expectedin the image,andtypicallyattemptto ndthe best match of the model to the datain a the model ,one can thenmake measurements or testwhetherthe targetis is a `top-down' strategy, and di erssigni cantly from`bottom-up'or `data-driven' the latterthe imagedatais examinedat a low level, lookingfor localstructuressuch as edgesor regions,which are assembledinto groupsin an attemptto identifyobjectsof globalmodel of whatto expect.

6 Thisapproach is di cultandproneto widevariety of model basedapproaches have been explored(seethe reviewbelow). Thiswork will concentrateon a statisticalapproach, in which a model is builtfromanalysingtheappearanceof a set of in shape or texture,it is possibleto learnwhatare plausiblevariationsand whatare newimagecan be interprettedby ndingthe best plausiblematch of the model to the such amethod are that It is be appliedto many di erent problems,merelyby presentingdi erent trainingexamples. Expert knowledgecan be capturedin the systemin the annotationof the trainingexamples. Themodelsgive a compactrepresentationof allowablevariation,but are speci cenoughnot to allow arbitraryvariationdi erent fromthatseenin the trainingset. Thesystemneedmake few priorassumptionsabout the natureof the objectsbeingmod-elled,otherthanwhatit learnsfromthe trainingset.

7 (For instance,thereare no boundarysmoothnessparametersto be set.)Themodelsdescribed below requirea userto be ableto mark`landmark'points on each ofa set of trainingimagesin such a way thateach landmarkrepresents a distinguishablepointpresent on every instance,whenbuildinga model of the appearanceof aneye in a faceimage,good landmarkswouldbe the cornersof the eye, as thesewouldbe easytoidentify and markin each sortsof applicationsto which the methodcan be applied- it requiresthatthe topologyof the object cannotchangeand thatthe object isnot so amorphousthatno distinctlandmarkscan be rulesout suchthingsas cellsor simpleorganismswhich exhibitlargechangesin is in two rstdescribes buildingstatisticalmodelsof shapeand how thesemodelscan be usedto minimisinga costfunctionde ninghow well a particularinstanceof a modeldescribes the evidencein the approaches are described.

8 The rst,Active ShapeModels,manipulatesa shape model to describe the locationof structuresin a ,Active AppearanceModels(AAMs),manipulatea model cabableof synthesisingnewimagesof the object of nd the model parameterswhich8generatea syntheticimageas closeas possibleto the each casethe parametersof the best ttingmodel instancecan be usedfor furtherprocessing,such as for measurementor classi now a considerableliteratureon usingdeformablemodels to Imentiononlya few of the relevant works.(Oneday I may get the timeto extendthis reviewtobe morecomprehensive).The simplestmodel of an object is to use a typicalexampleas a `goldenimage'.A correlationmethod can be usedto match (or register)the goldenimageto a structuresinthe goldenimagehave been labelled,thismatch thengives the approximatepositionof thestructuresin the instance,one can determinethe approximatelocationsofmany structuresin an MR imageof a brainby registeringa standardimage,wherethe standardimagehas been suitablyannotatedby , the variability of both shapeand textureof mosttargetslimitsthe precisionof this to representingthe variationsobserved in an imageis to `hand-craft'a modelto solve the particularproblemcurrently instanceYuilleet al[129]buildup amodel of a humaneye usingcombinationsof parameterisedcirclesand canbe e ective it is complicated,and a completelynewsolutionis requiredfor every [111]represent the shapes of objectsin medicalimagesusingfourierdescriptorsof closedcurves.

9 The choiceof coe cients a ectsthe curve complexity. Placinglimitson each coe cient constrainsthe shape somewhatbut not in a systematicway. It can be shownthatsuch fouriermodelscan be madedirectlyequivalent to the statisticalmodelsdescribedbelow, but are not as instance,theycannoteasilyrepresent open al[65] introducedActive ContourModels (or `snakes') which are the originalformulationthe energyhas an internaltermwhich aimsto imposesmoothnesson the curve, andan externaltermwhich encouragesmovement toward particularlyusefulfor locatingthe outlineof , sinceno model (otherthansmoothness)is imposed,theyare not optimalfor locatingobjectswhich have a known statisticalapproaches are described by Grenanderet al [46] andMardiaet al[78].Theseare,however, di cultto use in [44]and Bookstein[8] use statisticaltechniquesfor morphometricanalysis,but do not addresstheproblemof and Sirovich [67] describe statisticalmodellingofgrey-level Appearance (particularlyfor faceimages)but do not addressshape morecomprehensive survey of deformablemodelsusedin medicalimageanalysisis givenin [81].

10 Variousapproaches to modellingvariability have been described. Themostcommonis toallow a prototype to vary accordingto Kovacic[1] describea volumemodel (of the brain)thatdeformselasticallyto al.[19, 18] describe a viscous ow model of deformationwhich theyalsoapplyto the brain,but is very computationallyexpensive. Parket. al.[91] andPentlandandSclaro [93] bothrepresent the outlinesor surfacesof prototype objectsusing niteelement methods and describevariability in termsof vibrationalmodes thoughthereis no guarantee thatthis is and Pentland[116]use principalcomponent analysisto describe the intensity patternsinface imagesin termsof a set of basisfunctions,or `eigenfaces'.Thoughvalid modes of variationare learnt froma trainingset, andare morelikely to be moreappropriatethana `physical' model , the representationis not robustto shape changes,and does not dealwell withvariabilityin pose and ,however, be matched to [39] [62] synthesizenewviewsof an objectfroma set of t the model to an unseenviewby a slow, but can be robustbecauseof the quality of the al[24]describe a 3D model of the grey-level surface,allowingfull synthesisof shape and , theydo not suggesta plausiblesearch algorithmto match the model to a al.