Fast-FACS: A Computer-Assisted System to …

1 Fast-FACS: A Computer-Assisted System to IncreaseSpeed and Reliability of Manual FACS CodingFernando De la Torre1,TomasSimon1, Zara Ambadar2, and Jeffrey F. Cohn21 Robotics Institute, Carnegie Mellon university , pittsburgh , PA 15213, USA2 university of pittsburgh , pittsburgh , PA 15260, (Facial Action Coding System ) coding is the state of the art inmanual measurement of facial actions. FACS coding, however, is labor intensiveand difficult to standardize. A goal of automated FACS coding is to eliminate theneed for manual coding and realize automatic recognition and analysis of facialactions. Success of this effort dependsin part on access to reliably coded corpora;however, manual FACS coding remains expensive and slow. This paper proposesFast-FACS, a computer vision aided System that improves speed and reliabilityof FACS coding. Three are the main novelties of the System : (1) to the best ofour knowledge, this is the first paper to predict onsets and offsets from peaks, (2)use Active Appearance Models for computer assisted FACS coding, (3) learn anoptimal metric to predict onsets and offsets from peaks.

2 The System was tested inthe RU-FACS database, which consists of natural facial behavior during a two-person interview. Fast-FACS reduced manual coding time by nearly 50% anddemonstrated strong concurrent validity with manual FACS :Facial Action Coding System , Action Unit IntroductionFACS (Facial Action Coding System [1]) coding is the state of the art in manual mea-surement of facial action. FACS coding, however, is labor intensive and difficult tostandardize across coders. A goal of automated FACS coding [2,3,4] is to eliminatethe need for manual coding and realize automatic recognition and analysis of facial of this effort depends on access toreliably coded corporaof FACS-codedimages from well-chosen observational scenarios. Completing the necessary FACS cod-ing for training and testing algorithms has been a rate-limiter. Manual FACS codingremains expensive and inefficiency of current approaches for FACS coding is not inherent to FACSbut to the failure to make use of technology to make coders more productive.

3 Thispaper proposes an hybrid System , Fast-FACS, that combines automated facial imageprocessing with manual coding to increase the speed and reliability of FACS 1 shows the main idea of the paper. The specific aims are to: (1) Reduce time andeffort required for manual FACS coding by using novel computer vision and machinelearning techniques. (2) increase reliability of FACS coding by increasing the internalconsistency of manual FACS coding. (3) Develop an intuitive graphical user interfacethat is comparable to commercially available packages in ease of use, while enablingfast reliable D Mello et al. (Eds.): ACII 2011, Part I, LNCS 6974, pp. 57 66, Springer-Verlag Berlin Heidelberg 201158F. De la Torre et Unit Manual FACS codingFastFACSO fffsffetOnsetFig. coding typically involves frame-by-frame inspection of the video, paying closeattention to subtle cues such as wrinkles, bulges, and furrows.

4 Left to right, evolution of an AU12 (involved in smiling), from onset, peak, to offset. Using FastFACS only the peak needs to belabeled and the onset/offset are estimated Facial Action Coding System (FACS)FACS [1] is a comprehensive, anatomically-based System for measuring nearly all vi-sually discernible facial movement. FACS describes facial activity on the basis of 44unique action units (AUs), as well as several categories of head and eye positions andmovements. Facial movement is thus described in terms of constituent components, orAUs. FACS is recognized as the most comprehensive and objective means for mea-suring facial movement currently available, and it has become the standard for facialmeasurement in behavioral research [5].Human-observer-based methods like FACS are time consuming to learn and use,and they are difficult to standardize, especially across laboratories and over time.

5 Agoal of automated FACS coding [2,3,4] is to eliminate the need for manual coding andrealize automatic recognition and analysis of facial actions. The success of this effortdepends on access to reliably coded corpora of FACS-coded images from well-chosenobservational scenarios, which entails extensive need for manual , FACS coders typically proceedin either single or multiple passes throughthe video. When a single-pass procedure is used, they view the video and code the oc-currences of all target AU in each frame. FACS coders view video atboth regular videorate and in slow motion to detect often subtlechanges in facial features, such as wrin-kling of facial skin, that indicate the occurrence, timing, and intensity of facial intensity is coded on a 5-point ordinal intensity scale from trace to maximal inten-sity. FACS scoring produces a list of AUs, their intensity, and the video frames or timesat which each began ( onset), peaked (highest intensity observed), and ended ( ,offset).

6 Fig. 1 shows an example of onset, peak and offset of AU12, which raises the lipcorners obliquely. Until now, manual FACS coding was slow and achieving reliabilitywas : A Computer-Assisted System for FACS Automatic FACS Segmentation and Recognition from VideoAdvances in computer vision over the pastdecades have yielded advances toward thegoal of automatic FACS. That is, to eliminate the need for manual coding and realizeautomatic recognition and analysis of facial main streams on automatic analysis offacial expression consider emotion-specified expressions ( , happy or sad) andanatomically based facial actions ( ,FACS). Most relevant to Fast-FACS is work that addresses the temporal segmentationof AUs into onset, offset, and peak. Pantic and Pantras [4] used a rule-based method toseparate onset, apex and offset. Valstar and Pantic [6] combined Hidden Markov Modelsand Support Vector Machines to model the temporal dynamics of facial actions.

7 Theyconsidered the onset, apex, and offset frames as different classes. Accuracy was mea-sured as precision-recall in these classes. These approaches all used supervised learningwith the goal of fully automated expression or AU recently, two groups have proposed hybrid systems that make use of moreunsupervised learning techniques to augment manual coding of AUs. Zhang et al. [7]proposed an active learning approach to improve speed and accuracy in AU their approach, a sequence is labeled with an automatic System , and a user then isasked to label the frames that are considered ambiguous by the System . De la Torreet al. [8] proposed an unsupervised algorithm to segment facial behavior into AUs, anapproach that achieved concurrent validity with manual FACS coding. Subsequently,we found that this unsupervised approach could achieve fast, accurate, robust coding ofAU onsets and offsets when coupled with manual coding of AU Fast-FACSThis section describes Fast-FACS, that uses advances in computer vision and machinelearning to increase the efficiency and reliability of FACS Active Appearance TrackingThere exist a variety of methods for facial feature tracking.

8 Over the last decade, ap-pearance models have become increasingly prominent in computer vision and Appearance Models (PAMs) have been proven useful for alignment, de-tection, tracking, and face synthesis [9,10,11]. In particular, Active Appearance Models(AAMs) have proven an excellent tool for detecting and aligning facial features. AAMs[9,11,10] typically fit their shape and appearance componentsto an image through a gra-dient descent, although other optimization approaches have been employed with similarresults. Figure 1 shows how a person-dependent AAM [11,9] is able to track the facialfeatures in a video segment that includes smiling (AU12). A person-dependent AAMis built by manually annotating about3%of the video to use for training. The AAM iscomposed of66landmarks that deform to fit perturbations in facial features. To the bestof our knowledge, the work described here is the first to use the results of AAMs in ahybrid System to improve the speed and reliability of FACS coding.

9 The hybrid systemaugments the skill of highly trained FACS coders with computer vision and machinelearning based video editing and estimation of AU onsets and De la Torre et Peak, Onset, and Offset CodingIn the first step of Fast-FACS, the user annotates the peak of a facial action. The systemthen automatically estimates the remaining boundaries of the event, that is, the onsetand offset (extent) of the AU. The estimation of the position of the onset and offset ofa given event peak is based on a similarity measure defined on features derived fromthe AAM mesh of the tracked face and on the expected distribution of onset and offsetdurations (for a given AU) derived from a database of manually coded ) Similarity matrix for a video segment. The red rectangle denotes a specific AU 12instance as coded manually. The red circle marks the user-labeled peak. Observe that the AUdefines a region bounded by sharp edges in the similarity matrix.

10 Right) Similarity curve forthe marked peak ( ,jfor alljin a neighborhood). Note how the estimated onset andoffset snap to local minima on the similarity construct a symmetric affinity matrixK n n, where each entrykij [0,1]represents the similarity between framesiandj,andndenotes the number offrames [8]. This similarity measure will be used to decide where best to partition theAU into onset, peak and offset compute the similarity measure (a qualitative distance from the peak frame),kij,we use the66shape landmarks from the tracking. The particular distance measure willbe addressed in section The description of the feature extraction process follows:The AAM mesh is first interpolated to a finer resolution using B-Spline fitting in theregion of interest (upper or lower face). The resulting mesh from frameiis aligned withrespect to framejusing an affine transform intended to remove the rigid movement ofthe head while retaining the elastic deformations of facial actions.