Observer-Based Measurement of Facial Expression …

1 13 Observer-Based Measurement of Facial ExpressionWith the Facial action coding SystemJeffrey F. CohnZara AmbadarPaul Ekman203 Facial Expression has been a focus of emotion research for overa hundred years (Darwin, 1872/1998). It is central to severalleading theories of emotion (Ekman, 1992; Izard, 1977;Tomkins, 1962) and has been the focus of at times heateddebate about issues in emotion science (Ekman, 1973, 1993;Fridlund, 1992; Russell, 1994). Facial Expression figuresprominently in research on almost every aspect of emotion,including psychophysiology (Levenson, Ekman, & Friesen,1990), neural bases (Calder et al., 1996; Davidson, Ekman,Saron, Senulis, & Friesen, 1990), development (Malatesta,Culver, Tesman, & Shephard, 1989; Matias & Cohn, 1993),perception (Ambadar, Schooler, & Cohn, 2005), social pro-cesses (Hatfield, Cacioppo, & Rapson, 1992; Hess & Kirouac,2000), and emotion disorder (Kaiser, 2002; Sloan, Straussa,Quirka, & Sajatovic, 1997), to name a of its importance to the study of emotion, a num-ber of Observer-Based systems of Facial Expression measure-ment have been developed (Ekman & Friesen, 1978, 1982;Ekman, Friesen, & Tomkins, 1971; Izard, 1979, 1983; Izard& Dougherty, 1981; Kring & Sloan, 1991; Tronick, Als, &Brazelton, 1980).

2 Of these various systems for describingfacial Expression , the Facial action coding system (FACS;Ekman & Friesen, 1978; Ekman, Friesen, & Hager, 2002) isthe most comprehensive, psychometrically rigorous, andwidely used (Cohn & Ekman, 2005; Ekman & Rosenberg,2005). Using FACS and viewing video-recorded Facial behav-ior at frame rate and slow motion, coders can manually codenearly all possible Facial expressions, which are decomposedinto action units (AUs). action units, with some qualifica-tions, are the smallest visually discriminable Facial move-ments. By comparison, other systems are less thorough(Malatesta et al., 1989), fail to differentiate between someanatomically distinct movements (Oster, Hegley, & Nagel,1992), consider movements that are not anatomically dis-tinct as separable (Oster et al., 1992), and often assume a one-to-one mapping between Facial Expression and emotion (fora review of these systems, see Cohn & Ekman, in press).

3 Unlike systems that use emotion labels to describe ex-pression, FACS explicitly distinguishes between Facial actionsand inferences about what they mean. FACS itself is descrip-tive and includes no emotion-specified descriptors. Hypoth-eses and inferences about the emotional meaning of facialactions are extrinsic to FACS. If one wishes to make emo-tion-based inferences from FACS codes, a variety of relatedresources exist. These include the FACS Investigators Guide(Ekman et al., 2002), the FACS interpretive database (Ekman,Rosenberg, & Hager, 1998), and a large body of empiricalresearch.(Ekman & Rosenberg, 2005). These resources sug-gest combination rules for defining emotion-specified expres-sions from FACS action units, but this inferential step remainsextrinsic to FACS. Because of its descriptive power, FACSis regarded by many as the standard measure for Facial be-havior and is used widely in diverse fields. Beyond emo-tion science, these include Facial neuromuscular disorders(Van Swearingen & Cohn, 2005), neuroscience (Bruce &Young, 1998; Rinn, 1984, 1991), computer vision (Bartlett,UNPROOFED PAGES204 Emotion AssessmentEkman, Hager, & Sejnowski, 1999; Cohn, Zlochower, Lien,& Kanade, 1999; Pantic & Rothkrantz, 2000; Tian, Cohn,& Kanade, 2005), computer graphics and animation (Breidt,Wallraven, Cunningham, & Buelthoff, 2003; Parke & Wa-ters, 1996), and face encoding for digital signal processing(International Organization for Standardization, 2002; Tao,Chen, Wu, & Huang, 1999).

4 In this chapter, we discuss the conceptual basis for FACS,the numerical listing of discrete Facial movements identifiedby the system , the evaluative psychometrics of the system ,and the recommended training requirements. We also in-clude information on how to obtain software for computer-assisted FACS has progressed through three versions: the initialversion (FACS 1978), a document-based update (FACS 1992),and a new edition (FACS 2002), which includes improve-ments in scoring criteria and in didactic materials, extensiveuse of hyperlinked cross-referenced text, and embeddedvideo links in the CD version. Throughout this chapter, weuse publication date when referring to a specific version BasisSign Versus Message JudgmentEkman and Friesen (Ekman, 1964, 1965; Ekman & Friesen,1969) distinguished two conceptual approaches to studyingfacial behavior, namely, measuring judgments about one oranother message and measuring the sign vehicles that con-vey the message.

5 In message judgment, the observer s task isto make inferences about something underlying the facialbehavior emotion, mood, traits, attitudes, personality, andthe like; for this reason observers typically are referred to as judges or raters. In measuring sign vehicles, observersdescribe the surface of behavior; they count how many timesthe face moves a certain way, or how long a movement lasts,or whether it was a movement of the frontalis or corrugatormuscle. As an example, on seeing a smiling face, an observerwith a judgment-based approach would make judgmentssuch as happy, whereas an observer with a sign-based ap-proach would code the face as having an upward, obliquemovement of the lip corners. Observers with a sign-basedapproach are supposed to function like machines and typi-cally are referred to as coders. Though message- and sign-based approaches can some-times answer the same questions, they can also answer dif-ferent questions, for they focus on different judgment research is not typically focused on theface.

6 The face is but an input. The focus is on the personobserving the face and/or on the message obtained. Ques-tions have to do with whether a difference is detectable oraccurate, whether there are individual differences amongraters, reflecting skill, gender, or personality, and whethermessages obtained are best represented as dimensions sign vehicles are measured when the focus is onunearthing something fairly specific about Facial behavioritself, not about the perception of the face. It is the onlymethod that can be used to answer such questions as:1. To what extent is the Facial activity shown by new-borns and infants systematic, not random, and whichparticular actions first show such systematic organiza-tion? To answer this question, Facial behavior shownduring samples taken at different developmentalpoints or in different situational contexts can bemeasured. Then the probabilities of particular co-occurrences and sequential patterns of Facial actionscan be evaluated (Cohn & Tronick, 1983; Oster &Ekman, 1978).

7 2. Which particular Facial actions are employed to signalemphasis in conversation? Facial actions that co-occurwith verbal or vocal emphasis must be measured todetermine whether there are any actions that consis-tently accompany any emphasis (Ekman, 1980).3. Is there a difference in the smile during enjoyment ascompared with a discomfort smile? The particularfacial actions evident in smiling movements must bemeasured when persons are known, by means otherthan the face, to be experiencing positive and negativeaffect (Ekman, Friesen, & Ancoli, 1980; Frank,Ekman, & Friesen, 1993).4. Are there differences in heart rate that accompany nosewrinkling and upper lip raising versus opening theeyes and raising the brows? Facial behavior must bemeasured to identify the moments when theseparticular Facial configurations occur in order toexamine coincident heart rate activity (Levenson et al.,1990).The preceding examples are not intended to convey thefull range of issues that can be addressed only by measuringfacial sign vehicles.

8 They should, however, serve to illustratethe variety of questions that require this approach. One mightexpect the Measurement of sign vehicles approach to havebeen followed often, as it is required for study of many dif-ferent problems. However, there have been only a few suchstudies compared with the many that have measured themessages judged when viewing the face. It is much easier toperform the latter sort of study. The investigator need nottamper with the face itself, other than by picking somesamples to show. Data are obtained quickly: One can mea-sure observers judgments much more quickly than one candescribe reliably the flow and variety of Facial the advent of FACS, an important obstacle to re-search measuring sign vehicles has been the lack of any ac-cepted, standard, ready-for-use technique for measuringfacial movement. Investigators who have measured facialmovement have invented their techniques in large part deUNPROOFED PAGESO bserver-Based Measurement of Facial Expression With the Facial action coding System205novo, rarely making use of the work of their have seemed to be uninformed by the previous litera-ture.

9 Even the more scholarly have found it difficult to buildon the methods previously reported, because descriptionsof Facial activity are often less clear than they appear on firstreading. A Facial action may seem to be described in suffi-cient detail and exactness until an attempt is made to applythat description to the flow of Facial behavior. For instance,descriptions of brow motion that omit specific appearancechanges in Facial lines and furrows and in the appearance ofthe upper eyelid omit information that may be needed todiscriminate among related but different Facial actions. FACS addresses the need for a comprehensive system that can bereadily learned, that is psychometrically sound, and that hashigh utility for various research for Deriving action UnitsThe anatomical basis of Facial action (Figure ) providesthe basis for deriving units of behavior. With few exceptions,all people have the same Facial muscles (Schmidt & Cohn,2001).

10 FACS action units are based on what the muscles al-low the face to do. To determine the appearance changesassociated with each muscle, Ekman and Friesen began byelectrically stimulating individual muscles and by learningto control them voluntarily. The result is that each action unitis associated with one or more Facial selecting Facial actions, Ekman and Friesen (1978)used the criterion that observers were capable of reliablydistinguishing all appearance changes resulting from thevarious muscles. If two appearance changes could not bereliably distinguished, they were combined, even if differentmuscles were involved. Conversely, some actions proved toosubtle for reliable Measurement . Visemes, for instance, arevisually distinguishable phonemes (Massaro, Cohen, Beskow,Daniel, & Cole, 1998); with some exceptions, they are notincluded as AUs in action UnitsFACS 2002 specifies 9 action units in the upper face and 18in the lower face.