Real-Time Eye Blink Detection using Facial Landmarks

1 21stComputer Vision Winter WorkshopLuka Cehovin, Rok Mandeljc, Vitomir Struc (eds.)Rimske Toplice, Slovenia, February 3 5, 2016 Real-Time Eye Blink Detection using Facial LandmarksTereza Soukupov a and Jan CechCenter for Machine Perception, Department of CyberneticsFaculty of Electrical Engineering, Czech Technical University in Real-Time algorithm to detect eye blinksin a video sequence from a standard camera is pro-posed. Recent landmark detectors, trained on in-the-wild datasets exhibit excellent robustness against ahead orientation with respect to a camera, varyingillumination and Facial expressions. We show thatthe Landmarks are detected precisely enough to reli-ably estimate the level of the eye opening.

2 The pro-posed algorithm therefore estimates the landmarkpositions, extracts a single scalar quantity eye as-pect ratio (EAR) characterizing the eye opening ineach frame. Finally, an SVM classifier detects eyeblinks as a pattern of EAR values in a short tempo-ral window. The simple algorithm outperforms thestate-of-the-art results on two standard IntroductionDetecting eye blinks is important for instance insystems that monitor a human operator vigilance, driver drowsiness [5, 13], in systems that warna computer user staring at the screen without Blink -ing for a long time to prevent the dry eye and thecomputer vision syndromes [17, 7, 8], in human-computer interfaces that ease communication for dis-abled people [15], or for anti-spoofing protection inface recognition systems [11].

3 Existing methods are either active or passive. Ac-tive methods are reliable but use special hardware,often expensive and intrusive, infrared camerasand illuminators [2], wearable devices, glasses witha special close-up cameras observing the eyes [10].While the passive systems rely on a standard remotecamera methods have been proposed to automati-cally detect eye blinks in a video sequence. Severalmethods are based on amotion estimationin the eyeregion. Typically, the face and eyes are detected byFigure 1: Open and closed eyes with landmarkspiautomatically detected by [1]. The eye aspect ratioEAR in Eq. (1) plotted for several frames of a videosequence. A single Blink is Viola-Jones type detector. Next, motion in the eyearea is estimated from optical flow, by sparse track-ing [7, 8], or by frame-to-frame intensity differenc-ing and adaptive thresholding.

4 Finally, a decision ismade whether the eyes are or are not covered by eye-lids [9, 15]. A different approach is to infer thestateof the eye opening from a single image, as bycorrelation matching with open and closed eye tem-plates [4], a heuristic horizontal or vertical image in-tensity projection over the eye region [5, 6], a para-metric model fitting to find the eyelids [18], or activeshape models [14].A major drawback of the previous approaches isthat they usually implicitly impose too strong re-quirements on the setup, in the sense of a relativeface-camera pose (head orientation), image resolu-tion, illumination, motion dynamics, etc. Especiallythe heuristic methods that use raw image intensityare likely to be very sensitive despite their nowadays, robust real-timefacial land-mark detectorsthat capture most of the character-istic points on a human face image, including eyecorners and eyelids, are available, see Fig.

5 1. Mostof the state-of-the-art landmark detectors formulatea regression problem, where a mapping from an im-age into landmark positions [16] or into other land-mark parametrization [1] is learned. These mod-ern landmark detectors are trained on in-the-wilddatasets and they are thus robust to varying illu-mination, various Facial expressions, and moderatenon-frontal head rotations. An average error of thelandmark localization of a state-of-the-art detectoris usually below five percent of the inter-ocular dis-tance. Recent methods run even significantly superreal-time [12].Therefore, we propose a simple but efficient al-gorithm to detect eye blinks by using a recent faciallandmark detector.

6 A single scalar quantity that re-flects a level of the eye opening is derived from thelandmarks. Finally, having a per-frame sequence ofthe eye opening estimates, the eye blinks are foundby an SVM classifier that is trained on examples ofblinking and non-blinking segmentation model presented in [14] issimilar to the proposed method. However, their sys-tem is based on active shape models with reportedprocessing time of about 5 seconds per frame for thesegmentation, and the eye opening signal is normal-ized by statistics estimated by observing a longer se-quence. The system is thus usable for offline pro-cessing only. The proposed algorithm runs Real-Time ,since the extra costs of the eye opening from land-marks and the linear SVM are contributions of the paper are:1.

7 Ability of two state-of-the-art landmark de-tectors [1, 16] to reliably distinguish betweenthe open and closed eye states is quantita-tively demonstrated on a challenging in-the-wild dataset and for various face image A novel Real-Time eye Blink Detection algorithmwhich integrates a landmark detector and a clas-sifier is proposed. The evaluation is done on twostandard datasets [11, 8] achieving state-of-the-art rest of the paper is structured as follows: Thealgorithm is detailed in Sec. 2, experimental valida-Eye aspect thresholding (t = ):EAR SVM output:Ground-truth:blinkblinkblinknon-b linknon-blinknon-blinkhalfFigure 2: Example of detected blinks. The plots ofthe eye aspect ratio EAR in Eq.

8 (1), results of theEAR thresholding (threshold set to ), the blinksdetected by EAR SVM and the ground-truth labelsover the video sequence. Input image with detectedlandmarks (depicted frame is marked by a red line).tion and evaluation is presented in Sec. 3. Finally,Sec. 4 concludes the Proposed methodThe eye Blink is a fast closing and reopening ofa human eye. Each individual has a little bit differentpattern of blinks. The pattern differs in the speed ofclosing and opening, a degree of squeezing the eyeand in a Blink duration. The eye Blink lasts approxi-mately 100-400 propose to exploit state-of-the-art Facial land-mark detectors to localize the eyes and eyelid con-tours. From the Landmarks detected in the image,we derive the eye aspect ratio (EAR) that is used asan estimate of the eye opening state.

9 Since the per-frame EAR may not necessarily recognize the eyeblinks correctly, a classifier that takes a larger tem-poral window of a frame into account is Description of featuresFor every video frame, the eye Landmarks are de-tected. The eye aspect ratio (EAR) between heightand width of the eye is p2 p6 + p3 p5 2 p1 p4 ,(1)wherep1,..,p6are the 2D landmark locations, de-picted in Fig. EAR is mostly constant when an eye is openand is getting close to zero while closing an eye. Itis partially person and head pose insensitive. Aspectratio of the open eye has a small variance among indi-viduals and it is fully invariant to a uniform scaling ofthe image and in-plane rotation of the face. Since eyeblinking is performed by both eyes synchronously,the EAR of both eyes is averaged.

10 An example ofan EAR signal over the video sequence is shown inFig. 1, 2, similar feature to measure the eye opening wassuggested in [9], but it was derived from the eye seg-mentation in a binary ClassificationIt generally does not hold that low value of theEAR means that a person is blinking. A low valueof the EAR may occur when a subject closes his/hereyes intentionally for a longer time or performs a fa-cial expression, yawning, etc., or the EAR captures ashort random fluctuation of the , we propose a classifier that takes alarger temporal window of a frame as an input. Forthe 30fps videos, we experimentally found that 6frames can have a significant impact on a Blink detec-tion for a frame where an eye is the most closed whenblinking.