RetinaFace: Single-Shot Multi-Level Face Localisation in ...

1 RetinaFace: Single-Shot Multi-Level Face Localisation in the WildJiankang Deng* 1,2,3 Jia Guo* 2 Evangelos Ververas1,3 Irene Kotsia4 Stefanos Zafeiriou1,31 Imperial College2 InsightFace3 FaceSoft4 Middlesex University London{ , , tremendous strides have been made in uncon-trolled face detection, accurate and efficient 2D face align-ment and 3D face reconstruction in-the-wild remain anopen challenge. In this paper, we present a novel Single-Shot , Multi-Level face Localisation method, named Reti-naFace, which unifies face box prediction, 2D facial land-mark Localisation and 3D vertices regression under onecommon target: point regression on the image plane.}

2 Tofill the data gap, we manually annotated five facial land-marks on the WIDER FACE dataset and employed a semi-automatic annotation pipeline to generate 3D vertices forface images from the WIDER FACE, AFLW and FDDB datasets. Based on extra annotations, we propose a mu-tually beneficial regression target for 3D face reconstruc-tion, that is predicting 3D vertices projected on the imageplane constrained by a common 3D topology. The proposed3D face reconstruction branch can be easily incorporated,without any optimisation difficulty, in parallel with the exist-ing box and 2D landmark regression branches during jointtraining.

3 Extensive experimental results show that Reti-naFace can simultaneously achieve stable face detection,accurate 2D face alignment and robust 3D face reconstruc-tion while being efficient through Single-Shot IntroductionAutomatic face Localisation is a prerequisite for facialimage analysis in many applications such as facial attributeanalysis( expression [64] and age [41,39]) and facialidentity recognition [18,12,56]. A narrow definition of facelocalisation may refer to traditional face detection [54,62],which aims at estimating the face bounding boxes withoutpossessing any scale and position prior.

4 Nevertheless, inthis paper we refer to a broader definition of face locali-*Equal is a nonprofit Github project for 2D and 3D face 1. Face Localisation tasks from coarse to fine. Face detec-tion only predicts one center point and scales. Face pose estima-tion calculates the scale, 3D rotation and translation face alignment localises more semantic points. Face seg-mentation computes pixel-wise label maps for different semanticcomponents ( mouth, eyes). 3D face reconstruction can estab-lish dense 3D correspondence for every pixel of a face, which isthe most informative and demanding face Localisation which includes face detection [43], face pose esti-mation [48,60,31,5], face alignment [14,57,17,16,15,58,23], face segmentation [50,34] and 3D face reconstruc-tion [72,1,19,70].

5 In , we show face localisationtasks with multiple levels of detail, from coarse to , face pose estimation, face alignment, face seg-mentation and 3D face reconstruction are steps subsequentto face detection. These fine-grind face Localisation tasksare performed on individual face crops and the compu-tational complexity increases linearly with the number offaces in the input image. Since all face Localisation tasksfrom face detection to 3D face reconstruction aim at estab-lishing the semantic correspondence between different faceimages, with their main difference being only in the infor-mation level, the question that arises is if can we combinethem into a united framework by jointly training and make15203different tasks benefit from each training process for face detection usually containsclassification and box regression losses [21].

6 Chenet al. [8]proposed to combine face detection and alignment in a jointcascade framework based on the observation that alignedface shapes can provide better features for face classifica-tion. Inspired by [8], MTCNN [66] and STN [7] simul-taneously detected faces and five facial landmarks. Due totraining data limitation, JDA [8], MTCNN [66] and STN [7]have not verified whether detection of tiny faces can benefitfrom the extra supervision of five facial Mask R-CNN [25], the detection performance is sig-nificantly improved by adding a branch for predicting anobject mask in parallel with the existing branch for bound-ing box classification and regression.

7 This confirms thatdense pixel-wise annotations are also beneficial for improv-ing detection. In FAN [55], an anchor-level attention mapis proposed to improve the occluded face detection. Nev-ertheless, the proposed attention map is quite coarse anddoes not contain semantic information. In MFN [6], a sin-gle end-to-end network is presented to jointly predict thebounding box locations and 3 DMM parameters for multiplefaces. This contributes to more precise face detection in-the-wild by leveraging both 2D information from boundingboxes and 3D information from 3 DMM parameters. How-ever, 3 DMM parameter prediction constitutes an indirectregression target when compared to semantic point predic-tion ( box center) on the image plane.

8 In this paper, weexplore joint learning for different face Localisation tasks(face detection, 2D face alignment and 3D face reconstruc-tion) based on the Single-Shot [40,69,51] framework. Toovercome the limitation of training data [6], we have man-ually annotated five facial landmarks fromthe WIDER FACE training dataset [59]. In addition, we setup a semi-automatic annotation pipeline to generate 1k 3 Dvertices for22kfaces from the WIDER FACE dataset [59], from the AFLW dataset [30], the FDDB full image set [28]. Based on these train-ing data, we propose an innovative, straightforward and ef-fective 3D mesh regression method.

9 More specifically, wedirectly regressx,yandzcoordinates in the image spaceand add a regularization term to control the edge distanceof triangles in the mesh for more accurate prediction learning of face bounding box locations, five fa-cial landmarks and 1k 3D vertices forces the network tolearn exclusive facial features that characterize face pose,shape, and expression, in addition to differentiating face re-gions from the background. As five facial landmarks lo-calisation and 3D vertices regression both target on predict-ing semantic points on the image plane, face box predictionbenefits from joint learning and becomes more accurate andstable, producing less false positives.

10 Also, as the anno-tated but challenging face detection data [59] are employedin the joint training of face detection and the rest fine-grindface Localisation tasks (for which usually less challengingdatasets are employed, [30]), they directly contribute torobust 3D mesh summarise, our key contributions are: We integrate face bounding box prediction, 2D faciallandmark Localisation and 3D vertices regression un-der a unified Multi-Level face Localisation task with acommon goal: point regression on the image plane. Based on a Single-Shot inference, we propose a mutu-ally beneficial learning strategy to train a unified Multi-Level face Localisation method that simultaneously pre-dicts face bounding boxes, five 2D facial landmarks,and 1k 3D vertices.