RetinaFace: Single-Shot Multi-Level Face Localisation in ...

1 RetinaFace: Single-Shot Multi-Level Face Localisation in the WildJiankang Deng* 1,2,3 Jia Guo* 2 Evangelos Ververas1,3 Irene Kotsia4 Stefanos Zafeiriou1,31 Imperial College2 InsightFace3 FaceSoft4 Middlesex University London{ , , tremendous strides have been made in uncon-trolled face detection, accurate and efficient 2D face align-ment and 3D face reconstruction in-the-wild remain anopen challenge. In this paper, we present a novel Single-Shot , Multi-Level face Localisation method, named Reti-naFace, which unifies face box prediction, 2D facial land-mark Localisation and 3D vertices regression under onecommon target: point regression on the image plane. Tofill the data gap, we manually annotated five facial land-marks on the WIDER FACE dataset and employed a semi-automatic annotation pipeline to generate 3D vertices forface images from the WIDER FACE, AFLW and FDDB datasets. Based on extra annotations, we propose a mu-tually beneficial regression target for 3D face reconstruc-tion, that is predicting 3D vertices projected on the imageplane constrained by a common 3D topology.}

2 The proposed3D face reconstruction branch can be easily incorporated,without any optimisation difficulty, in parallel with the exist-ing box and 2D landmark regression branches during jointtraining. Extensive experimental results show that Reti-naFace can simultaneously achieve stable face detection,accurate 2D face alignment and robust 3D face reconstruc-tion while being efficient through Single-Shot IntroductionAutomatic face Localisation is a prerequisite for facialimage analysis in many applications such as facial attributeanalysis( expression [64] and age [41,39]) and facialidentity recognition [18,12,56]. A narrow definition of facelocalisation may refer to traditional face detection [54,62],which aims at estimating the face bounding boxes withoutpossessing any scale and position prior. Nevertheless, inthis paper we refer to a broader definition of face locali-*Equal is a nonprofit Github project for 2D and 3D face 1.

3 Face Localisation tasks from coarse to fine. Face detec -tion only predicts one center point and scales. Face pose estima-tion calculates the scale, 3D rotation and translation face alignment localises more semantic points. Face seg-mentation computes pixel-wise label maps for different semanticcomponents ( mouth, eyes). 3D face reconstruction can estab-lish dense 3D correspondence for every pixel of a face, which isthe most informative and demanding face Localisation which includes face detection [43], face pose esti-mation [48,60,31,5], face alignment [14,57,17,16,15,58,23], face segmentation [50,34] and 3D face reconstruc-tion [72,1,19,70]. In , we show face localisationtasks with multiple levels of detail, from coarse to , face pose estimation, face alignment, face seg-mentation and 3D face reconstruction are steps subsequentto face detection. These fine-grind face Localisation tasksare performed on individual face crops and the compu-tational complexity increases linearly with the number offaces in the input image.

4 Since all face Localisation tasksfrom face detection to 3D face reconstruction aim at estab-lishing the semantic correspondence between different faceimages, with their main difference being only in the infor-mation level, the question that arises is if can we combinethem into a united framework by jointly training and make15203different tasks benefit from each training process for face detection usually containsclassification and box regression losses [21]. Chenet al. [8]proposed to combine face detection and alignment in a jointcascade framework based on the observation that alignedface shapes can provide better features for face classifica-tion. Inspired by [8], MTCNN [66] and STN [7] simul-taneously detected faces and five facial landmarks. Due totraining data limitation, JDA [8], MTCNN [66] and STN [7]have not verified whether detection of tiny faces can benefitfrom the extra supervision of five facial Mask R-CNN [25], the detection performance is sig-nificantly improved by adding a branch for predicting anobject mask in parallel with the existing branch for bound-ing box classification and regression.

5 This confirms thatdense pixel-wise annotations are also beneficial for improv-ing detection. In FAN [55], an anchor-level attention mapis proposed to improve the occluded face detection. Nev-ertheless, the proposed attention map is quite coarse anddoes not contain semantic information. In MFN [6], a sin-gle end-to-end network is presented to jointly predict thebounding box locations and 3 DMM parameters for multiplefaces. This contributes to more precise face detection in-the-wild by leveraging both 2D information from boundingboxes and 3D information from 3 DMM parameters. How-ever, 3 DMM parameter prediction constitutes an indirectregression target when compared to semantic point predic-tion ( box center) on the image plane. In this paper, weexplore joint learning for different face Localisation tasks(face detection, 2D face alignment and 3D face reconstruc-tion) based on the Single-Shot [40,69,51] framework. Toovercome the limitation of training data [6], we have man-ually annotated five facial landmarks fromthe WIDER FACE training dataset [59].

6 In addition, we setup a semi-automatic annotation pipeline to generate 1k 3 Dvertices for22kfaces from the WIDER FACE dataset [59], from the AFLW dataset [30], the FDDB full image set [28]. Based on these train-ing data, we propose an innovative, straightforward and ef-fective 3D mesh regression method. More specifically, wedirectly regressx,yandzcoordinates in the image spaceand add a regularization term to control the edge distanceof triangles in the mesh for more accurate prediction learning of face bounding box locations, five fa-cial landmarks and 1k 3D vertices forces the network tolearn exclusive facial features that characterize face pose,shape, and expression, in addition to differentiating face re-gions from the background. As five facial landmarks lo-calisation and 3D vertices regression both target on predict-ing semantic points on the image plane, face box predictionbenefits from joint learning and becomes more accurate andstable, producing less false positives.

7 Also, as the anno-tated but challenging face detection data [59] are employedin the joint training of face detection and the rest fine-grindface Localisation tasks (for which usually less challengingdatasets are employed, [30]), they directly contribute torobust 3D mesh summarise, our key contributions are: We integrate face bounding box prediction, 2D faciallandmark Localisation and 3D vertices regression un-der a unified Multi-Level face Localisation task with acommon goal: point regression on the image plane. Based on a Single-Shot inference, we propose a mutu-ally beneficial learning strategy to train a unified Multi-Level face Localisation method that simultaneously pre-dicts face bounding boxes, five 2D facial landmarks,and 1k 3D vertices. Our method achieves state-of-the-art performance inface detection and 2D face alignment, as well as robust3D face reconstruction with Single-Shot Related WorkFace by generic object detection meth-ods [21,46,38,44,45,35,36], face detection has recentlyachieved remarkable progress [27,40,69,10,51].

8 Differ-ent from generic object detection, face detection featuressmaller ratio variations (from 1:1 to 1 ) but much largerscale variations (from several pixels to thousands of pix-els). The most recent state-of-the-art methods [40,69,51]focus on Single-Shot design [38,36] which densely samplesface locations and scales on feature pyramids [35], demon-strating promising performance and yielding faster infer-ence compared to two-stage methods [46,63,10]. Follow-ing this route, we improve the performance of single-shotface detection by exploiting extra-supervisions from Multi-Level face Localisation Face dense pixel-to-pixelcorrespondence is one of the most fundamental problemsin 3D face reconstruction from 2D images. Recently, a lotof works follow the approach of regressing 3 DMM param-eters from 2D images using CNNs [29,72,53,47,24,52].Jourablooet al. [29] employs a cascade of CNNs to alter-nately regress the shape and pose parameters.

9 3 DDFA [72]utilizes cascade iterations on a single CNN to jointly regressthe shape and pose parameters. However, as pose and3 DMM parameters are indirect information for a 2D faceimage, the variations from network prediction can exerthuge visual error. Most recently, model parameter regres-sion methods have changed into dense correspondence re-gression approaches [1,19]. By using the intermediate UVrepresentation, DenseReg [1] predicts the UV coordinatesand PRN [19] forecasts 3D coordinates rearranged in theUV space. However, UV transformation is still an indirectrepresentation for a 2D image. In this paper, we resort themost straightforward 3D representation: 3D vertices pro-5204jected on the image plane. This representation is consistentwith the regression targets of face detection and 2D faciallandmark Localisation , and easy to optimise in a Single-Shot , Multi-Level face Localisation framework. Due to the paralleltraining with face detection and 2D face alignment, our 3 Dface reconstruction branch is very robust under Proposed 3D Face ReconstructionIn , we show a fixed number ofNvertices(V= [x1, y1, z1;x2, y2, z2; ;xN, yN, zN]) on a pre-defined topological triangle context.

10 These correspondingvertices share the same semantic meaning across differentfaces. With the fixed triangle topology, every pixel on theface can be indexed by the barycentric coordinates and thetriangle index, thus there exists a pixel-wise correspondencewith the 3D face. Comparing Mesh68 and Mesh1k in ,it becomes obvious that more vertices make the mesh moreinformative and smooth. As the parameters of the last layerincrease linearly with the size of the regression output, wechoose to regress 1k + 68 vertices, which is a subset of53,215vertices [42], carefully sampled to sufficiently re-tain the 3D faces this paper, we directly regress 3D vertices on the 2 Dimage plane. As each densely aligned3D face is repre-sented by concatenating itsNvertex coordinates, we em-ploy the following vertex loss to constrain the location ofvertices:Lvert=1NN i=1||Vi(x, y, z) V i(x, y, z)||1,(1)whereN= 1103is the number of vertices,Vis the predic-tion of our model andV is the of visible vertices in the im-age space can be directly learned from input face , predicting thezcoordinates and thexandyco-ordinates of invisible vertices is challenging due to the in-formation loss occurring when projecting a face from 3D to2D.