Pose-Robust Face Recognition via Deep Residual …

1 Pose-Robust Face Recognition via Deep Residual Equivariant MappingKaidi Cao2 Yu Rong1,2 Cheng Li2 Xiaoou Tang1 Chen Change Loy11 Department of Information Engineering, The Chinese university of Hong Kong2 SenseTime Research{ry017, ccloy, Recognition achieves exceptional success thanks tothe emergence of deep learning. However, many contempo-rary face Recognition models still perform relatively poor inprocessing profile faces compared to frontal faces. A keyreason is that the number of frontal and profile trainingfaces are highly imbalanced - there are extensively morefrontal training samples compared to profile ones. In ad-dition, it is intrinsically hard to learn a deep represen-tation that is geometrically invariant to large pose varia-tions.}

2 In this study, we hypothesize that there is an inher-ent mapping between frontal and profile faces, and conse-quently, their discrepancy in the deep representation spacecan be bridged by an equivariant mapping. To exploit thismapping, we formulate a novel Deep Residual EquivAriantMapping (DREAM) block, which is capable of adaptivelyadding residuals to the input deep representation to trans-form a profile face representation to a canonical pose thatsimplifies Recognition . The DREAM block consistently en-hances the performance of profile face Recognition for manystrong deep networks, including ResNet models, without de-liberately augmenting training data of profile faces.

3 Theblock is easy to use, light-weight, and can be implementedwith a negligible computational IntroductionThe emergence of deep learning greatly advances thefrontier of face Recognition [29, 30]. The main focus tendsto center around near-frontal faces while there can be noassurance of view consistency when face Recognition isconducted in unconstrained environments. Although hu-man performance only drops slightly from frontal-frontal indicates shared first authorship1 Codes and models are available PositivesFalseNegativesFigure 1. A state-of-the-art face Recognition model [34] tested ona challenging frontal-profile faces dataset [26].

4 It is observed thatprofile faces of different persons are easily to be mismatched (falsepositives), and profile and frontal faces of the same identity maynot trigger a match leading to false frontal-profile face verification, many existing algorithmscan suffer a drop of over 10% [26]. Thus, large pose vari-ation remains to be a significant challenge that confrontsreal-world face provide an example in Figure 1 to show the fail-ure modes of a state-of-the-art face verification model. Wetrained the same ResNet-18 model as reported in [34]. Thismodel achieves a high accuracy of on the LFWbenchmark [12]. Despite the strong model, it tends tofalsely match profile faces of different identities yieldinga number of false positives.

5 In addition, the model is alsolikely to miss frontal and profile faces of the same identityleading to false does face Recognition work poorly on profile faces?Modern deep learning is heavily data-driven [11, 7]. Thegeneralization power of deep models is usually proportionalto the training data size. Given an uneven distribution ofprofile and frontal faces in the dataset, deeply learned fea-tures tend to bias on distinguishing frontal faces rather thanprofile faces. When it is infeasible to collect a massivedataset that covers all possible poses with even distribution, [ ] 2 Mar 2018 Actual imagesReconstructed imagesfrom featuresadds residualsDeep feature spaceFigure 2.

6 At the top of this figure, we illustrate the deep featureembedding of a subject in different poses. The proposed DREAM block is capable of adding Residual to the feature of a profile faceand map it to the frontal space. At the bottom of the figure,we show the actual reconstructed image of a profile face and itsmapped frontal have turned to alternative approaches to betterhandle the Recognition of profile faces. A large body ofmethods normalize images to a single frontal pose beforerecognition, either through elaborated dense 3D facial land-mark detection and warping [30], or another deep model (orgenerative adversarial network) specialized in face frontal-ization [31].

7 Such methods would add processing burdento the whole system. In addition, face frontalization in thewild, especially with extreme profile faces, is still consid-ered challenging. Often, synthesized frontal faces wouldcontain artifacts caused by occlusions and non-rigid expres-sions. Another potential solution is divide-and-conquer, ,training separate models for learning pose-specific identityfeatures [19]. This strategy tends to increase computationalcost due to the use of multiple this study, we hypothesize that the profile face domainpossesses a gradual connection with the frontal face domainin the deep feature space. Figure 2 illustrates a deep repre-sentation embedding of faces belong to the same subjectbut in different poses.

8 Given an input image of arbitrarypose, we can actually map its feature to the frontal spacethrough a mapping function that adds Residual . This obser-vation is closely connected to the notion offeature equivari-ance[15], which finds the representation of many deep lay-ers depends upon transformations of the input image. Inter-estingly, such transformations can be learned by a mappingfunction from data and the function can be subsequently ap-plied to manipulate the representation of an input image toachieve the desired by this observation, we formulate a novelmodule calledDeep Residual EquivAriant Mapping(DREAM) block, which can model the transformation be-tween frontal-profile faces in the high-level deep featurespace.

9 The block adaptively adds residuals to an input rep-resentation to transform a profile face to a canonical poseto simplify Recognition . The residuals are generated con-ditioned on the preceding feature representation via a fewadditional weight layers. To accommodate input faces ofarbitrary pose, a soft gate is introduced to adaptively con-trol the amount of residuals such that more residuals areadded to extreme profile faces while keeping the represen-tation unchanged if the input is already in a frontal work is conceptually related to the face frontaliza-tion [31] in that our approach also performs frontaliza-tion but not in the image space.

10 We observe from our ex-periments that transforming profile face features to frontalfeatures could yield better performance than image-levelfrontalization, which is susceptible to the negative influ-ence of artifacts as a result of image synthesis. To our bestknowledge, this study is the first attempt to performprofile-to-frontal face transformation in the deep feature DREAM block is appealing in several aspects:1. It is simple to implement. Specifically, the DREAM block is implemented as a simple yet effective gatedresidual can be integrated into exist-ing convolutional neural network (CNN) architecturesthrough stitching the block to the base network.