Feature Pyramid Networks for Object Detection

1 Feature Pyramid Networks for Object DetectionTsung-Yi Lin1,2, Piotr Doll ar1, Ross Girshick1,Kaiming He1, Bharath Hariharan1, and Serge Belongie21 Facebook AI Research (FAIR)2 Cornell University and Cornell TechAbstractFeature pyramids are a basic component in recognitionsystems for detecting objects at different scales. But recentdeep learning Object detectors have avoided Pyramid rep-resentations, in part because they are compute and memoryintensive. In this paper, we exploit the inherent multi-scale,pyramidal hierarchy of deep convolutional Networks to con-struct Feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed forbuilding high-level semantic Feature maps at all scales. Thisarchitecture, called a Feature Pyramid Network (FPN),shows significant improvement as a generic Feature extrac-tor in several applications.

2 Using FPN in a basic FasterR-CNN system, our method achieves state-of-the-art single-model results on the COCO Detection benchmark withoutbells and whistles, surpassing all existing single-model en-tries including those from the COCO 2016 challenge win-ners. In addition, our method can run at 5 FPS on a GPUand thus is a practical and accurate solution to multi-scaleobject Detection . Code will be made publicly IntroductionRecognizing objects at vastly different scales is a fun-damental challenge in computer pyramidsbuilt upon image pyramids(for short we call thesefeatur-ized image pyramids) form the basis of a standard solution[1] ( (a)). These pyramids are scale-invariant in thesense that an Object s scale change is offset by shifting itslevel in the Pyramid . Intuitively, this property enables amodel to detect objects across a large range of scales byscanning the model over both positions and Pyramid image pyramids were heavily used in theera of hand-engineered features [5,25].

3 They were socritical that Object detectors like DPM [7] required densescale sampling to achieve good results ( , 10 scales peroctave). For recognition tasks, engineered features have(a) Featurized image pyramidpredictpredictpredictpredict(b) Single Feature mappredict(d) Feature Pyramid Networkpredictpredictpredict(c) Pyramidal Feature hierarchypredictpredictpredictFigure 1. (a) Using an image Pyramid to build a Feature are computed on each of the image scales independently,which is slow. (b) Recent Detection systems have opted to useonly single scale features for faster Detection . (c) An alternative isto reuse the pyramidal Feature hierarchy computed by a ConvNetas if it were a featurized image Pyramid . (d) Our proposed FeaturePyramid Network (FPN) is fast like (b) and (c), but more this figure, Feature maps are indicate by blue outlines and thickeroutlines denote semantically stronger been replaced with features computed by deep con-volutional Networks (ConvNets) [19,20].

4 Aside from beingcapable of representing higher-level semantics, ConvNetsare also more robust to variance in scale and thus facilitaterecognition from features computed on a single input scale[15,11,29] ( (b)). But even with this robustness, pyra-mids are still needed to get the most accurate results. All re-cent top entries in the ImageNet [33] and COCO [21] detec-tion challenges use multi-scale testing on featurized imagepyramids ( , [16,35]). The principle advantage of fea-turizing each level of an image Pyramid is that it producesa multi-scale Feature representation in whichall levels aresemantically strong, including the high-resolution , featurizing each level of an image pyra-mid has obvious limitations. Inference time increases con-siderably ( , by four times [11]), making this approachimpractical for real applications.

5 Moreover, training deep12117networks end-to-end on an image Pyramid is infeasible interms of memory, and so, if exploited, image pyramids areused only at test time [15,11,16,35], which creates aninconsistency between train/test-time inference. For thesereasons, Fast and Faster R-CNN [11,29] opt to not use fea-turized image pyramids under default , image pyramids are not the only way to com-pute a multi-scale Feature representation. A deep ConvNetcomputes afeature hierarchylayer by layer, and with sub-sampling layers the Feature hierarchy has an inherent multi-scale, pyramidal shape. This in-network Feature hierarchyproduces Feature maps of different spatial resolutions, butintroduces large semantic gaps caused by different high-resolution maps have low-level features that harmtheir representational capacity for Object Single Shot Detector (SSD) [22] is one of the firstattempts at using a ConvNet s pyramidal Feature hierarchyas if it were a featurized image Pyramid ( (c)).

6 Ideally,the SSD-style Pyramid would reuse the multi-scale featuremaps from different layers computed in the forward passand thus come free of cost. But to avoid using low-levelfeatures SSD foregoes reusing already computed layers andinstead builds the Pyramid starting from high up in the net-work ( , conv43 of VGG nets [36]) and then by addingseveral new layers. Thus it misses the opportunity to reusethe higher-resolution maps of the Feature hierarchy. Weshow that these are important for detecting small goal of this paper is to naturally leverage the pyra-midal shape of a ConvNet s Feature hierarchy while cre-ating a Feature Pyramid that has strong semantics at allscales. To achieve this goal, we rely on an architecture thatcombines low-resolution, semantically strong features withhigh-resolution, semantically weak features via a top-downpathway and lateral connections ( (d)).

7 The result isa Feature Pyramid that has rich semantics at all levels andis built quickly from a single input image scale. In otherwords, we show how to create in-network Feature pyramidsthat can be used to replace featurized image pyramids with-out sacrificing representational power, speed, or architectures adopting top-down and skip con-nections are popular in recent research [28,17,8,26]. Theirgoals are to produce a single high-level Feature map of a fineresolution on which the predictions are to be made ( ). On the contrary, our method leverages the architectureas a Feature Pyramid where predictions ( , Object detec-tions) are independently made on each level ( ).Our model echoes a featurized image Pyramid , which hasnot been explored in these evaluate our method, called a Feature Pyramid Net-work (FPN), in various systems for Detection and segmen-tation [11,29,27].

8 Without bells and whistles, we re-port a state-of-the-art single-model result on the challengingCOCO Detection benchmark [21] simply based on FPN andpredictpredictpredictpredictFigure 2. Top: a top-down architecture with skip connections,where predictions are made on the finest level ( , [28]). Bottom:our model that has a similar structure but leverages it as afeaturepyramid, with predictions made independently at all basic Faster R-CNN detector [29], surpassing all exist-ing heavily-engineered single-model entries of competitionwinners. In ablation experiments, we find that for bound-ing box proposals, FPN significantly increases the AverageRecall (AR) by points; for Object Detection , it improvesthe COCO-style Average Precision (AP) by points andPASCAL-style AP by points, over a strong single-scalebaseline of Faster R-CNN on ResNets [16].

9 Our method isalso easily extended to mask proposals and improves bothinstance segmentation AR and speed over state-of-the-artmethods that heavily depend on image addition, our Pyramid structure can be trained end-to-end with all scales and is used consistently at train/test time,which would be memory-infeasible using image a result, FPNs are able to achieve higher accuracy thanall existing state-of-the-art methods. Moreover, this im-provement is achieved without increasing testing time overthe single-scale baseline. We believe these advances willfacilitate future research and applications. Our code will bemade publicly Related WorkHand-engineered features and early neural features [25] were originally extracted at scale-spaceextrema and used for Feature point matching. HOG fea-tures [5], and later SIFT features as well, were computeddensely over entire image pyramids.

10 These HOG and SIFT pyramids have been used in numerous works for imageclassification, Object Detection , human pose estimation, andmore. There has also been significant interest in comput-ing featurized image pyramids quickly. Doll aret al. [6]demonstrated fast Pyramid computation by first computinga sparsely sampled (in scale) Pyramid and then interpolat-ing missing levels. Before HOG and SIFT, early work onface Detection with ConvNets [38,32] computed shallownetworks over image pyramids to detect faces across ConvNet Object the developmentof modern deep ConvNets [19], Object detectors like Over-Feat [34] and R-CNN [12] showed dramatic improvementsin accuracy. OverFeat adopted a strategy similar to earlyneural network face detectors by applying a ConvNet asa sliding window detector on an image Pyramid . R-CNNadopted a region proposal-based strategy [37] in which eachproposal was scale-normalized before classifying with aConvNet.