EfficientDet: Scalable and Efficient Object Detection

1 EfficientDet: Scalable and Efficient Object Detection Mingxing Tan Ruoming Pang Quoc V. Le Google Research, Brain Team {tanmingxing, rpang, [ ] 27 Jul 2020. Abstract EfficientDet-D7. D6. D5. Model efficiency has become increasingly important in 50. AmoebaNet + NAS-FPN + AA. D4. computer vision. In this paper, we systematically study neu- D3. ral network architecture design choices for Object Detection 45 ResNet + NAS-FPN. and propose several key optimizations to improve efficiency. D2. COCO AP. First, we propose a weighted bi-directional feature pyra- mid network (BiFPN), which allows easy and fast multi- RetinaNet 40 D1. scale feature fusion; Second, we propose a compound scal- Mask R-CNN. ing method that uniformly scales the resolution, depth, and AP FLOPs (ratio). 35. width for all backbone, feature network, and box/class pre- EfficientDet-D0 YOLOv3 [34] 71B (28x). diction networks at the same time. Based on these optimiza- YOLOv3 EfficientDet-D1 RetinaNet [24] 97B (16x).}

2 Tions and better backbones, we have developed a new family 30 EfficientDet-D7x 410B. of Object detectors, called EfficientDet, which consistently AmoebaNet+ NAS-FPN +AA [45] 3045B (13x). Not plotted. achieve much better efficiency than prior art across a wide 0 200 400 600 800 1000 1200. spectrum of resource constraints. In particular, with single- FLOPs (Billions). model and single-scale, our EfficientDet-D7 achieves state- Figure 1: Model FLOPs vs. COCO accuracy All num- of-the-art AP on COCO test-dev with 77M param- bers are for single-model single-scale. Our EfficientDet eters and 410B FLOPs1 , being 4x 9x smaller and using achieves new state-of-the-art COCO AP with much 13x 42x fewer FLOPs than previous detectors. Code is fewer parameters and FLOPs than previous detectors. More available at studies on different backbones and FPN/NAS-FPN/BiFPN. master/efficientdet. are in Table 4 and 5. Complete results are in Table 2.

3 Stage [27, 33, 34, 24] and anchor-free detectors [21, 44, 40], 1. Introduction or compress existing models [28, 29]. Although these meth- ods tend to achieve better efficiency, they usually sacrifice Tremendous progresses have been made in recent years accuracy. Moreover, most previous works only focus on a towards more accurate Object Detection ; meanwhile, state- specific or a small range of resource requirements, but the of-the-art Object detectors also become increasingly more variety of real-world applications, from mobile devices to expensive. For example, the latest AmoebaNet-based NAS- datacenters, often demand different resource constraints. FPN detector [45] requires 167M parameters and 3045B. A natural question is: Is it possible to build a scal- FLOPs (30x more than RetinaNet [24]) to achieve state-of- able Detection architecture with both higher accuracy and the-art accuracy. The large model sizes and expensive com- better efficiency across a wide spectrum of resource con- putation costs deter their deployment in many real-world straints ( , from 3B to 300B FLOPs)?

4 This paper aims applications such as robotics and self-driving cars where to tackle this problem by systematically studying various model size and latency are highly constrained. Given these design choices of detector architectures. Based on the one- real-world resource constraints, model efficiency becomes stage detector paradigm, we examine the design choices for increasingly important for Object Detection . backbone, feature fusion, and class/box network, and iden- There have been many previous works aiming to de- tify two main challenges: velop more efficient detector architectures, such as one- Challenge 1: efficient multi-scale feature fusion Since 1 Similar to [14, 39], FLOPs denotes number of multiply-adds. introduced in [23], FPN has been widely used for multi- 1. scale feature fusion. Recently, PANet [26], NAS-FPN [10], have attracted substantial attention due to their efficiency and other studies [20, 18, 42] have developed more network and simplicity [21, 42, 44].

5 In this paper, we mainly follow structures for cross-scale feature fusion. While fusing dif- the one-stage detector design, and we show it is possible ferent input features, most previous works simply sum them to achieve both better efficiency and higher accuracy with up without distinction; however, since these different input optimized network architectures. features are at different resolutions, we observe they usu- Multi-Scale Feature Representations: One of the main ally contribute to the fused output feature unequally. To difficulties in Object Detection is to effectively represent and address this issue, we propose a simple yet highly effective process multi-scale features. Earlier detectors often directly weighted bi-directional feature pyramid network (BiFPN), perform predictions based on the pyramidal feature hierar- which introduces learnable weights to learn the importance chy extracted from backbone networks [4, 27, 36].

6 As one of different input features, while repeatedly applying top- of the pioneering works, feature pyramid network (FPN). down and bottom-up multi-scale feature fusion. [23] proposes a top-down pathway to combine multi-scale Challenge 2: model scaling While previous works features. Following this idea, PANet [26] adds an extra mainly rely on bigger backbone networks [24, 35, 34, 10] or bottom-up path aggregation network on top of FPN; STDL. larger input image sizes [13, 45] for higher accuracy, we ob- [43] proposes a scale-transfer module to exploit cross-scale serve that scaling up feature network and box/class predic- features; M2det [42] proposes a U-shape module to fuse tion network is also critical when taking into account both multi-scale features, and G-FRNet [2] introduces gate units accuracy and efficiency. Inspired by recent works [39], we for controlling information flow across features. More re- propose a compound scaling method for Object detectors, cently, NAS-FPN [10] leverages neural architecture search which jointly scales up the resolution/depth/width for all to automatically design feature network topology.

7 Although backbone, feature network, box/class prediction network. it achieves better performance, NAS-FPN requires thou- Finally, we also observe that the recently introduced Effi- sands of GPU hours during search, and the resulting feature cientNets [39] achieve better efficiency than previous com- network is irregular and thus difficult to interpret. In this monly used backbones. Combining EfficientNet backbones paper, we aim to optimize multi-scale feature fusion with a with our propose BiFPN and compound scaling, we have more intuitive and principled way. developed a new family of Object detectors, named Effi- cientDet, which consistently achieve better accuracy with Model Scaling: In order to obtain better accuracy, it much fewer parameters and FLOPs than previous Object is common to scale up a baseline detector by employing detectors. Figure 1 and Figure 4 show the performance bigger backbone networks ( , from mobile-size models comparison on COCO dataset [25].)

8 Under similar accu- [38, 16] and ResNet [14], to ResNeXt [41] and AmoebaNet racy constraint, our EfficientDet uses 28x fewer FLOPs than [32]), or increasing input image size ( , from 512x512. YOLOv3 [34], 30x fewer FLOPs than RetinaNet [24], and [24] to 1536x1536 [45]). Some recent works [10, 45] show 19x fewer FLOPs than the recent ResNet based NAS-FPN that increasing the channel size and repeating feature net- [10]. In particular, with single-model and single test-time works can also lead to higher accuracy. These scaling scale, our EfficientDet-D7 achieves state-of-the-art AP methods mostly focus on single or limited scaling dimen- with 77M parameters and 410B FLOPs, outperforming pre- sions. Recently, [39] demonstrates remarkable model effi- vious best detector [45] by 4 AP while being smaller ciency for image classification by jointly scaling up network and using fewer FLOPs. Our EfficientDet is also up to width, depth, and resolution.

9 Our proposed compound scal- 4x to 11x faster on GPU/CPU than previous detectors. ing method for Object Detection is mostly inspired by [39]. With simple modifications, we also demonstrate that our single-model single-scale EfficientDet achieves 3. BiFPN. mIOU accuracy with 18B FLOPs on Pascal VOC 2012 se- In this section, we first formulate the multi-scale feature mantic segmentation, outperforming DeepLabV3+ [6] by fusion problem, and then introduce the main ideas for our better accuracy with fewer FLOPs. proposed BiFPN: efficient bidirectional cross-scale connec- tions and weighted feature fusion. 2. Related Work Problem Formulation One-Stage Detectors: Existing Object detectors are mostly categorized by whether they have a region-of- Multi-scale feature fusion aims to aggregate features at interest proposal step (two-stage [11, 35, 5, 13]) or not (one- different resolutions. Formally, given a list of multi-scale stage [36, 27, 33, 24]).

10 While two-stage detectors tend to be features P~ in = (Plin 1. , Plin 2. , ..), where Plin i represents the more flexible and more accurate, one-stage detectors are of- feature at level li , our goal is to find a transformation f that ten considered to be simpler and more efficient by leverag- can effectively aggregate different features and output a list ing predefined anchors [17]. Recently, one-stage detectors of new features: P~ out = f (P~ in ). As a concrete example, 2. repeated blocks repeated blocks P7 P7 P7. P7. P6 P6 P6. P6. P5 P5 P5. P5. P4 P4 P4. P4. P3 P3 P3. P3. (a) FPN (b) PANet (c) NAS-FPN (d) BiFPN. Figure 2: Feature network design (a) FPN [23] introduces a top-down pathway to fuse multi-scale features from level 3 to 7 (P3 - P7 ); (b) PANet [26] adds an additional bottom-up pathway on top of FPN; (c) NAS-FPN [10] use neural architecture search to find an irregular feature network topology and then repeatedly apply the same block ; (d) is our BiFPN with better accuracy and efficiency trade-offs.