Dynamic DETR: End-to-End Object Detection With Dynamic ...

1 Dynamic DETR: End-to-End Object Detection with Dynamic AttentionXiyang DaiYinpeng ChenJianwei YangPengchuan ZhangLu YuanLei ZhangMicrosoft{xidai, yiche, jianwyan, penzhan, luyuan, this paper, we present a novel Dynamic DETR (De-tection with Transformers) approach by introducing dy-namic attentions into both the encoder and decoder stagesof DETR to break its two limitations on small feature res-olution and slow training convergence. To address thefirst limitation, which is due to the quadratic computa-tional complexity of the self-attention module in Trans-former encoders, we propose adynamic encoderto ap-proximate the Transformer encoder s attention mechanismusing a convolution-based Dynamic encoder with variousattention types.}

2 Such an encoder can dynamically adjustattentions based on multiple factors such as scale impor-tance, spatial importance, and representation ( , featuredimension) importance. To mitigate the second limitationof learning difficulty, we introduce adynamic decoderbyreplacing the cross-attention module with a ROI-based dy-namic attention in the Transformer decoder. Such a decodereffectively assists Transformers to focus on region of inter-ests from a coarse-to-fine manner and dramatically lowersthe learning difficulty, leading to a much faster convergencewith fewer training epochs.

3 We conduct a series of experi-ments to demonstrate our advantages. Our Dynamic DETR significantly reduces the training epochs (by14 ), yet re-sults in a much better performance ( mAP). Mean-while, in the standard1 setup with ResNet-50 backbone,we archive a new state-of-the-art performance that furtherproves the learning effectiveness of the proposed IntroductionObject Detection aims at predicting a set of boundingboxes and category labels for each Object of interest. Mod-ern Object detectors are based on convolutional neural net-works, and share the same paradigm a backbone for fea-ture extraction and a head for localization and classifica-tion tasks [22, 10].

4 Until recently, Detection Transformer(DETR) has been proposed as an alternative solution to theFigure 1. Convergence curve comparison between our proposedapproach and state-of-the-art Dynamic DETR largely reduces the training epochs (by14 ), yet results in a sig-nificantly better performance (by ). Object Detection problem. It views Object Detection as aset-based matching problem. By leveraging Transformers[25] originally developed for language tasks, it is able tomodel the relations of objects and their global image con-text from a set of learned Object queries.

5 It then performs aglobal optimization that forces unique predictions from ob- ject queries via bipartite matching, effectively removing theneed of hand-designed components such as non-maximumsuppression (NMS) and anchor generation in traditional ob- ject Detection , DETR suffers from several problems that pre-vent it from wide adoption in the community. On one hand,the input resolution of features maps is limited in the nativeTransformer as feature encoder, since the complexity of theself-attention module grows quadratically with the increaseof the input resolution.

6 It results in incompatibility to thetypical feature pyramid that is widely used in modern ob- ject detectors, and relatively low performance at detectingsmall objects. On the other hand, it requires much longer2988training epochs to converge than the existing Object detec-tors since the cross-attention module struggles to learn on alarge global feature map from an initial dense attention toa final sparse attention. Thus, it is the high demand of aneffective solution for improving work Deformable DETR [29], which combinesthe sparse spatial sampling of deformable convolution, andthe relation modeling capability of Transformers, to miti-gate the slow convergence and high complexity issues ofDETR.

7 It has achieved noticeable improvements on perfor-mance and efficiency in training. It is interesting to exploitif the efficiency and performance of DETR can be this paper, we propose an alternative solution to ad-dress the above two problems of DETR by a Dynamic at-tention framework, calledDynamic DETR, which consistsof a Dynamic encoder and a Dynamic decoder. We replacethe Transformer encoder in DETR with a new convolution-based Dynamic encoder, which apply Dynamic attention onfull scales of feature pyramid based on scale importance,spatial importance, and representation ( , feature dimen-sion) importance.

8 Since it makes self-attentions feasible onfull scale of representations from low to high resolutions,the performance of DETR can be significantly boosted. Inaddition, the Dynamic decoder replaces the cross-attentionmodule in the DETR decoder with a ROI-based Dynamic at-tention, which can effectively assists Transformers to focuson regions of interest in a coarse-to-fine manner and dra-matically lowers the learning difficulty, leading to a muchfaster convergence with fewer training contribution can be summarized in three-folds.

9 We propose a novelDynamic DETR approach, whichcoherently combines a Dynamic convolution-based en-coder and a Dynamic Transformer-based decoder. Theproposed approach significantly improves the repre-sentation ability of Object Detection head and the learn-ing efficiency without any computational overhead. Compared to the original DETR, Our Dynamic DETR largely reduces the training epochs (by14 ), yet re-sults in a significantly better performance (by ),shown in Figure 1. To our best knowledge, we are the first end-to-endmethod that achieves a better than traditional perfor-mance in the standard 1x setup with a ResNet-50 back-bone, at BackgroundFeature objects at vastly differ-ent scales that co-existed in natural images is a fundamen-tal challenge in computer vision.

10 Researches have exploredmany directions to incorporate multiple scale into Object de-tection. [6] first introduced image pyramid to Object de-tection by independently computing features on differentscales and regions of the images separately. Later, [22, 21]discarded such idea and only unitized single scale featuresdue to the slow speed of extracting features from multiplescales of images. To overcome the obvious disadvantages ofimage pyramid , feature pyramid [14] was proposed, whichcombined both the feature hierarchy computed from a neu-ral network and a top down architecture with lateral connec-tions to enhance the semantics at all levels.