SeqFormer: a Frustratingly Simple Model for Video Instance ...

1 SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation Junfeng Wu1 Yi Jiang2 Wenqing Zhang1 Xiang Bai1 Song Bai2. 1 2. Huazhong University of Science and Technology ByteDance [ ] 15 Dec 2021. 60. Abstract SeqFormer 55. In this work, we present SeqFormer, a Frustratingly sim- YouTube-VIS 2019 AP. ple Model for Video Instance segmentation. SeqFormer fol- 50. Propose-Reduce lows the principle of vision transformer that models in- stance relationships among Video frames. Nevertheless, we 45 IFC. observe that a stand-alone Instance query suffices for cap- 40 VisTR. turing a time sequence of instances in a Video , but atten- CrossVIS.

2 Tion mechanisms should be done with each frame indepen- STEm-Seg 35. dently. To achieve this, SeqFormer locates an Instance in Mask-Track each frame and aggregates temporal information to learn 30. a powerful representation of a Video -level Instance , which 30 80 130 180 230. is used to predict the mask sequences on each frame dy- Params(Millions). namically. Instance tracking is achieved naturally with- Figure 1. Performance vs. Model Size. All results are reported out tracking branches or post-processing. On the YouTube- with single Model and single-scale inference. SeqFormer signifi- VIS dataset, SeqFormer achieves AP with a ResNet-50 cantly outperforms the previous method with similar parameters.

3 Backbone and AP with a ResNet-101 backbone with- out bells and whistles. Such achievement significantly ex- ceeds the previous state-of-the-art performance by and categories. The first mainstream follows the tracking-by- , respectively. In addition, integrated with the recently- detection paradigm, extending image Instance segmentation proposed Swin transformer, SeqFormer achieves a much models with a tracking branch [3,13,26,27]. These methods higher AP of We hope SeqFormer could be a strong first predict candidate detection and segmentation frame- baseline that fosters future research in Video Instance seg- by-frame, and then associate them by classification [27, 27].

4 Mentation, and in the meantime, advances this field with or re-identification [3, 13] to track the Instance through a a more robust, accurate, neat Model . The code and the Video . However, the tracking process is sensitive to occlu- pre-trained models are publicly available at https:// sions and motion blur that are common in videos. Another mainstream predicts clip-level Instance masks by taking a Video clip [1, 2] or the entire Video [8, 24] as input. It di- vides a Video into multiple overlapping clips and generates 1. Introduction mask sequences with clip-by-clip matching on overlapping frames. Recently, VisTR [24] adapts transformer [22] to Video Instance Segmentation (VIS) [26] is an emerg- VIS and uses Instance queries to obtain Instance sequence ing vision task that aims to simultaneously perform detec - from Video clips.

5 After that, IFC [8] improves the perfor- tion, classification, segmentation, and tracking of object in- mance and efficiency by building communications between stances in videos. Compared to image Instance segmenta- frames in a transformer encoder. tion [6], Video Instance segmentation is much more chal- In this paper, we present Sequential Transformer (Se- lenging since it requires accurate tracking of objects across qFormer), which follows the principle of vision trans- an entire Video . former [4, 24] and models Instance relationships among Previous VIS algorithms can be roughly divided into two Video frames. As in [8], we observe that a stand-alone in- Work done during an internship at ByteDance AI Lab.

6 Stance query suffices although an object may be of different Corresponding author. positions, sizes, shapes, and various appearances. Neverthe- less, it is witnessed that the attention process shall be done The code and the pre-trained models are publicly avail- with each frame independently, so that the Model will attend able. We hope the SeqFormer, with the idea of mak- to locations following the movement of Instance through the ing attention follow the movement of object, could be Video . This observation aligns with the conclusion drawn a strong baseline that fosters future research in Video in action recognition [17, 28], in which the 1D time do- Instance segmentation, and in the meantime, advances main and 2D space domain have different characteristics this field with a more robust, accurate, neat Model .

7 And should be handled in a different fashion. Considering the movement of an Instance in a Video , a 2. Related Work Model is supposed to attend to different spatial locations following the motion of the Instance . We decompose the Image Instance Segmentation Instance Segmentation is shared Instance query into frame-level box queries for the the most fundamental and challenging task in computer vi- attention mechanism to guarantee that the attention focuses sion, which aims to detect every Instance and segment every on the same Instance on each frame. The box queries are pixel respectively in static images. Instance segmentation kept on each frame and used to predict the bounding box was dominated by Mask R-CNN [6] for a long time, Mask sequences.

8 Then the features within the bounding boxes R-CNN [6] directly introduces fully convolutional mask are aggregated to refine the box queries on the current head to Faster R-CNN [18] in a multi-task learning manner. frame. By repeating this refinement process through de- Recently, one stage models emerged as excellent frame- coder layers, SeqFormer locates the Instance in each frame works for Instance segmentation. Solo [23] and CondInst in a coarse-to-fine manner, in a similar way to Deformable [20] propose one stage Instance segmentation pipeline and DETR [29]. achieve comparable performance. CondInst [20] proposes However, to mitigate redundant information from non- to dynamically generate the mask head parameters for each Instance frames, those box queries are aggregated in a Instance , which is used to predict the mask of the corre- weighted manner, where the weights are learned upon the sponding Instance .

9 Dynamic mask head can be efficiently box embeddings. The generated representation, which re- adopted into Video segmentation tasks because the same in- tains richer object cues, is used to predict the category and stance in different frames can share the same mask head. generate dynamic convolution weights of mask head. Since Video Instance Segmentation. Video Instance segmenta- the box sequences are predicted and refined in the decoder, tion is extended from the traditional image Instance segmen- SeqFormer naturally and succinctly establishes the associa- tation, and aims to simultaneously segment and track all ob- tion of instances across frames.

10 Ject instances in the Video . The baseline method MaskTrack In summary, SeqFormer enjoys the following advan- R-CNN [26] is built upon Mask R-CNN [6] and introduces tages: a tracking head to associate each Instance in the Video . Sip- SeqFormer is a neat and Simple end-to-end framework. Mask [3] follows the similar pipeline based on the one- Given an arbitrary long Video as input, SeqFormer stage FCOS [21]. MaskProp [2] introduces a mask prop- predicts the classification results, box sequences, and agation module that propagates Instance masks from each mask sequences in one step without the need for addi- Video frame to all the other frames in a Video clip, which tional tracking branches or hand-craft post-processing.