VideoMAE: Masked Autoencoders are Data-Efficient Learners ...

1 VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Zhan Tong 1 2 * Yibing Song 2 Jue Wang 2 Limin Wang 1. { , 100. Abstract previous SOTA without extra data [ ] 23 Mar 2022. from scratch (ViT-B). Pre-training video transformers on extra large- 80. MoCo V3. VideoMAE. scale datasets is generally required to achieve pre- Top-1 Accuracy (%). mier performance on relatively small datasets. In 60. this paper, we show that video Masked autoen- coders (VideoMAE) are data-efficient Learners for 40 self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE (He et al., 2021) 20 and propose customized video tube masking and reconstruction. These simple designs turn out to 0. UCF 101 HMDB 51. be effective for overcoming information leakage Figure 1.}

2 VideoMAE is a data-efficient learner that allows to caused by the temporal correlation during video effectively train video transformers only from limited video data reconstruction. We obtain three important find- ( , clips in UCF101, and clips in HMDB51) without ings on SSVP: (1) An extremely high proportion any ImageNet pre-training. VideoMAE significantly outperforms of masking ratio ( , 90% to 95%) still yields training from scratch, MoCo v3 pre-training (Chen et al., 2021b), favorable performance of VideoMAE. The tem- and the previous best performance (Diba et al., 2021) without extra porally redundant video content enables higher data on these small-scale video datasets. masking ratio than that of images. (2) VideoMAE. Brown et al., 2020; Radford et al.

3 , 2018). The vision trans- achieves impressive results on very small datasets former (Dosovitskiy et al., 2021) also improves a series of ( , around 3k-4k videos) without using any extra computer vision tasks including image classification (Tou- data. This is partially ascribed to the challenging vron et al., 2021; Zhou et al., 2021), object detection (Carion task of video reconstruction to enforce high-level et al., 2020; Liu et al., 2021a), semantic segmentation (Xie structure learning. (3) VideoMAE shows that data et al., 2021), and video recognition (Bertasius et al., 2021;. quality is more important than data quantity for Arnab et al., 2021). The multi-head self-attention upon lin- SSVP. Domain shift between pre-training and tar- early projected image/video tokens is capable of modeling get datasets are important issues in SSVP.

4 No- global dependency among visual content either spatially or tably, our VideoMAE with the vanilla ViT back- temporally. The inductive bias is effectively reduced via this bone can achieve on Kinects-400, flexible attention mechanism. on Something-Something V2, on UCF101, and on HMDB51 without using any extra Training effective vision transformers (ViTs) typically de- data. Code will be released here. pends on large-scale supervised datasets. Initially, pre- training ViTs achieves favorable performance by using hun- dreds of millions of labeled images (Dosovitskiy et al., 2021). For video transformers (Arnab et al., 2021; Bertasius 1. Introduction et al., 2021), they are usually derived from image-based Transformer (Vaswani et al., 2017) has brought significant transformers and heavily depend on the pre-trained models progress in natural language processing (Devlin et al.)

5 , 2019; from large-scale image data ( , ImageNet (Russakovsky et al., 2015)). Previous trials (Arnab et al., 2021; Bertasius *. Work is done during internship at Tencent AI Lab. 1 State et al., 2021) on training video transformers from scratch Key Laboratory for Novel Software Technology, Nanjing Uni- yield unsatisfied results (except for MViT (Fan et al., 2021). versity 2 Tencent AI Lab. Correspondence to: Limin Wang with a strong inductive bias). Therefore, the learned video transformers are naturally biased by image-based models, Preprint. Copyright 2022 by the authors. and it still remains a challenge that how to effectively train VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Time Time Time Time Encoder Decoder.

6 Downsampled video clip Tube masking Target video clip Tokens w/o [M]. keeping masking Figure 2. Our VideoMAE Architecture. Similar to ImageMAE (He et al., 2021), our VideoMAE performs the task of masking random cubes and reconstructing the missing ones with an asymmetric encoder-decoder architecture. Due to high redundancy and temporal correlation in videos, we present two customized designs in our VideoMAE: (1) an extremely high masking ratio (90% to 95%), (2) a tube masking strategy. These two core designs enable us to create a challenging self-supervised task to alleviate the information leakage during reconstruction and make the learned representations capture useful spatiotemporal structures. a vanilla vision transformer from video itself without using due to temporal redundancy, we use an extremely high any pre-trained model or extra image data.

7 In addition, masking ratio to drop the cubes from the downsampled the existing video datasets are relatively small compared clips. This simple strategy not only effectively increases with image datasets, which further adds the difficulty of the pre-training performance but also greatly reduces the training video transformers from scratch. Meanwhile, self- computational cost due to the asymmetric encoder-decoder supervised learning has shown remarkable performance by architecture. Second, to consider temporal correlation, we using large-scale image datasets (Chen et al., 2021b; Caron devise a simple yet effective tube masking strategy, which et al., 2021). The learned representations have outperformed turns out to be helpful to relieve the risk of information the ones via supervised learning when being transferred to leakage during reconstruction.

8 With these two core de- downstream tasks. It is expected that this self-supervised signs in our VideoMAE, we are able to successfully train learning paradigm can provide a promising solution to ad- vanilla ViT backbones on the relatively small-scale video dress the challenge of training video transformers. datasets such as Something-Something (Goyal et al., 2017), UCF101 (Soomro et al., 2012), and HMDB51 (Kuehne et al., Following the success of Masked autoencoding in NLP (De- 2011), which significantly outperform the previous state of vlin et al., 2019) and images (He et al., 2021; Bao et al., the art under the setting without extra data. In summary, the 2021), we present a new self-supervised video pre-training main contribution of this paper is threefold: (SSVP) method, termed as Video Masked Autoencoder (VideoMAE).

9 Our VideoMAE inherits the simple pipeline To our best knowledge, we present the first Masked of masking random cubes and reconstructing the missing video modeling framework that performs well for ones. However, the extra time dimension of videos makes it SSVP on relatively small-scale video datasets. To ad- different from images in this Masked modeling. First, video dress the information leakage issue caused by temporal frames are often densely captured, and their semantics varies redundancy and correlation in video data, we present slowly in time (Zhang & Tao, 2012). This temporal redun- two core designs of extremely high masking ratio and dancy would increase the risk of recovering missing pixels tube masking, which turn out to be the key to the suc- from the spatiotemporal neighborhood with little high-level cess of our VideoMAE.

10 Understanding. Furthermore, video could be viewed as the temporal evolution of static appearance, and there exists Aligned with the results in NLP and Images on Masked a correspondence between frames. This temporal correla- modeling, our VideoMAE demonstrates that this sim- tion could lead to information leakage during reconstruction ple masking and reconstruction strategy provides a unless a specific masking strategy is considered. In this good solution to self-supervised video pre-training. sense, for each Masked cube, it is easy to find a correspond- The models pre-trained with our VideoMAE signif- ing and unmasked copy in adjacent frames. This property icantly outperform those trained from scratch or pre- would make the learned models identify some shortcut trained with contrastive learning methods.