Momentum Contrast for Unsupervised Visual Representation ...

1 Momentum Contrast for Unsupervised Visual Representation LearningKaiming HeHaoqi FanYuxin WuSaining XieRoss GirshickFacebook AI Research (FAIR)AbstractWe present Momentum Contrast (MoCo) for unsuper-vised Visual Representation learning. From a perspective oncontrastive learning [29] as dictionary look-up, we builda dynamic dictionary with a queue and a moving-averagedencoder. This enables building a large and consistent dic-tionary on-the-fly that facilitates contrastive provides competitive results under thecommon linear protocol on ImageNet classification.

2 Moreimportantly, the representations learned by MoCo transferwell to downstream tasks. MoCo canoutperformits super-vised pre-training counterpart in7detection/segmentationtasks on PASCAL VOC, COCO, and other datasets, some-times surpassing it by large margins. This suggests thatthe gap between Unsupervised and supervised representa-tion learning has been largely closed in many vision : IntroductionUnsupervised Representation learning is highly success-ful in natural language processing, , as shown by GPT[50, 51] and BERT [12]. But supervised pre-training is stilldominant in computer vision, where Unsupervised meth-ods generally lag behind.

3 The reason may stem from dif-ferences in their respective signal spaces. Language taskshave discrete signal spaces (words, sub-word units,etc.)for building tokenizeddictionaries, on which unsupervisedlearning can be based. Computer vision, in Contrast , furtherconcerns dictionary building [54, 9, 5], as the raw signal isin a continuous, high-dimensional space and is not struc-tured for human communication ( , unlike words).Several recent studies [61, 46, 36, 66, 35, 56, 2] presentpromising results on Unsupervised Visual representationlearning using approaches related to thecontrastive loss[29].

4 Though driven by various motivations, these methodscan be thought of as buildingdynamic dictionaries. The keys (tokens) in the dictionary are sampled from data( , images or patches) and are represented by an encodernetwork. Unsupervised learning trains encoders to performdictionary look-up: an encoded query should be similarto its matching key and dissimilar to others. Learning isformulated as minimizing a contrastive loss [29].encodermomentumencoderqcontrastive 1. Momentum Contrast (MoCo) trains a Visual represen-tation encoder by matching an encoded queryqto a dictionaryof encoded keys using a contrastive loss.

5 The dictionary keys{k0,k1,k2,..}are defined on-the-fly by a set of data dictionary is built as a queue, with the current mini-batch en-queued and the oldest mini-batch dequeued, decoupling it fromthe mini-batch size. The keys are encoded by a slowly progressingencoder, driven by a Momentum update with the query method enables a large and consistent dictionary for learningvisual this perspective, we hypothesize that it is desirableto build dictionaries that are: (i) largeand(ii) consistentas they evolve during training.

6 Intuitively, a larger dictio-nary may better sample the underlying continuous, high-dimensional Visual space, while the keys in the dictionaryshould be represented by the same or similar encoder so thattheir comparisons to the query are consistent. However, ex-isting methods that use contrastive losses can be limited inone of these two aspects (discussed later in context).We presentMomentum Contrast (MoCo) as a way ofbuilding large and consistent dictionaries for unsupervisedlearning with a contrastive loss (Figure 1). We maintain thedictionary as aqueueof data samples: the encoded repre-sentations of the current mini-batch are enqueued, and theoldest are dequeued.

7 The queue decouples the dictionarysize from the mini-batch size, allowing it to be large. More-over, as the dictionary keys come from the preceding sev-eral mini-batches, aslowly progressingkey encoder, imple-mented as a Momentum -based moving average of the queryencoder, is proposed to maintain [ ] 23 Mar 2020 MoCo is a mechanism for building dynamic dictionar-ies for contrastive learning, and can be used with variouspretext tasks. In this paper, we follow a simple instancediscrimination task [61, 63, 2]: a query matches a key ifthey are encoded views ( , different crops) of the sameimage.

8 Using this pretext task, MoCo shows competitiveresults under the common protocol of linear classificationin the ImageNet dataset [11].A main purpose of Unsupervised learning is to pre-trainrepresentations ( , features) that can be transferred todownstream tasks by fine-tuning. We show that in 7 down-stream tasks related to detection or segmentation, MoCounsupervised pre-training cansurpassits ImageNet super-vised counterpart, in some cases by nontrivial margins. Inthese experiments, we explore MoCo pre-trained on Ima-geNet or on aone-billionInstagram image set, demonstrat-ing that MoCo can work well in a more real-world, billion-image scale, and relatively uncurated scenario.

9 These re-sults show that MoCo largely closes the gap between un-supervised and supervised Representation learning in manycomputer vision tasks, and can serve as an alternative to Im-ageNet supervised pre-training in several Related WorkUnsupervised/self-supervised1learnin g methods gener-ally involve two aspects: pretext tasks and loss term pretext implies that the task being solved is notof genuine interest, but is solved only for the true purposeof learning a good data Representation . Loss functions canoften be investigated independently of pretext tasks.

10 MoCofocuses on the loss function aspect. Next we discuss relatedstudies with respect to these two common way of defining a loss functionis to measure the difference between a model s predictionand afixedtarget, such as reconstructing the input pixels( , auto-encoders) by L1 or L2 losses, or classifying theinput into pre-defined categories ( , eight positions [13],color bins [64]) by cross-entropy or margin-based alternatives, as described next, are also losses [29] measure the similarities of sam-ple pairs in a Representation space.