Co2L: Contrastive Continual Learning

1 Co2L: Contrastive Continual LearningHyuntak ChaJaeho LeeJinwoo ShinKAISTD aejeon, South Korea{ , jaeho-lee, breakthroughs in self-supervised Learning showthat such algorithms learn visual representations that can betransferred better to unseen tasks than cross-entropy basedmethods which rely on task-specific supervision. In this pa-per, we found that the similar holds in the Continual learningcontext: contrastively learned representations are more ro-bust against the catastrophic forgetting than ones trainedwith the cross-entropy objective. Based on this novel ob-servation, we propose a rehearsal-based Continual learningalgorithm that focuses on continually Learning and main-taining transferable representations.}

2 More specifically, theproposed scheme (1) learns representations using the con-trastive Learning objective, and (2) preserves learned rep-resentations using a self-supervised distillation step. Weconduct extensive experimental validations under popularbenchmark image classification datasets, where our methodsets the new state-of-the-art performance. Source code isavailable IntroductionModern deep Learning algorithms show impressive per-formances on the task at hand, but it is well known thatthey often struggle to retain their knowledge on previouslylearned tasks after being trained on a new one [32]. Tomitigate such catastrophic forgetting, prior works in thecontinual Learning literature focus onpreservingthe previ-ously learned knowledge using various types of informationabout the past task.

3 Replay-based approaches store a smallportion of past samples and rehearse the samples along withpresent task samples [35,29,34,5]. Regularization-basedapproaches force the current model to be sufficiently closeto the past model which may be informative about the pasttask in the parameter/functional space distance [25,6,39].Expansion-based approaches allocate a unit ( , networknode, sub-network) for each task and keep the unit untouchedduring the training for other tasks [38, 31].In this paper, instead of asking how to isolate previousknowledge from new knowledge, we draw attention to thefollowing fundamental question:What type of knowledge is likely to be usefulfor future tasks (and thus not get forgotten), andhow can we learn and preserve such knowledge?

4 To demonstrate its significance, consider the simple scenariothat the task at hand is to classify the given image as anapple or a banana. An easy way to solve this problem is toextract and use the color feature of the image; red meansapple, and yellow means banana. The color, however, willno longer be useful if our future task is to classify anotherset of images as apples or strawberries; color may not beused anymore and eventually get forgotten. On the otherhand, if the model had learned more complicated features, , shape/polish/texture, the features may be re-used forfuture tasks and remain unforgotten. This line of thoughtssuggests that forgetting does not only come from the limitedaccess to the past experience, but also from the innately re-stricted access to future events; to suffer less from forgetting, Learning moretransferable representationsin the first handmay be as important as carefully preserving the knowledgegained in the learn more transferable representations for continuallearning, we draw inspirations from a recent advance inself-supervised Learning , in particular, Contrastive Learning [19,10].

5 Contrastive methods learn representations usingthe inductive bias that the prediction should be invariantto certain input transformations instead of relying on task-specific supervisions. Despite their simplicity, such methodsare known to be surprisingly effective; for ImageNet clas-sification [37], contrastively trained representations closelyachieve the fully-supervised performance even without labels[10] and outperform counterparts in the supervised case [24].More importantly, while the methods are originally proposedfor better in-domain1performance, recent works also showthat such methods provide significant performance gains onunseen domains [10,21].

6 Under a Continual scenario, wemake a similar observation:contrastively learned represen-tations suffer less from forgetting than the ones trained withcross-entropy loss(see Section for details).1 The term in-domain is used here for the setup where data distributionsfor representation Learning and linear classifier training are the 1. An overview of the Co2L framework. Mini-batch samples from the current task and the memory buffer are augmented and passedthrough current and past (stored at the end of the previous task) representations. Co2L minimizes the weighted sum of two losses: (1)Asymmetric SupCon loss contrasts anchor samples from the current task against the samples from other classes (Section ).

7 (2) IRD lossmeasures the drift of the instance-wise similarities given by the current model from the one given by the previous model (Section ).Unfortunately, applying this idea to Continual settings isnot straightforward due to at least two reasons: First, hav-ing access to informative negative samples is known to becrucial for the success of Contrastive Learning [36], whilethe instantaneous demographics of negatives samples areseverely restricted under standard Continual setups; in class-incremental Learning , for instance, it is common to assumethat the learner can access samples from only a small numberof classes at each time step.

8 Second, the question of howto preserve the contrastively learnedrepresentationson con-tinual Learning setups has not been fully answered. Indeed,recent works on representation Learning for Continual setupsaim to learn representations accelerating future Learning un-der a similar decoupled Learning setup but lack an explicitdesign to preserve address these challenges, we propose anew rehearsal-based Continual Learning algorithm, coinedCo2L(ContrastiveContinualLearning) . Unlike previouscontinual (representation) Learning methods, we aim tolearnandpreserverepresentations continually in a decoupledrepresentation-classifier scheme.

9 The overview of Co2 Lis illustrated in Figure contribution under this setup is Learning :We design an asymmetric versionof supervised Contrastive loss for Learning representationsunder Continual Learning setup (Section ) and empir-ically show its benefits on improving the representations:We propose a novel preserva-tion mechanism for contrastively learned representations,which works by self-distillation of instance-wise rela-tions (Section ); to the best of our knowledge, this is afirst method explicitly designed to preserve contrastivelylearned representations on Continual validate Co2L under various experimental scenarios en-compassing task-incremental Learning , domain-incrementallearning, and class-incremental Learning .

10 Co2L consistentlyoutperforms all baselines on various datasets, scenarios, andmemory setups. With careful ablation studies, we also showthat both components we propose (asymmetric supervisedcontrastive loss, instance-wise relation distillation) are es-sential for performance. In the ablation of distillation, weempirically show that distillation preserves learned represen-tations and efficiently uses buffered samples, which mightbe the main source of consistent gains over all comparisons:distillation provides and relative improve-ments with/without buffered samples respectively on theSeq-CIFAR-10 dataset. In the ablation of asymmetric su-pervised Contrastive loss, we quantitatively verify that theasymmetric version consistently provides performance gainsover the original one on all setups, , relative im-provement on the Seq-CIFAR-10 with buffer size 500.