Semi-Supervised Semantic Segmentation With Directional ...

1 Semi-Supervised Semantic Segmentation with Directional Context-awareConsistencyXin Lai1*Zhuotao Tian1 Li Jiang1 Shu Liu2 Hengshuang Zhao3 Liwei Wang1 Jiaya Jia1,21 The Chinese University of Hong Kong2 SmartMore3 University of Segmentation has made tremendous progress inrecent years. However, satisfying performance highly de-pends on a large number of pixel-level annotations. There-fore, in this paper, we focus on the Semi-Supervised seg-mentation problem where only a small set of labeled data isprovided with a much larger collection of totally unlabeledimages. Nevertheless, due to the limited annotations, mod-els may overly rely on the contexts available in the trainingdata, which causes poor generalization to the scenes un-seen before. A preferred high-level representation shouldcapture the contextual information while not losing self-awareness. Therefore, we propose to maintain the context-aware consistency between features of the same identity butwith different contexts, making the representations robust tothe varying environments.

2 Moreover, we present the Direc-tional Contrastive Loss (DC Loss) to accomplish the consis-tency in a pixel-to-pixel manner, only requiring the featurewith lower quality to be aligned towards its addition, to avoid the false-negative samples and filterthe uncertain positive samples, we put forward two sam-pling strategies. Extensive experiments show that our sim-ple yet effective method surpasses current state-of-the-artmethods by a large margin and also generalizes well withextra image-level IntroductionSemantic Segmentation , as a fundamental tool, has prof-ited many downstream applications, and deep learning fur-ther boosts this area with remarkable progress. However,training a strong Segmentation network highly relies on suf-ficient finely annotated data to yield robust representationsfor input images, and dense pixel-wise labeling is rathertime-consuming, , the annotation process costs morethan on average for a single image in Cityscapes [12].

3 *Equal ContributionInputSupOnlyOursFigure 1. Grad-CAM [46] visualizations of the regional contribu-tion to the feature of interest ( , the yellow cross shown in theinput). The red region corresponds to high contribution. SupOnly:the model trained with only 1/8 labeled data. More illustrationsare shown in the alleviate this problem, weaker forms of segmentationannotation, , bounding boxes [13,48], image-level la-bels [56,1,44] and scribbles [34,51,52], have been ex-ploited to supplement the limited pixel-wise labeled , collecting these weak labels requires additional hu-man efforts. Instead, in this paper, we focus on the Semi-Supervised scenario where the Segmentation models aretrained with a small set of labeled data and a much largercollection of unlabeled networks can not predict a label for eachpixel merely based on its RGB values. Therefore, the con-textual information is essential for Semantic models ( , DeepLab [7] and PSPNet [60]) havealso shown satisfying performance by adequately aggregat-ing the contextual cues to individual pixels before makingfinal predictions.

4 However, in the Semi-Supervised setting,models are prone to overfit the quite limited training data,which results in poor generalization on the scenes unseenduring training. In this case, models are easy to excessivelyrely on the contexts to make predictions. Empirically, asshown in , we find that after training with only the la-beled data, features oftrainandpersonoverly focus on thecontexts ofskyanddogbut overlook themselves. There-fore, to prevent the model abusing the contexts and also1205 Unlabeled ImageCrop1 Crop2 Figure 2. Crop1 and Crop2 are randomly cropped from the sameimage with an overlapping region. The consistency (representedby the solid white line) is maintained between representations forthe overlapping region in the two crops under different contexts(represented by the dashed white line), in a pixel-to-pixel enhance self-awareness, our solution in this work isto make the representations more robust to the changing en-vironments, which we call thecontext-aware , as shown in , we crop two randompatches from an unlabeled image and they are confinedto have an overlapping region, which can be deemed thatthe overlapping region is placed into two different envi-ronments, ,contextual augmentation.

5 Even though theground-truth labels are unknown, the consistency of high-level features under different environments can still bemaintained because there exists a pixel-wise one-to-one re-lationship between the overlapping regions of the two accomplish the consistency, we propose the DirectionalContrastive Loss that encourages the feature to align to-wards the one with generally higher quality, rather than bi-laterally in the vanilla contrastive loss. Also, we put forwardtwo effective sampling strategies that filter out the com-mon false negative samples and the uncertain positive sam-ples respectively. Owing to the context-aware consistencyand the carefully designed sampling strategies, the proposedmethod brings significant performance gain to the proposed method is simple yet effective. Only afew additional parameters are introduced during trainingand the original model is kept intact for inference, so it canbe easily applied to different models without structural con-straints.

6 Extensive experiments on PASCAL VOC [15] andCityscapes [12] show the effectiveness of our sum, our contributions are three-fold: To alleviate the overfitting problem, we propose tomaintain context-aware consistency between pixelsunder different environments to make models robustto the contextual variance. To accomplish the contextual alignment, we design theDirectional Contrastive Loss, which applies the con-trastive learning in a pixel-wise manner. Also, two ef-fective sampling strategies are proposed to further im-prove performance. Extensive experiments demonstrate that our proposedmodel surpasses current state-of-the-art methods by alarge margin. Moreover, our method can be extendedto the setting with extra image-level Related WorkSemantic SegmentationSemantic Segmentation is a fun-damental yet rather challenging task. High-level seman-tic features are used to make predictions for each [47] is the first Semantic Segmentation network to re-place the last fully-connected layer in a classification net-work by convolution layers.

7 As the final outputs of FCN aresmaller than the input images, methods based on encoder-decoder structures [40,2,45] are demonstrated to be ef-fective by refining the outputs step by step. Although thesemantic information has been encoded in the high-leveloutput features, it cannot well capture the long-range re-lationships. Therefore, dilated convolution [6,58], globalpooling [36], pyramid pooling [60,59,57] and attentionmechanism [24,21,61,63] are used to better aggregate thecontexts. Despite the success of these models, they all needsufficient pixel-wise annotations to accomplish representa-tion learning, which costs lots of human LearningSemi-supervised learningaims to exploit unlabeled data to further improve the rep-resentation learning given limited labeled data [16,30,35,28]. Adversarial based methods [14,32,50] lever-age discriminators to align the distributions of labeledand unlabeled data in the embedding space.

8 Our methodin this paper follows another line based on [39] applies adversarial perturbations to the outputand -Model [29] applies different data augmentations anddropout to form the perturbed samples and aligns betweenthem. Dual Student [26] generates perturbed outputs forthe same input via two networks with different initializa-tions. Data interpolation is another feasible way to get per-turbed samples in MixMatch [4] and ReMixMatch [3]. Be-sides, consistency training can be accomplished with con-fident target samples. Temporal Model [29] ensembles thepredictions over epochs as the targets and makes the out-puts consistent with them. Mean Teacher [53] yields thetarget samples via exponential moving average. Also, ideasof self-supervised learning have been exploited to tackle thesemi-supervised learning recently [54,5], and we also in-corporate the contrastive loss that has been well studied inthe self-supervised learning [17,19,11,27,9,10] as theconstraint to accomplish consistency Semantic SegmentationPixel-wiselabelling is more costly than image-level annotations.

9 Weaklabels including bounding boxes [13,48], image-level la-bels [56,1,44,55] and scribbles [34,51,52] are used toalleviate this issue, but they still require human efforts. Toexploit the unlabeled data, adversarial learning and consis-tency training are leveraged for Semi-Supervised segmenta-tion. Concretely, both AdvSemiSeg [23] and S4 GAN [38]utilize a discriminator to provide additional supervision tounlabeled samples. Similar to Mean Teacher, S4 GAN [38]1206I. OriginalII. Contextual augIII. Low-level augFigure 3. Visual comparison betweencontextual augmentation(Iand II) andlow-level augmentation(I and III) using t-SNE visu-alization for features of the overlapping region (shown in yellowbox).Top: input crops from the same image, where II and III ap-ply thecontextualandlow-level : t-SNE results of the model trained with labeled data that the three visualizations are in the same t-SNE space, andthe dots with the same color represent the features of the : t-SNE results of our uses the teacher-student framework and the final multi-class classifier to filter out uncertain categories by scalingthe predictions.

10 [49] adds new samples that are synthesizedbased on the unlabeled data. The idea of self-correctionhas been exploited in ECS [37] and GCT [25] by creatingthe Correction Network and Flaw Detector respectively toamend the defects in predictions. Nevertheless, CCT [41]aligns the outputs of the main decoder and several auxiliarydecoders with different perturbations to enforce a consis-tency that improves feature representations. Unlike thesemethods, our proposed context-aware consistency bringssignificant performance gain by explicitly alleviating thecontextual bias caused by limited training MethodIn the following sections, we firstly present our motiva-tion in , and then elaborate the proposed context-aware consistency in Also, to accomplish theconsistency, we propose the Directional Contrastive Loss Moreover, two sampling strategies further improvethe baseline as shown in In , our methodgeneralizes well with extra image-level MotivationConsistency-based methods [29,53,39] have achieveddecent performance in Semi-Supervised learning by main-taining the consistency between perturbed images or fea-tures to learn robust representations.