PANet: Few-Shot Image Semantic Segmentation With …

1 PANet: Few-Shot Image Semantic Segmentation with Prototype AlignmentKaixin Wang1 Jun Hao Liew2 Yingtian Zou2 Daquan Zhou1 Jiashi Feng21 NGS, National University of Singapore2 ECE Department, National University of Singapore{ , the great progress made by deep CNNs in imagesemantic Segmentation , they typically require a large num-ber of densely-annotated images for training and are diffi-cult to generalize to unseen object categories. Few-Shot seg-mentation has thus been developed to learn to perform seg-mentation from only a few annotated examples. In this pa-per, we tackle the challenging Few-Shot Segmentation prob-lem from a metric learning perspective and present PANet,a novel prototype alignment network to better utilize theinformation of the support set. Our PANet learns class-specific prototype representations from a few support im-ages within an embedding space and then performs segmen-tation over the query images through matching each pixel tothe learned prototypes.}

2 With non-parametric metric learn-ing, PANet offers high-quality prototypes that are represen-tative for each Semantic class and meanwhile discriminativefor different classes. Moreover, PANet introduces a proto-type alignment regularization between support and this, PANet fully exploits knowledge from the supportand provides better generalization on Few-Shot segmenta-tion. Significantly, our model achieves the mIoU score and on PASCAL-5ifor 1-shot and 5-shot set-tings respectively, surpassing the state-of-the-art method and IntroductionDeep learning has greatly advanced the development ofsemantic Segmentation with a number of CNN based ar-chitectures like FCN [13], SegNet [1], DeepLab [2] andPSPNet [29]. However, training these models typicallyrequires large numbers of images with pixel-level annota-tions which are expensive to obtain.

3 Semi- and weakly-supervised learning methods [26, 3, 9, 15] alleviate suchrequirements but still need many weakly annotated trainingimages. Besides their hunger for training data, these modelsalso suffer rather poor generalizability to unseen classes. Todeal with the aforementioned challenges, Few-Shot learning,which learns new concepts from a few annotated examples,has been actively explored, mostly concentrating on imageFigure 1: Overview of our model (PANet) for Few-Shot seg-mentation. PANet first maps the support and query imagesinto embedding features (circles and triangles respectively)and learns prototypes for each class (blue and yellow solidcircles). Segmentation over the query is then performed bymatching its features to a nearest prototype within the em-bedding space (dashed lines).

4 PANet further introduces aprototype alignment regularization during training to alignthe prototypes from support and query images within theembedding space by performing Few-Shot Segmentation re-versely from query to support (right panel). Segmentationmasks with dashed border denote ground truth [25, 23, 24, 18, 6, 20, 12, 14] and a few tar-geting at Segmentation tasks [21, 17, 4, 28, 4, 8].Existing Few-Shot Segmentation methods generally learnfrom a handful ofsupportimages and then feed learnedknowledge into a parametric module for segmenting thequery. However, such schemes have two drawbacks andthus generalize unsatisfactorily. First, they do not differ-entiate the knowledge extraction and Segmentation process,which may be problematic since the Segmentation modelrepresentation is mixed with the Semantic features of thesupport.

5 We therefore propose to separate these two partsas prototype extraction and non-parametric metric prototypes are optimized to be compact and robust rep-resentations for each Semantic class and the non-parametricmetric learning performs Segmentation through pixel-levelmatching within the embedding space. Moreover, insteadof using the annotations of the support only for masking asin previous methods, we propose to leverage them also for9197supervising the Few-Shot learning process. To this end, weintroduce a novel prototype alignment regularization by per-forming the Few-Shot Segmentation in a reverse , the query Image together with its predicted maskis considered as a new support set and used to segment theprevious support images. In this way, the model is encour-aged to generate more consistent prototypes between sup-port and query, offering better generalization , we develop a Prototype Alignment Net-work (PANet) to tackle Few-Shot Segmentation , as shownin Figure 1.

6 PANet first embeds different foreground ob-jects and background into different prototypes via a sharedfeature extractor. In this way, each learned prototype is rep-resentative for the corresponding class and meanwhile issufficiently distinguishable from other classes. Then, eachpixel of the query Image is labeled by referring to the class-specific prototypes nearest to its embedding find that even with only one support Image per class,PANet can provide satisfactory Segmentation results, out-performing the state-of-the-arts. Furthermore, it imposes aprototype alignment regularization by forming a new sup-port set with the query Image and its predicted mask andperforming Segmentation on the original support set. Wefind this indeed encourages the prototypes generated fromthe queries to align well with those of the supports.

7 Notethat the model is regularized only in training and the queryimages should be not confused with the testing structure design of the proposed PANet has severaladvantages. First, it introduces no extra learnable parame-ters and thus is less prone to over-fitting. Second, withinPANet, the prototype embedding and prediction are per-formed on the computed feature maps and therefore seg-mentation requires no extra passes through the network. Inaddition, as the regularization is only imposed in training,the computation cost for inference does not Few-Shot Segmentation model is a generic one. Anynetwork with a fully convolutional structure can be used asthe feature extractor. It also learns well from weaker an-notations, , bounding boxes or scribbles, as shown inexperiments.

8 To sum up, the contributions of this work are: We propose a simple yet effective PANet for few-shotsegmentation. The model exploits metric learning overprototypes, which differs from most existing worksthat adopt a parametric classification architecture. We propose a novel prototype alignment regularizationto fully exploit the support knowledge to improve thefew-shot learning. Our model can be directly applied to learning from afew examples with weak annotations. Our PANet achieves mIoU of and onPASCAL-5ifor 1-shot and 5-shot settings, outper-forming state-of-the-arts by a margin up to %.2. Related workSemantic segmentationSemantic Segmentation aims toclassify each pixel of an Image into a set of predefinedsemantic classes. Recent methods are mainly based ondeep convolutional neural networks [13, 10, 1, 29, 2].

9 For example, Longet al. [13] first adopted deep CNNsand proposed Fully Convolutional Network (FCN) whichgreatly improves Segmentation performance. Dilated con-volutions [27, 2] are widely used to increase the receptivefield without losing spatial resolution. In this work, we fol-low the structure of FCN to perform dense prediction andalso adopt dilated convolutions to enjoy a larger receptivefield. Compared to models trained with full supervision, ourmodel can generalize to new categories with only a handfulof annotated learningFew-shot learning targets at learningtransferable knowledge across different tasks with only afew examples. Many methods have been proposed, such asmethods based on metric learning [25, 23], learning the op-timization process [18, 6] and applying graph-based meth-ods [20, 12].

10 Vinyalset al. [25] encoded input into deepneural features and performed weighted nearest neighbormatching to classify unlabelled data. Snellet al. [23] pro-posed a Prototypical Network to represent each class withone feature vector (prototype). Sunget al. [24] used a sep-arate module to directly learn the relation between supportfeatures and query features. Our model follows the Proto-typical Network [23] and can be seen as an extension of itto dense prediction tasks, enjoying a simple design yet segmentationFew-shot Segmentation is re-ceiving increasing interest recently. Shabanet al. [21] firstproposed a model for Few-Shot Segmentation using a con-ditioning branch to generate a set of parameters fromthe support set, which is then used to tune the segmen-tation process of the query set.