Foreground-Aware Relation Network for Geospatial Object ...

1 Foreground-Aware Relation Network for Geospatial Object Segmentation inHigh Spatial Resolution remote sensing ImageryZhuo Zheng Yanfei Zhong Junjue WangAilong MaWuhan University, Wuhan, China{zhengzhuo, zhongyanfei, kingdrone, Object segmentation, as a particular se-mantic segmentation task, always faces with larger-scalevariation, larger intra-class variance of background, andforeground-background imbalance in the high spatial res-olution (HSR) remote sensing imagery. However, generalsemantic segmentation methods mainly focus on scale vari-ation in the natural scene, with inadequate considerationof the other two problems that usually happen in the largearea earth observation scene. In this paper, we argue thatthe problems lie on the lack of foreground modeling andpropose a Foreground-Aware Relation Network (FarSeg) fromthe perspectives of Relation -based and optimization-basedforeground modeling, to alleviate the above two perspective of Relation , FarSeg enhances the discrim-ination of foreground features via foreground-correlatedcontexts associated by learning foreground-scene , from perspective of optimization, a Foreground-Aware optimization is proposed to focus on foreground ex-amples and hard examples of background during trainingfor a balanced optimization.}

2 The experimental results ob-tained using a large scale dataset suggest that the proposedmethod is superior to the state-of-the-art general semanticsegmentation methods and achieves a better trade-off be-tween speed and IntroductionHigh spatial resolution earth observation technique hasprovided a large number of high spatial resolution (HSR) remote sensing images that can finely describe variousgeospatial objects, such as ship, vehicle and airplane, extracting objects of interest from HSR re-mote sensing imagery is very helpful for urban manage- Corresponding author. This work was supported by NationalKey Research and Development Program of China under Grant , National Natural Science Foundation of China underGrant Nos. 41771385, 41801267, and the China Postdoctoral ScienceFoundation under Grant withFigure 1.

3 The main challenges of Object segmentation in the HSRremote sensing imagery. (1) larger-scale variation. (2) foreground-background imbalance. (3) intra-class variance of , planing and monitoring [39, 40, 25, 26]. Geospatialobject segmentation, as a significant role in Object extrac-tion, can provide semantic and location information for theobjects of interest, which belongs to a particular semanticsegmentation task with the goal to divide image pixels intotwo subsets of the foreground objects and the backgroundarea. And meanwhile, it needs to further assign a uniquesemantic label to each pixel in the foreground Object with natural scene, Geospatial Object segmen-tation is more challenging in the HSR remote sensing im-ages. There are three reasons at least:1) The Object always has larger-scale variation in theHSR remote sensing images [14, 42].

4 This causes the multi-scale problem, which makes it difficult to locate and recog-nize the ) The background is much more complex in the HSRremote sensing images [36, 13], which causes serious falsealarms due to larger intra-class ) The foreground ratio is much less than it in the nat-4096ural images,as Fig. 1 shows, which causes foreground-background imbalance natural images, the Object segmentation task is di-rectly seen as a semantic segmentation task in the computervision field, the performance of which is mainly limitedby the multi-scale problem. Therefore, current state-of-the-art general semantic segmentation methods focus on scale-aware [7] and multi-scale [5, 6, 8, 44] modeling. However,for the HSR remote sensing images, false alarms problemand foreground-background imbalance problem are ignoredin these general semantic segmentation methods.

5 We ar-gue that this is because these methods are lack of explicitmodeling for the foreground. This seriously limits the fur-ther improvement of Object segmentation in the HSR remotesensing this paper, a Foreground-Aware Relation Network (FarSeg) is proposed to tackle aforementioned two prob-lems by exploiting explicitly foreground modeling for morerobust Object segmentation in the HSR remote sensing im-agery. We explore two perspectives of explicitly fore-ground modeling: Relation -based and optimization-basedforeground modeling, and we further propose two mod-ules in the FarSeg: foreground-scene Relation module andforeground-aware optimization. The foreground-scene rela-tion module learns the symbiotic Relation between scene andforeground to associate foreground-correlated contexts toenhance the foreground features, thus reducing false Foreground-Aware optimization focus the model on theforeground by suppressing numerous easy examples in thebackground to alleviate the foreground-background imbal-ance main contributions of our study are summarized asfollows:1.

6 A Foreground-Aware Relation Network (FarSeg) is pro-posed for Geospatial Object segmentation in HSR re-mote sensing To inherit multi-scale context modeling and learngeospatial scene representation, FarSeg builds a fore-ground branch based on the feature pyramid Network (FPN) and a scene embedding branch upon a sharedbackbone Network , namely multi-branch To suppress false alarms, F-S Relation module lever-ages the symbiotic Relation between Geospatial sceneand Geospatial objects, to associate foreground-correlated contexts and enhance the discrimination offoreground features. And meanwhile, the backgroundwithout any contribution is suppressed by this symbi-otic Relation , thus suppressing false To alleviate foreground-background imbalance, F-Aoptimization is proposed to focus the Network on hardexamples progressively, thus down-weighting gradientcontribution of numerous easy examples in the back-ground, for the foreground-background balanced Related WorkGeneral Semantic SegmentationTraditional methodsfirst extract features for each pixel by the handcrafted fea-ture descriptor.

7 The further promotion of these traditionalmethods mainly depends on the improvement of hand-crafted feature descriptors. However, designing a featuredescriptor is time-consuming and the handcrafted feature isnot robust due to limitation of prior knowledge of the success of deep learning-based methods lies in solv-ing this problem by learning feature representation fromdata directly [17]. Convolutional neural Network (CNN),as structured feature representation framework in deeplearning, has been explored for semantic segmentation viapatch-wise classification [11, 17, 19, 18, 37]. However,patch-wise fashion limits the spatial context modeling andbrings redundant computation on overlapped areas betweenpatches. To solve this problem, fully convolutional Network (FCN) [33] was proposed, which directly outputs the pixel-wise prediction from the input with arbitrary size via thein- Network upsampling layer.

8 FCN was the first pixels-to-pixels semantic segmentation method and was further exploit spatial context for semantic segmen-tation, deeplab v1 [4] utilized atrous convolution to enlargereceptive field of the CNN for wider spatial context model-ing. And a dense conditional random field (CRF) was usedas a postprocess to smooth the learn multi-scale feature representation, atrous spa-tial pyramidal pooling (ASPP) [5] and pyramid poolingmodule (PPM) [48] were proposed. ASPP utilized multi-ple atrous convolutions with different atrous rate to extractfeatures with the different receptive field, while PPM gen-erated pyramidal feature maps via pyramid pooling [20].The image-level features and batch normalization were em-bedded into ASPP for further improvement of accuracy indeeplab v3 [6].

9 DenseASPP [44] further enhanced multi-scale feature representation via densely connected ASPPto make the multi-scale features covering larger and denserscale range. However, these methods failed to extract finedetails of the Object , such as the [38] and SegNet [1] utilized a new encoder-decoder Network architecture, which reused the shallowfeatures with high spatial resolution to enhance the deepfeatures with strong semantics on spatial detail. RefineNet[29] proposed a multi-path refinement Network to progres-sively recover the spatial detail of deep features for bet-ter accuracy and visual performance. Deeplab v3+ alsoadopted encoder-decoder framework to further improve4097scene embedding branch1/41/81/161/321/41/41/41/4*8up Relation heatmapsscene embedding vector P2P3P4P5C2C3C4C5u(, )iuv (a) Multi-branch Encoder(b) Foreground-Scene Relation (c) Light-weight Decoder*4up *2up 4up foreground branch(, ) ()i () 4() 3() 2() iviv (, )()1+iiwiuvve step.

10 1 hard example estimation()iPobj class inputstep. 2 dynamic weightingHWWR step. 3 back-propagationinputargmax ()iPobj class (d) Foreground-Aware OptimizationFigure 2. Overview of FarSeg. (a) Multi-branch Encoder for multi-scale Object segmentation. (b) Foreground-scene Relation module. (c)Light-weight decoder. (d) Foreground-Aware optimization. The yellow dots indicate the relative positions of hard example in the rawimage, probability map and estimation surface for a simple via a more powerful backbone Xception [10]and a light-weight decoder to recover the spatial resolutionof features with a small general semantic segmentation methods mainlyfocus on multi-scale context modeling, ignoring the specialissues in the HSR remote sensing imagery, such as falsealarms and foreground-background imbalance.