arXiv:1702.08502v3 [cs.CV] 1 Jun 2018

1 understanding Convolution for Semantic SegmentationPanqu Wang1, Pengfei Chen1, Ye Yuan2, Ding Liu3, Zehua Huang1, Xiaodi Hou1, Garrison Cottrell41 TuSimple,2 Carnegie Mellon University,3 University of Illinois Urbana-Champaign,4UC San advances in deep learning, especially deep con-volutional neural networks (CNNs), have led to significantimprovement over previous semantic segmentation we show how to improve pixel-wise semantic seg-mentation by manipulating convolution-related operationsthat are of both theoretical and practical value. First, wedesign dense upsampling convolution (DUC) to generatepixel-level prediction, which is able to capture and decodemore detailed information that is generally missing in bi-linear upsampling. Second, we propose a hybrid dilatedconvolution (HDC) framework in the encoding phase.

2 Thisframework 1) effectively enlarges the receptive fields (RF)of the network to aggregate global information; 2) allevi-ates what we call the gridding issue caused by the stan-dard dilated convolution operation. We evaluate our ap-proaches thoroughly on the Cityscapes dataset, and achievea state-of-art result of mIOU in the test set at thetime of submission. We also have achieved state-of-the-art overall on the KITTI road estimation benchmark andthe PASCAL VOC2012 segmentation task. Our source codecan be found IntroductionSemantic segmentation aims to assign a categorical labelto every pixel in an image, which plays an important rolein image understanding and self-driving systems. The re-cent success of deep convolutional neural network (CNN)models [17, 26, 13] has enabled remarkable progress inpixel-wise semantic segmentation tasks due to rich hier-archical features and an end-to-end trainable framework[21, 31, 29, 20, 18, 3].

3 Most state-of-the-art semantic seg-mentation systems have three key components:1) a fully-convolutional network (FCN), first introduced in [21], re-placing the last few fully connected layers by convolutionallayers to make efficient end-to-end learning and inferencethat can take arbitrary input size; 2) Conditional RandomFields (CRFs), to capture both local and long-range depen-dencies within an image to refine the prediction map; 3) di-lated convolution (or Atrous convolution), which is used toincrease the resolution of intermediate feature maps in orderto generate more accurate predictions while maintaining thesame computational the introduction of FCN in [21], improvements onfully-supervised semantic segmentation systems are gener-ally focused on two perspectives: First, applying deeperFCN models.

4 Significant gains in mean Intersection-over-Union (mIoU) scores on PASCAL VOC2012 dataset [8]were reported when the 16-layer VGG-16 model [26] wasreplaced by a 101-layer ResNet-101 [13] model [3]; us-ing 152 layer ResNet-152 model yields further improve-ments [28]. This trend is consistent with the performance ofthese models on ILSVRC [23] object classification tasks, asdeeper networks generally can model more complex repre-sentations and learn more discriminative features that betterdistinguish among categories. Second, making CRFs morepowerful. This includes applying fully connected pairwiseCRFs [16] as a post-processing step [3], integrating CRFsinto the network by approximating its mean-field inferencesteps [31, 20, 18] to enable end-to-end training, and incor-porating additional information into CRFs such as edges[15] and object detections [1].

5 We are pursuing further improvements on semantic seg-mentation from another perspective: theconvolutionalop-erations for both decoding (from intermediate feature mapto output label map) and encoding (from input image to fea-ture map) counterparts. In decoding, most state-of-the-artsemantic segmentation systems simply use bilinear upsam-pling (before the CRF stage) to get the output label map[18, 20, 3]. Bilinear upsampling is not learnable and maylose fine details. Inspired by work in image super-resolution[25], we propose a method calleddense upsampling convo- [ ] 1 Jun 2018lution (DUC), which is extremely easy to implement andcan achieve pixel-level accuracy: instead of trying to re-cover the full-resolution label map at once, we learn anarray of upscaling filters to upscale the downsized featuremaps into the final dense feature map of the desired naturally fits the FCN framework by enabling end-to-end training, and it increases the mIOU of pixel-level se-mantic segmentation on the Cityscapes dataset [5] signifi-cantly, especially on objects that are relatively the encoding part, dilated convolution recently be-came popular [3, 29, 28, 32], as it maintains the resolutionand receptive field of the network by in inserting holes inthe convolution kernels, thus eliminating the need for down-sampling (by max-pooling or strided convolution).

6 How-ever, an inherent problem exists in the current dilated con-volution framework, which we identify as gridding : aszeros are padded between two pixels in a convolutional ker-nel, the receptive field of this kernel only covers an areawith checkerboard patterns - only locations with non-zerovalues are sampled, losing some neighboring problem gets worse when the rate of dilation increases,generally in higher layers when the receptive field is large:the convolutional kernel is too sparse to cover any local in-formation, since the non-zero values are too far apart. Infor-mation that contributes to a fixed pixel always comes fromits predefined gridding pattern, thus losing a huge portionof information. Here we propose a simplehybrid dilationconvolution (HDC)framework as a first attempt to addressthis problem: instead of using the same rate of dilation forthe same spatial resolution, we use a range of dilation ratesand concatenate them serially the same way as blocks inResNet-101 [13].

7 We show that HDC helps the network toalleviate the gridding problem. Moreover, choosing properrates can effectively increases the receptive field size andimproves the accuracy for objects that are relatively design DUC and HDC to makeconvolutionopera-tions better serve the need of pixel-level semantic segmen-tation. The technical details are described in Section 3 be-low. Combined with post-processing by Conditional Ran-dom Fields (CRFs), we show that this approach achievesstate-of-the art performance on the Cityscapes pixel-levelsemantic labeling task, KITTI road estimation benchmark,and PASCAL VOC2012 segmentation Related WorkDecoding of Feature Representation:In the pixel-wisesemantic segmentation task, the output label map has thesame size as the input image.

8 Because of the operationof max-pooling or strided convolution in CNNs, the sizeof feature maps of the last few layers of the network areinevitably downsampled. Multiple approaches have beenproposed to decode accurate information from the down-sampled feature map to label maps. Bilinear interpolationis commonly used [18, 20, 3], as it is fast and memory-efficient. Another popular method is called deconvolu-tion, in which the unpooling operation, using stored poolingswitches from the pooling step, recovers the informationnecessary for feature visualization [30]. In [21], a singledeconvolutional layer is added in the decoding stage to pro-duce the prediction result using stacked feature maps fromintermediate layers.

9 In [7], multiple deconvolutional layersare applied to generate chairs, tables, or cars from severalattributes. Noh et al. [22] employ deconvolutional layersas mirrored version of convolutional layers by using storedpooled location in unpooling step. [22] show that coarse-to-fine object structures, which are crucial to recover fine-detailed information, can be reconstructed along the propa-gation of the deconvolutional layers. Fischer at al. [9] usea similar mirrored structure, but combine information frommultiple deconvolutional layers and perform upsampling tomake the final Convolution:Dilated Convolution (or Atrousconvolution) was originally developed inalgorithme`a trousfor wavelet decomposition [14]. The main idea of dilatedconvolution is to insert holes (zeros) between pixels inconvolutional kernels to increase image resolution, thus en-abling dense feature extraction in deep CNNs.

10 In the se-mantic segmentation framework, dilated convolution is alsoused to enlarge the field of convolutional kernels. Yu &Koltun [29] use serialized layers with increasing rates ofdilation to enable context aggregation, while [3] design an atrous spatial pyramid pooling (ASPP) scheme to capturemulti-scale objects and context information by placing mul-tiple dilated convolution layers in parallel. More recently,dilated convolution has been applied to a broader range oftasks, such as object detection [6], optical flow [24], andaudio generation [27].3. Our Dense Upsampling Convolution (DUC)Suppose an input image has heightH, widthW, andcolor channelsC, and the goal of pixel-level semantic seg-mentation is to generate a label map with sizeH Wwhereeach pixel is labeled with a category label.