Semantic Image Synthesis With Spatially-Adaptive …

1 2337233823392340 LabelGround TruthCRN [8]pix2pixHD [45]OursFigure 5: Visual comparison of Semantic Image Synthesis results on the COCO-Stuff dataset. Our method successfullysynthesizes realistic details from Semantic TruthCRN [8]SIMS [40]pix2pixHD [45]OursFigure 6: Visual comparison of Semantic Image Synthesis results on the ADE20K outdoor and Cityscapes datasets. Ourmethod produces realistic images while respecting the spatial Semantic layout at the same [8] [40]N/AN/AN/AN/AN/ [45] 1: Our method outperforms current leading methods in Semantic segmentation scores (mean IoU and overall pixelaccuracy) and FID [17] on all the benchmark datasets. For mIoU and pixel accuracy, higher is better. For FID, lower is distance between the distributions of synthesized resultsand the distribution of real compare our method with three leading se-mantic Image Synthesis models: the pix2pixHD model [45],the cascaded refinement network model (CRN) [8], andthe semi-parametric Image Synthesis model (SIMS) [40].

2 Pix2pixHD is the current state-of-the-art GAN-based con-ditional Image Synthesis framework. CRN uses a deep net-work that repeatedly refines the output from low to high res-olution, while the SIMS takes a semi-parametric approachthat composites real segments from a training set and refinesthe boundaries. Both the CRN and SIMS are mainly trainedusing Image reconstruction loss. For a fair comparison, wetrain the CRN and pix2pixHD models using the implemen-tations provided by the authors. As synthesizing an imageusing SIMS requires many queries to the training dataset,it is computationally prohibitive for a large dataset such asCOCO-stuff and the full ADE20K. Therefore, we use theresult images provided by the authors whenever shown in Table1, our52341 Figure 7: Semantic Image Synthesis results on the Flickr Landscapes dataset. The images were generated from semanticlayout of photographs on outperforms the current state-of-the-art methods bya large margin in all the datasets.

3 For COCO-Stuff, ourmethod achieves a mIoU score of , which is about better than the previous leading method. Our FIDis also times better than the previous leading note that the SIMS model produces a lower FID scorebut has poor segmentation performances on the Cityscapesdataset. This is because the SIMS synthesizes an Image byfirst stitching Image patches from the training dataset. Asusing the real Image patches, the resulting Image distribu-tion can better match the distribution of real images. How-ever, because there is no guarantee that a perfect query ( ,a person in a particular pose) exists in the dataset, it tendsto copy objects with mismatched Figures5and6, we provide aqualitative comparison of the competing methods. We findthat our method produces results with much better visualquality and fewer artifacts, especially for diverse scenes inthe COCO-Stuff and ADE20K dataset. When the trainingdataset size is small, the SIMS model also renders imageswith good visual quality.

4 However, the depicted contentoften deviates from the input segmentation mask ( , theshape of the swimming pool in the second row of Figure6).In Figures7and8, we show more example results fromthe Flickr Landscape and COCO-Stuff datasets. The pro-posed method can generate diverse scenes with high imagefidelity. More results are included in the appendix of ourDatasetOurs 2: User preference study. The numbers indicate thepercentage of users who favor the results of the proposedmethod over the competing use Amazon Mechanical Turk(AMT) to compare the perceived visual fidelity of ourmethod against existing approaches. Specifically, we givethe AMT workers an input segmentation mask and twosynthesis outputs from different methods and ask them tochoose the output Image that looks more like a correspond-ing Image of the segmentation mask. The workers are givenunlimited time to make the selection. For each comparison,we randomly generate500questions for each dataset, andeach question is answered by5different workers.

5 For qual-ity control, only workers with a lifetime task approval rategreater than 98% can participate in our the evaluation results. We find that usersstrongly favor our results on all the datasets, especially onthe challenging COCO-Stuff and ADE20K datasets. For theCityscapes, even when all the competing methods achieve62342 Figure 8: Semantic Image Synthesis results on COCO-Stuff. Our method successfully generates realistic images in diversescenes ranging from animals to sports # w/ SPADE (Ours) decoder w/ w/ ++ w/ ++ w/ ++ pix2pixHD++ [45] 3: mIoU scores are boosted when SPADE lay-ers are used, for both the decoder architecture (Figure4)and encoder-decoder architecture of pix2pixHD++ (our im-proved baseline over pix2pixHD [45]). On the other hand,simply concatenating Semantic input at every layer fails todo so. Moreover, our compact model with smaller depth atall layers outperforms all Image fidelity, users still prefer our effectiveness of study the impor-tance of SPADE, we introduce a strong baseline calledpix2pixHD++, which combines all the techniques we finduseful for enhancing the performance of pix2pixHD exceptSPADE.

6 We also train models that receive segmentationmask input at all the intermediate layers via concatenation(pix2pixHD++ w/ Concat) in the channel direction. Finally,the model that combines the strong baseline with SPADEis denoted as pix2pixHD++ w/ SPADE. Additionally, wecompare models with different capacity by using a differentnumber of convolutional filters in the #params #params #params Batch 4: The SPADE generator works with different con-figurations. We change the input of the generator, the con-volutional kernel size acting on the segmentation map, thecapacity of the network, and the parameter-free normaliza-tion method. The settings used in the paper are shown in Table3the architectures with the pro-posed SPADE consistently outperforms its counterparts, inboth the decoder-style architecture described in Figure4and more traditional encoder-decoder architecture used inpix2pixHD. We also find that concatenating segmentationmasks at all intermediate layers, an intuitive alternative toSPADE to provide Semantic signal, does not achieve thesame performance as SPADE.

7 Furthermore, the decoder-style SPADE generator achieves better performance thanthe strong baselines even when using a smaller number 9: Our model attains multimodal Synthesis capability when trained with the Image encoder. During deployment,by using different random noise, our model synthesizes outputs with diverse appearances but all having the same semanticlayouts depicted in the input mask. For reference, the ground truth Image is shown inside the input segmentation of SPADE the per-formance of variations of our generator. First, we comparetwo types of the input to the generator: random noise ordownsampled segmentation maps. We find that both ren-der similar performance, and conclude that the modulationby SPADE alone provides sufficient signal about the inputmask. Second, we vary the type of parameter-free normal-ization layers before applying the modulation observe that SPADE works reliably across different nor-malization methods.

8 Next, we vary the convolutional kernelsize acting on the label map, and find that kernel size of1x1 hurts performance, likely because it prohibits utilizingthe context of the label. Lastly, we modify the capacity ofthe generator network by changing the number of convolu-tional filters. We present more variations and ablations inthearXiv versionfor more detailed Figure9, we show the mul-timodal Image Synthesis results on the Flickr Landscapedataset. For the same input segmentation mask, we sam-ple different noise inputs to achieve different outputs. Moreresults are included in manipulation and guided Image , we show an application where a user draws dif-ferent segmentation masks, and our model renders the cor-responding landscape images. Moreover, our model allowsusers to choose an external style Image to control the globalappearances of the output Image . We achieve it by replac-ing the input noise with the embedding vector of the styleimage computed by the Image ConclusionWe have proposed the Spatially-Adaptive normalization ,which utilizes the input Semantic layout while performingthe affine transformation in the normalization layers.

9 Theproposed normalization leads to the first Semantic imagesynthesis model that can produce photorealistic outputs fordiverse scenes including indoor, outdoor, landscape, andstreet scenes. We further demonstrate its application formulti-modal Synthesis and guided Image thank Alexei A. Efros and JanKautz for insightful advice. Taesung Park contributed tothe work during his internship at NVIDIA. His is sup-ported by a Samsung [1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gen-erative adversarial networks. InInternational Conference onMachine Learning (ICML), [2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer preprint , [3] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Gold-man. Patchmatch: A randomized correspondence algorithmfor structural Image editing. InACM SIGGRAPH, [4] D. Bau, Zhu, H. Strobelt, Z. Bolei, J. B. Tenenbaum,W. T. Freeman, and A. Torralba. Gan dissection: Visualizingand understanding generative adversarial networks.

10 InPro-ceedings of the International Conference on Learning Rep-resentations (ICLR), [5] A. Brock, J. Donahue, and K. Simonyan. Large scale gantraining for high fidelity natural Image Synthesis . InInter-national Conference on Learning Representations (ICLR), ,2[6] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing andstuff classes in context. InIEEE Conference on ComputerVision and Pattern Recognition (CVPR), ,4[7] Chen, G. Papandreou, I. Kokkinos, K. Murphy, andA. L. Yuille. Deeplab: Semantic Image segmentation withdeep convolutional nets, atrous convolution, and fully con-nected Transactions on Pattern Analysis and Ma-chine Intelligence (TPAMI), 40(4):834 848, [8] Q. Chen and V. Koltun. Photographic Image Synthesis withcascaded refinement networks. InIEEE International Con-ference on Computer Vision (ICCV), ,2,4,5[9] T. Chen, M. Lucic, N. Houlsby, and S. Gelly. On self mod-ulation for generative adversarial networks. InInternationalConference on Learning Representations, [10] M.