Arbitrary Style Transfer in Real-Time With Adaptive ...

1 Arbitrary Style Transfer in Real-Time with Adaptive Instance NormalizationXun HuangSerge BelongieDepartment of Computer Science & Cornell Tech, Cornell et al. recently introduced a neural algorithm thatrenders a content image in the Style of another image,achieving so-called Style Transfer . However, their frame-work requires a slow iterative optimization process, whichlimits its practical application. Fast approximations withfeed-forward neural networks have been proposed to speedup neural Style Transfer . Unfortunately, the speed improve-ment comes at a cost: the network is usually tied to a fixedset of styles and cannot adapt to Arbitrary new styles.

2 In thispaper, we present a simple yet effective approach that for thefirst time enables Arbitrary Style Transfer in Real-Time . At theheart of our method is a novel Adaptive instance normaliza-tion (AdaIN) layer that aligns the mean and variance of thecontent features with those of the Style features. Our methodachieves speed comparable to the fastest existing approach,without the restriction to a pre-defined set of styles. In ad-dition, our approach allows flexible user controls such ascontent- Style trade-off, Style interpolation, color & spatialcontrols, all using a single feed-forward neural IntroductionThe seminal work of Gatyset al.

3 [16] showed that deepneural networks (DNNs) encode not only the content butalso thestyleinformation of an image. Moreover, the im-age Style and content are somewhat separable: it is possibleto change the Style of an image while preserving its con-tent. The Style Transfer method of [16] is flexible enough tocombine content and Style of Arbitrary images. However, itrelies on an optimization process that is prohibitively effort has been devoted to accelerating neuralstyle Transfer . [24,51,31] attempted to train feed-forwardneural networks that perform stylization with a single for-ward pass. A major limitation of most feed-forward meth-ods is that each network is restricted to a single Style .

4 Thereare some recent works addressing this problem, but they areeither still limited to a finite set of styles [11,32,55,5], ormuch slower than the single- Style Transfer methods [6].In this work, we present the first neural Style transferalgorithm that resolves this fundamental flexibility-speeddilemma. Our approach can Transfer Arbitrary new stylesin Real-Time , combining the flexibility of the optimization-based framework [16] and the speed similar to the fastestfeed-forward approaches [24,52]. Our method is inspiredby theinstance normalization(IN) [52,11] layer, whichis surprisingly effective in feed-forward Style Transfer . Toexplain the success of instance normalization, we proposea new interpretation that instance normalization performsstyle normalization by normalizing feature statistics, whichhave been found to carry the Style information of an im-age [16,30,33].

5 Motivated by our interpretation, we in-troduce a simple extension to IN, namelyadaptive instancenormalization(AdaIN). Given a content input and a styleinput, AdaIN simply adjusts the mean and variance of thecontent input to match those of the Style input. Throughexperiments, we find AdaIN effectively combines the con-tent of the former and the Style latter by transferring featurestatistics. A decoder network is then learned to generate thefinal stylized image by inverting the AdaIN output back tothe image space. Our method is nearly three orders of mag-nitude faster than [16], without sacrificing the flexibility oftransferring inputs to Arbitrary new styles.

6 Furthermore, ourapproach provides abundant user controls at runtime, with-out any modification to the training Related WorkStyle problem of Style Transfer has its originfrom non-photo-realistic rendering [28], and is closely re-lated to texture synthesis and Transfer [13,12,14]. Someearly approaches include histogram matching on linear fil-ter responses [19] and non-parametric sampling [12,15].These methods typically rely on low-level statistics and of-ten fail to capture semantic structures. Gatyset al. [16] forthe first time demonstrated impressive Style Transfer resultsby matching feature statistics in convolutional layers of aDNN.

7 Recently, several improvements to [16] have beenproposed. Li and Wand [30] introduced a framework basedon markov random field (MRF) in the deep feature space toenforce local patterns. Gatyset al. [17] proposed ways tocontrol the color preservation, the spatial location, and thescale of Style Transfer . Ruderet al. [45] improved the quality11501of video Style Transfer by imposing temporal framework of Gatyset al. [16] is based on a slowoptimization process that iteratively updates the image tominimize a content loss and a Style loss computed by a lossnetwork. It can take minutes to converge even with mod-ern GPUs. On-device processing in mobile applications istherefore too slow to be practical.

8 A common workaroundis to replace the optimization process with a feed-forwardneural network that is trained to minimize the same ob-jective [24,51,31]. These feed-forward Style Transfer ap-proaches are about three orders of magnitude faster thanthe optimization-based alternative, opening the door to Real-Time applications. Wanget al. [53] enhanced the granularityof feed-forward Style Transfer with a multi-resolution archi-tecture. Ulyanovet al. [52] proposed ways to improve thequality and diversity of the generated samples. However,the above feed-forward methods are limited in the sense thateach network is tied to a fixed Style . To address this prob-lem, Dumoulinet al.

9 [11] introduced a single network thatis able to encode32styles and their interpolations. Con-current to our work, Liet al. [32] proposed a feed-forwardarchitecture that can synthesize up to300textures and trans-fer16styles. Still, the two methods above cannot adapt toarbitrary styles that are not observed during recently, Chen and Schmidt [6] introduced a feed-forward method that can Transfer Arbitrary styles thanks toa Style swap layer. Given feature activations of the contentand Style images, the Style swap layer replaces the contentfeatures with the closest-matching Style features in a patch-by-patch manner. Nevertheless, their Style swap layer cre-ates a new computational bottleneck: more than95%of thecomputation is spent on the Style swap for512 512inputimages.

10 Our approach also permits Arbitrary Style Transfer ,while being1-2orders of magnitude faster than [6].Another central problem in Style Transfer is which styleloss function to use. The original framework of Gatysetal. [16] matches styles by matching the second-order statis-tics between feature activations, captured by the Gram ma-trix. Other effective loss functions have been proposed,such as MRF loss [30], adversarial loss [31], histogramloss [54], CORAL loss [41], MMD loss [33], and distancebetween channel-wise mean and variance [33]. Note that allthe above loss functions aim to match some feature statisticsbetween the Style image and the synthesized generative image are several al-ternative frameworks for image generation, including varia-tional auto-encoders [27], auto-regressive models [40], andgenerative adversarial networks (GANs) [18].