Demystifying Neural Style Transfer

1 Demystifying Neural Style TransferYanghao Li Naiyan Wang Jiaying Liu Xiaodi Hou Institute of Computer Science and Technology, Peking University Style Transfer [Gatyset al., 2016]has re-cently demonstrated very exciting results whichcatches eyes in both academia and industry. De-spite the amazing results, the principle of neuralstyle Transfer , especially why the Gram matricescould represent Style remains unclear. In this pa-per, we propose a novel interpretation of neuralstyle Transfer by treating it as a domain adapta-tion problem. Specifically, we theoretically showthat matching the Gram matrices of feature mapsis equivalent to minimize the Maximum Mean Dis-crepancy (MMD) with the second order polynomialkernel.

2 Thus, we argue that the essence of neu-ral Style Transfer is to match the feature distribu-tions between the Style images and the generatedimages. To further support our standpoint, we ex-periment with several other distribution alignmentmethods, and achieve appealing results. We believethis novel interpretation connects these two impor-tant research fields, and could enlighten future IntroductionTransferring the Style from one image to another imageis an interesting yet difficult have beenmany efforts to develop efficient methods for automatic styletransfer[Hertzmannet al.]

3 , 2001; Efros and Freeman, 2001;Efros and Leung, 1999; Shihet al., 2014; Kwatraet al.,2005]. Recently, Gatyset a seminal work[Gatyset al., 2016]: It captures the Style of artistic images andtransfer it to other images using Convolutional Neural Net-works (CNN). This work formulated the problem as find-ing an image that matching both the content and Style statis-tics based on the Neural activations of each layer in CNN. Itachieved impressive results and several follow-up works im-proved upon this innovative approaches[Johnsonet al., 2016;Ulyanovet al., 2016; Ruderet al.

4 , 2016; Lediget al., 2016].Despite the fact that this work has drawn lots of attention, thefundamental element of Style representation: the Gram ma- trix in[Gatyset al., 2016]is not fully explained. The reason Corresponding authorwhy Gram matrix can represent artistic Style still remains this paper, we propose a novel interpretation of neu-ral Style Transfer by casting it as a special domain adapta-tion[Beijbom, 2012; Patelet al., 2015]problem. We theo-retically prove that matching the Gram matrices of the neuralactivations can be seen as minimizing a specific MaximumMean Discrepancy (MMD)[Grettonet al.

5 , 2012a]. This re-veals that Neural Style Transfer is intrinsically a process of dis-tribution alignment of the Neural activations between on this illuminating analysis, we also experiment withother distribution alignment methods, including MMD withdifferent kernels and a simplified moment matching methods achieve diverse but all reasonable Style trans-fer results. Specifically, a Transfer method by MMD with lin-ear kernel achieves comparable visual results yet with a lowercomplexity. Thus, the second order interaction in Gram ma- trix is not a must for Style Transfer . Our interpretation pro-vides a promising direction to design Style Transfer methodswith different visual results.

6 To summarize, our contributionsare shown as follows:1. First, we demonstrate that matching Gram matrices inneural Style Transfer [Gatyset al., 2016]can be reformu-lated as minimizing MMD with the second order poly-nomial Second, we extend the original Neural Style Transfer withdifferent distribution alignment methods based on ournovel Related WorkIn this section, we briefly review some closely related worksand the key concept MMD in our TransferStyle Transfer is an active topic in bothacademia and industry. Traditional methods mainly focus onthe non-parametric patch-based texture synthesis and Transfer ,which resamples pixels or patches from the original sourcetexture images[Hertzmannet al.]

7 , 2001; Efros and Freeman,2001; Efros and Leung, 1999; Lianget al., 2001]. Differentmethods were proposed to improve the quality of the patch-based synthesis and constrain the structure of the target im-age. For example, the image quilting algorithm based ondynamic programming was proposed to find optimal [ ] 1 Jul 2017boundaries in[Efros and Freeman, 2001]. A Markov RandomField (MRF) was exploited to preserve global texture struc-tures in[Frigoet al., 2016]. However, these non-parametricmethods suffer from a fundamental limitation that they onlyuse the low-level features of the images for , Neural Style Transfer [Gatyset al.

8 , 2016]hasdemonstrated remarkable results for image stylization. Itfully takes the advantage of the powerful representation ofDeep Convolutional Neural Networks (CNN). This methodused Gram matrices of the Neural activations from differentlayers of a CNN to represent the artistic Style of a it used an iterative optimization method to generate anew image from white noise by matching the Neural activa-tions with the content image and the Gram matrices with thestyle image. This novel technique attracts many follow-upworks for different aspects of improvements and speed up the iterative optimization process in[Gatyset al.

9 ,2016], Johnsonet al.[Johnsonet al., 2016]and Ulyanovetal.[Ulyanovet al., 2016]trained a feed-forward generativenetwork for fast Neural Style Transfer . To improve the trans-fer results in[Gatyset al., 2016], different complementaryschemes are proposed, including spatial constraints[Selimet al., 2016], semantic guidance[Champandard, 2016]andMarkov Random Field (MRF) prior[Li and Wand, 2016].There are also some extension works to apply Neural styletransfer to other applications. Ruderet al.[Ruderet al.,2016]incorporated temporal consistence terms by penaliz-ing deviations between frames for video Style Transfer .

10 Selimet al.[Selimet al., 2016]proposed novel spatial constraintsthrough gain map for portrait painting methods further improve over the original Neural styletransfer, they all ignore the fundamental question in neuralstyle Transfer :Why could the Gram matrices represent theartistic Style ?This vagueness of the understanding limits thefurther research on the Neural Style AdaptationDomain adaptation belongs to thearea of Transfer learning[Pan and Yang, 2010]. It aims totransfer the model that is learned on the source domain tothe unlabeled target domain. The key component of domainadaptation is to measure and minimize the difference betweensource and target distributions.