arXiv:2112.11641v1 [cs.CV] 22 Dec 2021

1 JoJoGAN: One Shot Face StylizationMin Jin Chong and David ForsythUniversity of Illinois at Urbana-Champaign{mchong6, 1: We perform arbitrary one-shot face stylization without any paired data. Only one singlereference image is needed for training which takes about 1 minute. After training, style can be appliedto any input image. We can selectively choose to preserve the original colors of the input images ortransfer the colors from the reference there have been recent advances in few-shot image stylization, these methodsfail to capture stylistic details that are obvious to humans. Details such as the shapeof the eyes, the boldness of the lines, are especially difficult for a model to learn,especially so under a limited data setting. In this work, we aim to perform one-shot image stylization that gets the details right.}

2 Given a reference style image,we approximate paired real data using GAN inversion and finetune a pretrainedStyleGAN using that approximate paired data. We then encourage the StyleGANto generalize so that the learned style can be applied to all other [ ] 22 Dec 20211 IntroductionStop spending your time collecting data! We present JoJoGAN (named after the best anime JoJo sBizarre Adventure, the inspiration for this work), a framework for arbitrary one-shot face single input style reference, we are able to apply the style onto any input image, carefullypreserving detailed style characteristics such as eye appearences, proportions, only a single reference style image, a skilled artist can reproduce new artworks that faithfullycapture the style. This is however, something that remains difficult with our current machine learningframeworks.

3 The best way to train an image translation framework is with paired training data. Thisis however not realistic under real world setting. Oftentimes we are not able to get a large amountof training data, let alone paired data. Thus, much emphasis is placed on unpaired image to imagetranslation that significantly reduces this constraint. Still, large amounts of data is needed, rangingfrom hundreds to thousands. While there have been some work on few shot translation [1,2,3], theresults fail to capture distinct style details, diversity, or lack image aims to solve this problem by first approximating a paired training dataset and thenfinetuning a StyleGAN to perform one-shot face stylization. We show that our method pays closeattention to style details with zero supervision and generalizes well across various different and demo code is available MethodologyGAN invert referenceForm style code and reference training pairsFinetune StyleGANto generate the reference from the style codeGenerate new images using finetuned StyleGANF igure 2:WorkflowJoJoGAN works by finetuning a pretrained StyleGAN2 [4] with a single reference style image.

4 Thereare several steps in the pipeline, prepare approximate paired training data by GAN inverting the reference style imagey,giving us style codewthat generates a plausible corresponding real face then find a family ofWthat generates a family of real face imagesXthat should matchreference style imagey. Form pairs of(wi,y)that serves as our paired training Finetune the StyleGAN based on those paired training Generate new samples using the finetuned Data PreparationTraining with paired data is optimal for the image stylization task. However, they are oftentimesvery difficult to come by, requiring significant time and money investment. No good open-sourcepaired dataset for our task is currently available. We aim to overcome this problem by generatingan approximate paired training dataset as shown in Figure 3.

5 Given a style reference imagey, weperform GAN inversion using e4e [5] to obtainw. As e4e is trained on real face images, it fails togeneralize to our out-of-distribution style image and thus giving us awthat approximates the real face image ofy, forming a paired(w,y)training using only a single datapoint leads to poor generalization to other images, see Figure 4. Weovercome this by generating more training datapoints. The idea is simple, many real face imagesshould match to the same style reference image. For example, faces with slightly different eye sizesor hair texture can reasonably be matched to the same reference image. It is trivial to generate thesesimilar samples in StyleGAN by perform style mixing at certain chosen layers. For a1024resolutionStyleGAN2 with18style modulation layers, our style code isw R18 512.

6 We define a maskM {0,1}18which selectively masks out parts of our style code that we want to style mix for dataaugmentation,FCas the style mapping layer of the StyleGAN,zi N(0,I)as our random noise2 GAN InvertRandom Style MixingColor Preserving yStyle Reference wStyleGAN StyleGANxFigure 3:Data Preparation:We perform GAN inversion on the style reference imageyto obtaina style codewfor the corresponding real face image. We can then perform random style mixingto obtain a family of these style codesWwhose face imagesXmap toy, forming(wi,y)pairedtraining data. AmongXis a subset of color preserving face imagesCthat have similar color profileasy. UsingXorCfor our training data leads to different results, allowing us to control if our facestylization preserves the original color of the input with xWith With Figure 4:Using different datasets:We compare finetuning StyleGAN using different finetuning with only a single datapointx, our model is unable to fully capture the styledetails of the reference image and generalize to other images.

7 For the datasetCwhich correspondsto face images with similar color profile to the reference image (refer to Figure 3), our finetunedmodel accurately captures style characteristics while keeping the originals colors of the input withXinstead copies the colors from the reference image, fully reproducing the , and U(0,1)a random scalar value controlling strength of style mixing. Our new stylecodewiis thus,wi= (1 )M w+ (1 M) FC(zi)(1)Our choice ofMdetermines the layers we opt to do style mixing. This choice of layers allows us togenerate a family of similar face imagesXand control the color profile of them. For example, wegenerate datasetXin Figure 3 by style mixing layers7to18, preserving pose and hairstyle, whilevarying colors, eye shapes, etc. By style mixing7to9, we can preserve the colors of the style image,giving us FinetuningAfter generating paired data, we can finetune a pretrained StyleGANG.

8 Let the training pairs be(wi,y)wherewiare the style codes we obtain previously, andyis the reference style image. We thendefine our loss as3 ReferencesInputs(a) JoJoGAN(b) StyleGAN-NADA(c) BlendGAN(d) JoJoGAN(e) StyleGAN-NADA(f) BlendGANF igure 5:Comparisons:We compare our non-color preserving model with StyleGAN-NADA [8]and BlendGAN [2] on one shot face stylization. Our method captures details such as the hairaccessories in (a) and the face paintings in (d). We also maintain the identity of the input image well,keeping expressions and hairstyles consistent. On the contrary, StyleGAN-NADA changes the facialidentities and fails to capture the complex face paintings. BlendGAN fails to capture meaningfulstyle iLPIPS(G(wi),y)(2)using LPIPS [6] as our perceptual loss. Our different choices of training datasets lead to differentresults as seen in Figure 4.

9 Unsurprisingly, training with only a single examplexleads to poorresults that fails to capture the proper style. As the training datasetChas the same color as thereference image, our finetuned StyleGAN learns to not drastically change the color of the imagesduring finetuning process. This results in our finetuned model preserving the colors of the inputimage, not affected strongly by the style reference. Training withXinstead fully captures the originalstyle, including the GenerationOur fintuned StyleGAN now generates images with the reference style. To perform image translation,we simply perform GAN inversion on the input image and perform inference on our Multi-shotNote that our method can easily extend to multi-shot stylization by simply minimizing Equation (2) style codes corresponding to more style SetupWe finetune JoJoGAN for500iterations with Adam optimizer [7] at a learning rate of2 10 on an Nvidia A40 takes about 1 minute.

10 We compare non-color preserving JoJoGANwith state-of-the-art one/few shot stylization methods StyleGAN-NADA [8] and BlendGAN [2].Compared to them, JoJoGAN can capture small details that define the style while maintaining clearfacial identity from the inputs. In Figure 5(a), our method captures the eye shapes and details4perfectly and also the hair accessories from the style reference, while in Figure 5(d), we capture thecomplex face paintings accurately. In contrast, while StyleGAN-NADA captures the overall Jokermakeup in Figure 5(e), it fails to capture the details such as the eyes and eyebrows paint. Identitiesare also majorly affected. BlendGAN fails to capture meaningful style details, with even hairstylesgetting the wrong Future WorkWhile JoJoGAN allows simple one-shot face stylization, it is not practical to perform video inferencedue to the GAN inversion step.