Globally and Locally Consistent Image Completion

1 Globally and Locally Consistent Image CompletionSATOSHI IIZUKA,Waseda UniversityEDGAR SIMO-SERRA,Waseda UniversityHIROSHI ISHIKAWA,Waseda UniversityFig. 1. Image Completion results by our approach. The masked area is shown in white. Our approach can generate novel fragments that are not presentelsewhere in the Image , such as needed for completing faces; this is not possible with patch-based methods. Photographs courtesy of Michael D Beckwith(CC0), Mon Mer (Public Domain), davidgsteadman (Public Domain), and Owen Lucas (Public Domain).We present a novel approach for Image Completion that results in imagesthat are both Locally and Globally Consistent . With a fully-convolutionalneural network, we can complete images of arbitrary resolutions by lling-in missing regions of any shape. To train this Image Completion network tobe Consistent , we use global and local context discriminators that are trainedto distinguish real images from completed ones.

2 The global discriminatorlooks at the entire Image to assess if it is coherent as a whole, while the localdiscriminator looks only at a small area centered at the completed region toensure the local consistency of the generated patches. The Image completionnetwork is then trained to fool the both context discriminator networks,which requires it to generate images that are indistinguishable from real oneswith regard to overall consistency as well as in details. We show that ourapproach can be used to complete a wide variety of scenes. Furthermore, incontrast with the patch-based approaches such as PatchMatch, our approachcan generate fragments that do not appear elsewhere in the Image , whichallows us to naturally complete the images of objects with familiar andhighly speci c structures, such as Concepts: Computing methodologies Image processing;Neu-ral networks;This work was partially supported by JST ACT-I Grant Number JPMJPR16U3 and JSTCREST Grant Number to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro t or commercial advantage and that copies bear this notice and the full citationon the rst page.

3 Copyrights for third-party components of this work must be all other uses, contact the owner/author(s). 2017 Copyright held by the owner/author(s). 0730-0301/2017/7-ART107 $ : Key Words and Phrases: Image Completion , convolutional neuralnetworkACM Reference format:Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally andLocally Consistent Image Trans. , 4, Article 107(July 2017), 14 : INTRODUCTIONI mage Completion is a technique that allows lling-in target regionswith alternative contents. This allows removing unwanted objectsor generating occluded regions for Image -based 3D many approaches have been proposed for Image comple-tion, such as patch-based Image synthesis [Barnes et ; Darabiet ; Huang et ; Simakov et ; Wexler et ],it remains a challenging problem because it often requires high-levelrecognition of scenes. Not only is it necessary to complete texturedpatterns, it is also important to understand the anatomy of the sceneand objects being completed.

4 Based on this observation, in this workwe consider both the local continuity and the global composition ofthe scene, in a single framework for Image work builds upon the recently proposed Context Encoder(CE) approach [Pathak et ], which employs a ConvolutionalACM Transactions on Graphics, Vol. 36, No. 4, Article 107. Publication date: July :2 Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi IshikawaNeural Network (CNN) that is trained with an adversarial loss [Good-fellow et ]. The CE approach was motivated by feature learn-ing, and did not fully describe how to handle arbitrary inpaintingmasks nor how to apply the approach to high resolution images. Ourproposed approach addresses these two points and further improvesthe visual quality of the results as we shall leverage a fully convolutional network as the basis of ourapproach, and propose a novel architecture that results in bothlocally and Globally Consistent natural Image Completion .

5 Our ar-chitecture is composed of three networks: a Completion network, aglobal context discriminator, and a local context discriminator. Thecompletion network is fully convolutional and used to completethe Image , while both the global and the local context discrimina-tors are auxiliary networks used exclusively for training. Thesediscriminators are used to determine whether or not an Image hasbeen completed consistently. The global discriminator takes thefull Image as input to recognize global consistency of the scene,while the local discriminator looks only at a small region aroundthe completed area in order to judge the quality of more detailedappearance. During each training iteration, the discriminators areupdated rst so that they correctly distinguish between real andcompleted training images. Afterwards, the Completion networkis updated so that it lls the missing area well enough to fool thecontext discriminator networks.

6 As shown in Fig. 1, using both thelocal and the global context discriminators is critical for obtainingrealistic Image evaluate and compare our approach with existing methods ona large variety of scenes. We also show results on more challengingspeci c tasks, such as face Completion , in which our approach cangenerate Image fragments of objects such as eyes, noses, or mouthsto realistically complete the faces. We evaluate the naturalness of thischallenging face Completion with a user study, where the di erencebetween our results and real faces is indiscernible 77% of the summary, in this paper we present: a high performance network model that can complete arbitrarymissing regions, a Globally and Locally Consistent adversarial training approachfor Image Completion , and results of applying our approach to speci c datasets for morechallenging Image RELATED WORKA variety of di erent approaches have been proposed for the imagecompletion task.

7 One of the more traditional approaches is thatof di usion-based Image synthesis. This technique propagates thelocal Image appearance around the target holes to ll them in. Forexample, the propagation can be performed based on the isophotedirection eld [Ballester et ; Bertalmio et ], or globalimage statistics based on the histograms of local features [Levinet ]. However, di usion-based approaches, in general, canonly ll small or narrow holes, such as scratches found commonlyin old contrast to the di usion-based techniques, patch-based ap-proaches have been able to perform more complicated Image com-pletion that can ll large holes in natural images. Patch-based imageTable 1. Comparison of di erent approaches for Completion . Patch-basedapproaches such as [Barnes et al. 2009] cannot generate new texture orobjects and only look at local similarity without taking into account thesemantics of the scene.

8 The context encoder [Pathak et al. 2016] handlesonly images of small fixed size without maintaining local consistency withthe surrounding region. In contrast, our method can complete images of anysize, generating new texture and objects according to the local and globalstructures of the Context encoder OursImage sizeAnyFixedAnyLocal ConsistencyYesNoYesSemanticsNoYesYesNove l objectsNoYesYescompletion was rst proposed for texture synthesis [Efros and Le-ung 1999; Efros and Freeman 2001], in which texture patches aresampled from a source Image and then pasted into a target was later extended with Image stitching [Kwatra et ]with graph cuts and texture generation [Kwatra et ] based onenergy optimization. For Image Completion , several modi cationssuch as optimal patch search have been proposed [Bertalmio et ; Criminisi et ; Drori et ]. In particular, Wexleretal. [2007] and Simakovet al. [2008] proposed a global-optimization-based method that can obtain more Consistent lls.

9 These techniqueswere later accelerated by a randomized patch search algorithm calledPatchMatch [Barnes et , 2010], which allows for real-timehigh-level Image editing of images. Darabiet al. [2012] demonstratedimproved Image Completion by integrating Image gradients into thedistance metric between patches. However, these methods dependon low-level features such as the sum of squared di erences of patchpixel values, which are not e ective to ll in holes on complicatedstructures. Furthermore, they are unable to generate novel objectsnot found in the source Image , unlike our tackle the problem of generating large missing regions of struc-tured scenes, there are some approaches that use structure guidance,which are generally speci ed manually, to preserve important un-derlying structures. This can be done by specifying points of inter-est [Drori et ], lines or curves [Barnes et ; Sun et ], and perspective distortion [Pavi et ].

10 Approaches forautomatic estimation of the scene structure have also been proposed:utilizing the tensor-voting algorithm to smoothly connect curvesacross holes [Jia and Tang 2003]; exploiting structure-based priorityfor patch ordering [Criminisi et ], tile-based search spaceconstraints [Kopf et ], statistics of patch o sets [He and Sun2012], and regularity in perspective planar surfaces [Huang et ]. These approaches improve the quality of the Image comple-tion by preserving important structures. However, such guidancesare based on the heuristic constraints of speci c types of scenes andthus are limited to speci c obvious limitation of most existing patch-based approaches isthat the synthesized texture only comes from the input Image . Thisis a problem when a convincing Completion requires textures thatare not found in the input Image . Hays and Efros [2007] proposedan Image Completion method using a large database of images.