Continuous Color Transfer - arXiv

1 1 Example-based Color Transfer with GaussianMixture ModelingChunzhi Gu, Xuequan Lu,and Chao ZhangAbstract Color Transfer , which plays a key role in imageediting, has attracted noticeable attention recently. It has re-mained a challenge to date due to various issues such as time-consuming manual adjustments and prior segmentation this paper, we propose to model Color Transfer under aprobability framework and cast it as a parameter estimationproblem. In particular, we relate the transferred image withthe example image under the Gaussian Mixture Model (GMM)and regard the transferred image Color as the GMM employ the Expectation-Maximization (EM) algorithm (E-step and M-step) for optimization.

2 To better preserve gradientinformation, we introduce a Laplacian based regularization termto the objective function at the M-step which is solved byderiving a gradient descent algorithm. Given the input of asource image and an example image, our method is able togenerate Continuous Color Transfer results with increasing EMiterations. Various experiments show that our approach generallyoutperforms other competitive Color Transfer methods, bothvisually and Terms Color Transfer , Gaussian mixture INTRODUCTIONEDITING an image by endowing it with the Color style ofanother image ( , Color Transfer ) is challenging, becausethe Color style can be significantly affected by various factorssuch as illumination or saturation.

3 Color Transfer , which aimsat transforming Color from an example image to a sourceimage such that two images appear consistent in Color and thetransferred result looks natural, is a fundamental problem inimage editing and Transfer techniques based on the source image only[4], [15] often require time-consuming manual adjustments. Toreduce the efforts involved in parameterization and subjectivejudgment, example ( , reference, exemplar or target) basedcolor Transfer methods have been introduced. The problemcan be reduced to interpreting the Color of the example imageover the source image. The majority of existing example-basedmethods [2], [12], [26], [27], [31] require image segmentationas a pre-process to allocate colors to different regions, whichis known as local Color Transfer .

4 Some researches [2], [26]improve the accuracy of segmentation to better capture colorvariations spatially. Nevertheless, local methods are prone tofailure for small regions with insufficient numbers of , global Color Transfer algorithms [7], [21] [24], [33] cansometimes generate Color that is inconsistent with the exampleManuscript received XX, xx; revised XX. (Corresponding author: C. Zhang.)C. Gu and C. Zhang* are with the School of Engineering, University of Fukui,Fukui, Japan (e-mails: Lu is with School of Information Techonolgy, Deakin University, Australia(e-mail: Source Image(b) Example Image(c)q=3(d)q=8(e)q=15(f)q=35 Fig.))

5 1: An example of Continuous intermediate results producedby our algorithm with (a) and (b) as inputs. (c) (f) are fourintermediate Color Transfer results at different , and tend to strike a balance between many kinds offeatures of an image, which leads to undesirable over-saturationor artifacts [12]. As a result, those Color Transfer methods havelimited applicability in the real this paper, we propose a novel global Color transferapproach, by relating the transferred image of the sourceimage with the example image under the Gaussian MixtureModel (GMM). It does not require any prior segmentationknowledge or results.

6 More specifically, we suppose that thecolor distribution of the example image follows a GMM, andtake the transferred image Color (pixels) as the GMM optimization is achieved via the Expectation-Maximization(EM) algorithm (E-step and M-step) [5]. To further retainthe gradient information of the source image, we introducea Laplacian regularization term to the objective function inthe M-step. This improves Color mapping by softly imposinggradient information of the source image to the transferredimage, thus alleviating artifacts ( , discontinuity in the colortransition). Also, we introduce a gradient descent algorithm tosolve the objective function in the M-step.

7 It is interesting thatthe proposed approach is able to generate a series ofcontinuouscolor Transfer images with the increasing optimization iterationsfor the EM procedure (Fig. 1). To our knowledge, most [ ] 6 Jan 20222methods can only output a single Color Transfer result given apair of images (source image and example image), while ourapproach enables the flexibility in generating different colortransfer results and offer multiple choices for validate the proposed method and demonstratethat our method generally produces better results than othercompetitive Color Transfer techniques. Themain contributionsof this paper are summarized as follows.

8 We propose a novel Color Transfer approach which enablescontinuous Color Transfer results given an example imageand a source image as input. We introduce a regularization term to better preservethe gradient information, and derive a gradient descentalgorithm to solve the objective function in the M-step. We conduct extensive experiments and demonstrate thatour method generally outperforms other RELATEDWORKIn this section, we first review previous example based colortransfer techniques. We then introduce the Gaussian mixturemodel in a few visual computing tasks, especially the colortransfer.

9 We finally review some deep learning based colortransfer Example based Color TransferTransferring the Color style of an image using an exampleimage has been extensively studied [7], [8], [12], [13], [20] [24], [26] [28], [30], [31], [33], [34]. It has a wide rangeof applications with which users can create a variety ofrendering effects by changing the example image. A pioneeringwork was introduced by Reinhard et al. [24], in which thecolor distributions of the source and example images basedon the global statistics ( , mean and standard deviation)in the uncorrelatedl space are matched with a lineartransformation.

10 However, it fails to Transfer these dominantstatistics when the example image requires non-linear colormapping. Several other approaches [21], [22], [33] have beenproposed to improve the results later. Xiao et al. [33] introducedthe covariance matrix to better explain the statistics, and freedthe restriction inl space. F. Pitie et al. [21] presented theMonge-Kantorovitch (MK) theory on linear transformation. Insome cases, simple linear transformation is limited in colormapping performance. F. Pitie et al. [22] designed a non-lineariterative Transfer to match pdfs (probability density functions)of two images, followed by a post-processing step to refinethe efforts have also been made to handle the Color transferproblem.