Color Transfer between Images - TAU

1 One of the most common tasks in imageprocessing is to alter an image s this means removing a dominant and undesirablecolor cast, such as the yellow in photos taken underincandescent illumination. This article describes amethod for a more general form of Color correction thatborrows one image s Color characteristics from anoth-er. Figure 1 shows an example of this process, where weapplied the colors of a sunset photograph to a daytimecomputer graphics can imagine many methodsfor applying the colors of one imageto another. Our goal is to do so witha simple algorithm, and our corestrategy is to choose a suitable colorspace and then to apply simple oper-ations there. When a typical threechannel image is represented in anyof the most well-known colorspaces, there will be correlationsbetween the different channels val-ues. For example, in RGB space,most pixels will have large values forthe red and green channel if the bluechannel is large.

2 This implies that ifwe want to change the appearanceof a pixel s Color in a coherent way,we must modify all Color channelsin tandem. This complicates any Color modificationprocess. What we want is an orthogonal Color spacewithout correlations between the , Ruderman et al. developed a Color space,called l , which minimizes correlation between chan-nels for many natural space is based ondata-driven human perception research that assumesthe human visual system is ideally suited for processingnatural scenes. The authors discovered l Color spacein the context of understanding the human visual sys-tem, and to our knowledge, l space has never beenapplied otherwise or compared to other Color s little correlation between the axes in l space,which lets us apply different operations in different colorchannels with some confidence that undesirable cross-channel artifacts won t occur. Additionally, this colorspace is logarithmic, which means to a first approxima-tion that uniform changes in channel intensity tend to beequally Color spaceBecause our goal is to manipulate RGB Images , whichare often of unknown phosphor chromaticity, we firstshow a reasonable method of converting RGB signalsto Ruderman et al.

3 S perception-based Color space l .Because l is a transform of LMS cone space, we firstconvert the image to LMS space in two steps. The firstis a conversion from RGB to XYZ tristimulus values. Thisconversion depends on the phosphors of the monitorthat the image was originally intended to be displayedon. Because that information is rarely available, we usea device-independent conversion that maps white inthe chromaticity diagram (CIE xy) to white in RGBspace and vice versa. Because we define white in thechromaticity diagram as x= X/(X+ Y+ Z) = , y= Y/(X+ Y+ Z) = , we need a transform thatmaps X= Y= Z= 1 to R= G= B= 1. To achieve this,we modified the XYZitu601-1 (D65) standard conver-sion matrix to have rows that add up to 1. The Interna- tional Telecommunications Union standard matrix is(1)By letting Mitux= (111)Tand solving for x, we obtain avector xthat we can use to multiply the columns ofmatrix Mitu, yielding the desired RGB to XYZ conversion:(2)Compared with Mitu,this normalization procedure con- XYZRGB = 0 5141 0 3239 0 16040 2651 0 6702 0 06410 0241 0 1228 0 Mitu= 0 4306 0 3415 0 17840 2220 0 7067 0 07130 0202 0 1295 0 $ 2001 IEEEA pplied Perception34 September/October 2001We use a simple statisticalanalysis to impose oneimage s Color characteristicson another.

4 We can achievecolor correction by choosingan appropriate source imageand apply its characteristicto another Reinhard, Michael Ashikhmin, Bruce Gooch,and Peter Shirley University of UtahColor Transferbetween Imagesstitutes a small adjustment to the matrix s in device-independent XYZ space, we can con-vert the image to LMS space using the followingconversion:4(3)Combining these two matrices gives the following trans-formation between RGB and LMS cone space:(4)The data in this Color space shows a great deal of skew,which we can largely eliminate by converting the datato logarithmic space:2L= log LM= log MS= log S(5)Using an ensemble of spectral Images that representsa good cross-section of naturally occurring Images , Rud-erman et al. proceed to decorrelate these axes. Theirmotivation was to better understand the human visualsystem, which they assumed would attempt to processinput signals similarly. We can compute maximal decor-relation between the three axes using principal compo-nents analysis (PCA), which effectively rotates three resulting orthogonal principal axes have sim-ple forms and are close to having integer to those nearby integer coefficients, Rudermanet al.

5 Suggest the following transform:(6)If we think of the Lchannel as red, the Mas green, andthe Sas blue, we can see that this is a variant of manyopponent- Color models:4 Achromatic r + g + bYellow blue r + g bRed green r g(7)Thus the laxis represents an achromatic channel,while the and channels are chromatic yellow blueand red green opponent channels. The data in thisspace are symmetrical and compact, while we achievedecorrelation to higher than second order for the set ofnatural Images et thiscolor space earlier because, in this Color space, theachromatic axis is orthogonal to the equiluminantplane. Our Color -correction method operates in this l space because decorrelation lets us treat the three colorchannels separately, simplifying the Color processing, which we explain in the nextsection, we must Transfer the result back to RGB to dis-play it. For convenience, here are the inverse opera -tions. We convert from l to LMS using this matrixmultiplication:(8) LMS = 11 111 1120330006600022l l = 13000160001211 111 2110 LMS LMSRGB = 0 3811 0 5783 0 04020 1967 0 7244 0 07820 0241 0 1288 0 LMSXYZ = 0 3897 0 6890 0 07870 2298 1 1834 0 04640 0000 0 0000 1.

6 IEEE Computer Graphics and Applications35(a)(b)(c)1 Applying asunset to anocean view. (a) Renderedimage1(imagecourtesy ofSimon Pre-moze), (b) photograph(image courtesyof the NationalOceanic andAtmosphericAdministrationPhoto Library),and (c) , after raising the pixel values to the power ten togo back to linear space, we can convert the data fromLMS to RGB using(9)Statistics and Color correctionThe goal of our work is to make a synthetic image takeon another image s look and feel. More formally thismeans that we would like some aspects of the distribu-tion of data points in l space to Transfer betweenimages. For our purposes, the mean and standard devi-ations along each of the three axes suffice. Thus, wecompute these measures for both the source and targetimages. Note that we compute the means and standarddeviations for each axis separately in l , we subtract the mean from the data points:(10)Then, we scale the data points comprising the syn-thetic image by factors determined by the respectivestandard deviations:(11)After this transformation, the resulting data points havestandard deviations that conform to the , instead of adding the averages that we previous-ly subtracted, we add the averages computed for thephotograph.

7 Finally, we convert the result back to RGBvia log LMS, LMS, and XYZ Color spaces using Equations8 and we assume that we want to Transfer oneimage s appearance to another, it s possible to selectsource and target Images that don t work well result s quality depends on the Images similarity incomposition. For example, if the synthetic image con-tains much grass and the photograph has more sky in it,then we can expect the Transfer of statistics to can easily remedy this issue. First, in this exam-ple, we can select separate swatches of grass and skyand compute their statistics, leading to two pairs of clus-ters in l space (one pair for the grass and sky swatch-es). Then, we convert the whole rendering to l scale and shift each pixel in the input image accord-ing to the statistics associated with each of the clusterpairs. Then, we compute the distance to the center ofeach of the source clusters and divide it by the cluster sstandard deviation c,s.

8 This division is required to com- = = =lltlsltsts ** lll**= = = RGBLMS = 4 4679 3 5873 0 11931 2186 2 3809 0 16240 0497 0 2439 1 ..Applied Perception36 September/October 20012 Color correc-tion in differentcolor top tobottom, theoriginal sourceand targetimages, fol-lowed by thecorrectedimages usingRGB, l , andCIECAM97scolor spaces.(Source imagecourtesy ofOliver Deussen.)pensate for different cluster sizes. We blend the scaledand shifted pixels with weights inversely proportionalto these normalized distances, yielding the final approach naturally extends to Images with morethan two clusters. We can devise other metrics to weightrelative contributions of each cluster, but weightingbased on scaled inverse distances is simple and workedreasonably well in our possible extension would be to computehigher moments such as skew and kurtosis, which arerespective measures of the lopsidedness of a distribu-tion and of the thickness of a distribution s tails.

9 Impos-ing such higher moments on a second image wouldshape its distribution of pixel values along each axis tomore closely resemble the corresponding distributionin the first image. While it appears that the mean andstandard deviation alone suffice to produce practicalresults, the effect of including higher moments remainsan interesting 1 and 2 showcase the main reason for devel-oping this technique. The synthetic image and the pho-tograph have similar compositions, so we can transferthe photograph s appearance to the synthetic 3 and 4 give other examples where we show thatfairly dramatic transfers are possible and still producebelievable results. Figure 4 also demonstrates the effec-tiveness of using small swatches to account for the dis-similarity in image 5 (next page) shows that nudging some of thestatistics in the right direction can sometimes make theresult more visually appealing. Directly applying themethod resulted in a corrected image with too much redin it.

10 Reducing the standard deviation in the red greenchannel by a factor of 10 produced the image in Figure5c, which more closely resembles the old photograph Computer Graphics and Applications373 Differenttimes of day. (a) Renderedimage6(imagecourtesy ofSimon Pre-moze), (b) photograph(courtesy of theNOAA PhotoLibrary), and (c) correctedrendering.(a)(b)(c)4 Using swatches. We applied (a) the atmosphere of Vincent van Gogh s Cafe Terrace on the Place du Forum, Arles, atNight(Arles, September 1888, oil on canvas; image from the Vincent van Gogh Gallery, ) to (b) a photograph of Lednice Castle near Brno in the Czech Republic. (c) We matched the blues of the sky inboth Images , the yellows of the cafe and the castle, and the browns of the tables at the cafe and the people at thecastle separately.(a)(b)(c)We show that we can use the l Color space for huecorrection using the gray world assumption. The ideabehind this is that if we multiply the three RGB chan-nels by unknown constants cr, cg, and cb, we won tchange the l channels variances, but will change theirmeans.