Deep Bilateral Learning for Real-Time Image Enhancement

1 Deep Bilateral Learning for Real-Time Image EnhancementMICHA L GHARBI,MIT CSAILJIAWEN CHEN,Google ResearchJONATHAN T. BARRON,Google ResearchSAMUEL W. HASINOFF,Google ResearchFR DO DURAND,MIT CSAIL / Inria, Universit C te d Azur12 megapixel 16-bit linear input(tone-mapped for visualization)tone-mapped with HDR+400 600 msprocessed with our algorithm61 ms, PSNR = dBFig. 1. Our novel neural network architecture can reproduce sophisticated Image enhancements with inference running in real time at full HD resolution onmobile devices. It can not only be used to dramatically accelerate reference implementations, but can also learn subjective effects from human is a critical challenge in mobile Image processing. Given a ref-erence imaging pipeline, or even human-adjusted pairs of images, we seekto reproduce the enhancements and enable Real-Time evaluation.

2 For this,we introduce a new neural network architecture inspired by Bilateral gridprocessing and local affine color transforms. Using pairs of input/output im-ages, we train a convolutional neural network to predict the coefficients of alocally-affine model in Bilateral space. Our architecture learns to make local,global, and content-dependent decisions to approximate the desired imagetransformation. At runtime, the neural network consumes a low-resolutionversion of the input Image , produces a set of affine transformations in bilat-eral space, upsamples those transformations in an edge-preserving fashionusing a newslicingnode, and then applies those upsampled transformationsto the full-resolution Image . Our algorithm processes high-resolution im-ages on a smartphone in milliseconds, provides a Real-Time viewfinder at1080p resolution, and matches the quality of state-of-the-art approximationtechniques on a large class of Image operators.

3 Unlike previous work, ourmodel is trained off-line from data and therefore does not require access tothe original operator at runtime. This allows our model to learn complex,scene-dependent transformations for which no reference implementation isavailable, such as the photographic edits of a human Concepts: Computing methodologies Computational pho-tography; Image processing;Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee.

4 Request permissions from 2017 Copyright held by the owner/author(s). Publication rights licensed to $ : Key Words and Phrases: Real-Time Image processing, deep learn-ing, data-driven methods, convolutional neural networksACM Reference format:Micha l Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff,and Fr do Durand. 2017. Deep Bilateral Learning for Real-Time Image Trans. , 4, Article 118 (July 2017), 12 : INTRODUCTIONThe high resolution of images and videos produced by contemporarycameras and mobile devices puts significant performance pressureon Image processing algorithms, requiring sophisticated code op-timization by skilled programmers. While systems contributionshave sought to facilitate the implementation of high-performanceexecutables, [Hegarty et ; Mullapudi et ; Ragan-Kelley et ], they require programmer expertise, their runtimecost still grows with the complexity of the pipeline, and they are onlyapplicable when source code is available for the filters.

5 Addition-ally, because Image Enhancement is subjective, it is often desirableto learn an Enhancement model directly from human adjustments, [Bychkovsky et ]. To this end, we present a machinelearning approach where the effect of a reference filter, pipeline, oreven subjective manual photo adjustment is learned by a deep net-work that can be evaluated quickly and with cost independent of thereference s complexity. We focus on photographic enhancementsthat do not spatially warp the Image or add new edges, [Aubryet al. 2014; Hasinoff et al. 2016].ACM Transactions on Graphics, Vol. 36, No. 4, Article 118. Publication date: July :2 Micha l Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff, and Fr do DurandWe share the motivation of prior work that seeks to accelerate black box Image processing operations, either by using a remoteserver, [Gharbi et ] or by processing a low-resolutionimage and then using the low-resolution output to approximate ahigh-resolution equivalent [Chen et ].

6 For some operations,these approaches can achieve large speedups but they suffer fromsignificant limitations: the underlying Image processing operationmust be somewhat scale-invariant (Figure 9), and must be fast toevaluate at low resolution. In addition, these techniques rely onthe availability of an explicit reference implementation, and there-fore cannot be used to learn an implicitly-defined operation from adatabase of human annotated input/output deep Learning architectures have been used for Image -to- Image transformations, [Isola et ; Liu et ; Longet ; Xu et ; Yan et ]. However, most priorwork incur a heavy computational cost that scales linearly withthe size of the input Image , usually because of the large number ofstacked convolutions and non-linearities that must be evaluated atfull resolution.

7 This general form allows for flexible models to belearned, but this expressivity comes at a price: such architectures areorders of magnitude too slow for Real-Time viewfinder applications,requiring seconds to process a1megapixel Image on the best desktopGPUs more than1000 slower than our proposed model (2msonGPU). Our speedup is enabled by specifically targeting photographictransformations, which are often well-approximated with linearoperations in Bilateral space [Chen et ], and accordinglylearning our model in this present a new network architecture that is capable of learn-ing a rich variety of photographic Image enhancements and canbe rapidly evaluated on high-resolution inputs. We achieve thisthrough three key strategies: 1) We perform most predictions in alow-resolution Bilateral grid [Chen et ], where each pixel sx,ycoordinates are augmented with a third dimension which is afunction of the pixel s color.

8 To do this, we introduce a new node fordeep Learning that performs a data-dependent lookup. This enablesthe so-called slicing operation, which reconstructs an output imageat full Image resolution from the 3D Bilateral grid by consideringeach pixel s input color in addition to itsx,ylocation. 2) We followprevious work which has observed that it is often simpler to pre-dict thetransformationfrom input to output rather than predictingthe output directly , [Chen et ; Gharbi et ; Shihet ]. This is why our architecture is designed to learn, asan intermediate representation, a local affine color transformationthat will be applied to the input through a new multiplicative ) While most of our Learning and inference is performed at lowresolution, the loss function used during training is evaluated atfull resolution, which causes the low-resolution transformations welearn to be directly optimized for their impact on together, these three strategies (slicing, affine color trans-form, and full-resolution loss) allow us to perform the bulk of ourprocessing at a low resolution (thereby saving substantial computecost) yet reproduce the high-frequency behavior of the demonstrate the expressiveness of our model on a bench-mark of 7 applications including.

9 Approximating published imagefilters [Aubry et ; Hasinoff et ], reverse-engineeringblack-box Photoshop actions, and Learning the retouching style ofphotographers [Bychkovsky et ] from a set of manually cor-rected photographs. Our technique produces output whose qualityis comparable to or better than previous work, while being morewidely applicable by not requiring some reference implementationof the Image operation being approximated, being end-to-end learn-able from input/output Image pairs, and running in Real-Time onmobile hardware. The forward pass of our network takes14mstoprocess a full screen resolution1920 1080image on a Google Pixelphone, thereby enabling Real-Time viewfinder effects RELATED WORKT hough Image Enhancement algorithms have been the focus of agreat deal of research, most sophisticated algorithms are too expen-sive to be evaluated quickly on mobile devices, which is where thevast majority of digital images are captured and processed.

10 Becauseof this, previous work has identified specific critical operations anddeveloped novel algorithms to accelerate them. For instance, Farb-man et al. [2011] introducedconvolution pyramidsto acceleratelinear translation-invariant filters. Similarly, many approaches havebeen proposed to accelerate Bilateral filtering, due to the ubiquity ofedge-aware Image processing [Adams et ; Chen et ;Paris and Durand 2006; Tomasi and Manduchi 1998].One way to accelerate an operator is to simply apply it at low res-olution and upsample the result. A na ve upsampling will generallylead to an unacceptably blurry output, but this issue can often beameliorated by using a more sophisticated upsampling techniquethat respects the edges of the original Image . Joint Bilateral upsam-pling [Kopf et ] does this by using a Bilateral filter on ahigh-resolution guidance map to produce a piecewise-smooth edge-aware upsampling.