Deep Bilateral Learning for Real-Time Image Enhancement

1 Deep Bilateral Learning for Real-Time Image EnhancementMICHA L GHARBI,MIT CSAILJIAWEN CHEN,Google ResearchJONATHAN T. BARRON,Google ResearchSAMUEL W. HASINOFF,Google ResearchFR DO DURAND,MIT CSAIL / Inria, Universit C te d Azur12 megapixel 16-bit linear input(tone-mapped for visualization)tone-mapped with HDR+400 600 msprocessed with our algorithm61 ms, PSNR = dBFig. 1. Our novel neural network architecture can reproduce sophisticated Image enhancements with inference running in real time at full HD resolution onmobile devices.

2 It can not only be used to dramatically accelerate reference implementations, but can also learn subjective effects from human is a critical challenge in mobile Image processing. Given a ref-erence imaging pipeline, or even human-adjusted pairs of images, we seekto reproduce the enhancements and enable Real-Time evaluation. For this,we introduce a new neural network architecture inspired by Bilateral gridprocessing and local affine color transforms. Using pairs of input/output im-ages, we train a convolutional neural network to predict the coefficients of alocally-affine model in Bilateral space.

3 Our architecture learns to make local,global, and content-dependent decisions to approximate the desired imagetransformation. At runtime, the neural network consumes a low-resolutionversion of the input Image , produces a set of affine transformations in bilat-eral space, upsamples those transformations in an edge-preserving fashionusing a newslicingnode, and then applies those upsampled transformationsto the full-resolution Image . Our algorithm processes high-resolution im-ages on a smartphone in milliseconds, provides a Real-Time viewfinder at1080p resolution, and matches the quality of state-of-the-art approximationtechniques on a large class of Image operators.

4 Unlike previous work, ourmodel is trained off-line from data and therefore does not require access tothe original operator at runtime. This allows our model to learn complex,scene-dependent transformations for which no reference implementation isavailable, such as the photographic edits of a human Concepts: Computing methodologies Computational pho-tography; Image processing;Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page.

5 Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from 2017 Copyright held by the owner/author(s). Publication rights licensed to $ : Key Words and Phrases: Real-Time Image processing, deep learn-ing, data-driven methods, convolutional neural networksACM Reference format:Micha l Gharbi, Jiawen Chen, Jonathan T.

6 Barron, Samuel W. Hasinoff,and Fr do Durand. 2017. Deep Bilateral Learning for Real-Time Image Trans. , 4, Article 118 (July 2017), 12 : INTRODUCTIONThe high resolution of images and videos produced by contemporarycameras and mobile devices puts significant performance pressureon Image processing algorithms, requiring sophisticated code op-timization by skilled programmers. While systems contributionshave sought to facilitate the implementation of high-performanceexecutables, [Hegarty et ; Mullapudi et ; Ragan-Kelley et ], they require programmer expertise, their runtimecost still grows with the complexity of the pipeline, and they are onlyapplicable when source code is available for the filters.

7 Addition-ally, because Image Enhancement is subjective, it is often desirableto learn an Enhancement model directly from human adjustments, [Bychkovsky et ]. To this end, we present a machinelearning approach where the effect of a reference filter, pipeline, oreven subjective manual photo adjustment is learned by a deep net-work that can be evaluated quickly and with cost independent of thereference s complexity. We focus on photographic enhancementsthat do not spatially warp the Image or add new edges, [Aubryet al.]

8 2014; Hasinoff et al. 2016].ACM Transactions on Graphics, Vol. 36, No. 4, Article 118. Publication date: July :2 Micha l Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff, and Fr do DurandWe share the motivation of prior work that seeks to accelerate black box Image processing operations, either by using a remoteserver, [Gharbi et ] or by processing a low-resolutionimage and then using the low-resolution output to approximate ahigh-resolution equivalent [Chen et ]. For some operations,these approaches can achieve large speedups but they suffer fromsignificant limitations: the underlying Image processing operationmust be somewhat scale-invariant (Figure 9), and must be fast toevaluate at low resolution.

9 In addition, these techniques rely onthe availability of an explicit reference implementation, and there-fore cannot be used to learn an implicitly-defined operation from adatabase of human annotated input/output deep Learning architectures have been used for Image -to- Image transformations, [Isola et ; Liu et ; Longet ; Xu et ; Yan et ]. However, most priorwork incur a heavy computational cost that scales linearly withthe size of the input Image , usually because of the large number ofstacked convolutions and non-linearities that must be evaluated atfull resolution.

10 This general form allows for flexible models to belearned, but this expressivity comes at a price: such architectures areorders of magnitude too slow for Real-Time viewfinder applications,requiring seconds to process a1megapixel Image on the best desktopGPUs more than1000 slower than our proposed model (2msonGPU). Our speedup is enabled by specifically targeting photographictransformations, which are often well-approximated with linearoperations in Bilateral space [Chen et ], and accordinglylearning our model in this present a new network architecture that is capable of learn-ing a rich variety of photographic Image enhancements and canbe rapidly evaluated on high-resolution inputs.