Multi-Scale Progressive Fusion Network for Single Image ...

1 Multi-Scale Progressive Fusion Network for Single Image DerainingKui Jiang1 Zhongyuan Wang1 Peng Yi1 Chen Chen2 Baojin Huang1 Yimin Luo3 Jiayi Ma1 Junjun Jiang41 Wuhan University2 University of North Carolina at Charlotte3 King s College London4 Harbin Institute of TechnologyAbstractRain streaks in the air appear in various blurring de-grees and resolutions due to different distances from theirpositions to the camera. Similar rain patterns are visi-ble in a rain Image as well as its Multi-Scale (or multi -resolution) versions, which makes it possible to exploit suchcomplementary information for rain streak this work, we explore the Multi-Scale collaborative rep-resentation for rain streaks from the perspective of inputimage scales and hierarchical deep features in a unifiedframework, termed Multi-Scale Progressive Fusion Network (MSPFN) for Single Image rain streak removal. For thesimilar rain streaks at different positions, we employ re-current calculation to capture the global texture, thus al-lowing to explore the complementary and redundant infor-mation at the spatial dimension to characterize target rainstreaks.

2 Besides, we construct Multi-Scale pyramid struc-ture, and further introduce the attention mechanism to guidethe fine Fusion of these correlated information from dif-ferent scales. This Multi-Scale Progressive Fusion strategynot only promotes the cooperative representation, but al-so boosts the end-to-end training. Our proposed methodis extensively evaluated on several benchmark datasets andachieves the state-of-the-art results. Moreover, we conductexperiments on joint deraining, detection, and segmenta-tion tasks, and inspire a new research direction of visiontask driven Image deraining. The source code is IntroductionDue to substantial degradation of the Image content inrain images and videos, traditional Image enhancement al-gorithms [27] struggle to make desirable improvements onimage quality. Therefore, developing specialized solution-s for Image deraining is imperative to a wide range oftasks [12], object detection and semantic segmentation.

3 Corresponding authorRain Image (original scale)Rain Image scale from downsamplingInformation Fusion within the same scaleInformation Fusion across other scalesFigure 1. Demonstration of the collaborative representation of rainstreaks. Specifically, similar rain patterns among rain streaks, bothwithin the same scale (highlighted in cyan, pink and dark blue box-es) or cross different scales (highlighted in red, yellow, orange andgreen boxes), can help reconstruct the target rain streak (white boxin the original rain Image ) with the complementary information( similar appearance, formation, etc.).Traditional deraining methods [2,1,5,9,32] use sim-ple linear-mapping transformations and are not robust tovariations of the input [11], , rain streaks with vari-ous directions, densities and sizes. Recently, deep-learningbased methods [6,35,16] which operate with convolution-al and non-linear layers have witnessed remarkable advan-tages over traditional methods.

4 Despite obvious improve-ments on feature representation brought by those method-s [6,16], their Single -scale frameworks can hardly capturethe inherent correlations of rain streaks across repetitive samples of rain streaks in a rain imageas well as its Multi-Scale versions ( Multi-Scale pyramid im-ages) may carry complementary information ( similarappearance) to characterize target rain streaks. As illustrat-ed in , the rain streaks (highlighted in the white box)in the original rain Image share the similar rain patterns withthe rain streaks (highlighted in the cyan, pink and dark blueboxes) at different positions as well as those (highlightedin the red, yellow, orange and green boxes) in the 1/2 s-cale rain Image . Therefore, rain streaks both from the samescale (solid arrows) and across different scales (dashed ar-8346rows) encode complementary or redundant information forfeature representation, which would help deraining in theoriginal Image .

5 This correlation of Image contents across s-cales has been successfully applied to other computer visiontasks [10,33]. Recently, authors in [8,44] construct pyra-mid frameworks to exploit the Multi-Scale knowledge forderaining. Unfortunately, those exploitations fail to makefull use of the correlations of Multi-Scale rain streaks (al-though restricted to a fixed scale-factor of 2 [10]). For ex-ample, Fuet al. [8] decompose the rain Image into differentpyramid levels based on its resolution, and then individu-ally solve the restoration sub-problems at the specific scalespace through several parallel sub-networks. Such decom-position strategy is the basic idea of many recurrent derain-ing frameworks [19]. Unlike [8] completing the derainingtask from each individual resolution level, Zhenget al. [44]present a density-specific optimization for rain streak re-moval in a coarse-to-fine fashion, and gradually produce therain-free Image stage-by-stage [15].

6 However, there are nodirect communications of the inter-level features across cas-caded pyramid layers except for the final outputs, thus fail-ing to take all-rounded advantages of the correlated infor-mation of rain streaks across different scales. Consequent-ly, these methods [8,44] are still far from producing thedesirable deraining results with the limited exploitation andutilization of Multi-Scale rain address these limitations of the prior works, we ex-plore the Multi-Scale representation from input Image scalesand deep neural Network representations in a unified frame-work, and propose a Multi-Scale Progressive Fusion net-work (MSPFN) to exploit the correlated information of rainstreaks across scales for Single Image deraining. Specifi-cally, we first generate the Gaussian pyramid rain imagesusing Gaussian kernels to down-sample the original rainimage in sequence. A coarse- Fusion module (CFM) ( )is designed to capture the global texture information fromthese Multi-Scale rain images through recurrent calculation(Conv-LSTM), thus enabling the Network to cooperativelyrepresent the target rain streak using similar counterpartsfrom global feature space.

7 Meanwhile, the representationof the high-resolution pyramid layer is guided by previousoutputs as well as all low-resolution pyramid layers. A fine- Fusion module (FFM) ( ) is followed to further integratethese correlated information from different scales. By us-ing the channel attention mechanism, the Network not onlydiscriminatively learns the scale-specific knowledge fromall preceding pyramid layers, but also reduces the featureredundancy effectively. Moreover, multiple FFMs can becascaded to form a Progressive Multi-Scale Fusion . Finally,a reconstruction module (RM) is appended to aggregate thecoarse and fine rain information extracted respectively fromCFM and FFM for learning the residual rain Image , which isthe approximation of real rain streak distribution. The over-all framework is outlined in The main contributionsof this paper are as follows: We uncover the correlations of rain streaks in an imageand propose a novel Multi-Scale Progressive Fusion net-work (MSPFN) which collaboratively represents rainstreaks from multiple scales via the pyramid represen-tation.

8 To better characterize rain streaks of different scales,we devise three basic modules, coarse- Fusion module(CFM), fine- Fusion module (FFM) and reconstructionmodule (RM), to effectively extract and integrate themulti-scale information. In these modules, the com-plementary information of similar patterns with rainstreaks, both within the same scale or across differ-ent scales (pyramid layers), is progressively fused tocharacterize the rain streaks distribution in a collabo-rative/cooperative manner. Apart from achieving the state-of-the-art derainingperformance in terms of the conventional quantitativemeasurements ( PSNR and SSIM), we build sever-al synthetic rain datasets based on COCO [3] and BD-D [38] datasets for joint Image deraining, detection andsegmentation tasks. To the best of our knowledge, weare the first to apply mainstream vision-oriented tasks(detection and segmentation) for comprehensively e-valuating the deraining Related WorkIn the last few years, substantial improvements [24,18,4,17] have been observed on rain Image restoration.

9 In thiswork, we mainly focus on Single Image deraining becauseit is more Single Image DerainingPrevious traditional methods for Single Image derain-ing [5,14] fail under the complex rain conditions and pro-duce degraded Image contents due to the limited linear-mapping transformation. Very recently, deep-learningbased approaches [24,29,39] have emerged for rain streakremoval and demonstrated impressive restoration perfor-mance. For example, Fuet al. [6] introduce a three-layerconvolutional neural Network (CNN) to estimate and re-move rain streaks from its rain-contaminated better represent rain streaks, Zhanget al. [40] take therain density into account and present a multi -task CNN forjoint rain density estimation and deraining. Later, Zhangetal. [41] further incorporate quantitative, visual and discrim-inative performance into the objective function, and proposea conditional generative adversarial Network for rain streakremoval.

10 In order to alleviate the learning difficulty, recur-rent frameworks [19,36,26] are designed to remove rainstreaks in a stage-wise *R = I RIDerainCoarse- Fusion ModuleReconstruction Module M Cascaded Fine- Fusion ModuleInitial LayerConvolutionResidual Recurrent Unit (RRU)U-shaped Residual Attention Block (URAB)CConcatenationShort/Long Skip ConnectionPixel-wise SummationReconstruction LayerDeconvolutionIRainIRain IRain Figure 2. Outline of the proposed Multi-Scale Progressive Fusion Network (MSPFN). We set the pyramid level to 3 as an example. MSPFN consists of four parts: initial feature extraction, coarse Fusion , fine Fusion , and rain streak reconstruction, which are combined to regressthe residual rain imageI R. We produce the rain-free imageIDerainby subtractingI Rfrom the original rain imageIRain. The goal is tomakeIDerainas close as possible to the rain free Multi-Scale LearningRain streaks in the air show the apparent self-similarity,both within the same scale or across different scales, whichmakes it possible to exploit the correlated informationacross scales for rain streak representation.