Dynamic Attentive Graph Learning for Image Restoration

1 Dynamic Attentive Graph Learning for Image RestorationChong Mou , Jian Zhang , , Zhuoyuan Wu Peking University Shenzhen Graduate School, Shenzhen, China Peng Cheng Laboratory, Shenzhen, self-similarity in natural images has been ver-ified to be an effective prior for Image Restoration . However,most existing deep non-local methods assign a fixed numberof neighbors for each query item, neglecting the dynamicsof non-local correlations. Moreover, the non-local correla-tions are usually based on pixels, prone to be biased due toimage degradation. To rectify these weaknesses, in this pa-per, we propose a Dynamic Attentive Graph Learning model(DAGL) to explore the Dynamic non-local property on patchlevel for Image Restoration .

2 Specifically, we propose an im-proved Graph model to perform patch-wise Graph convo-lution with a Dynamic and adaptive number of neighborsfor each node. In this way, Image content can adaptivelybalance over-smooth and over-sharp artifacts through thenumber of its connected neighbors, and the patch-wise non-local correlations can enhance the message passing pro-cess. Experimental results on various Image restorationtasks: synthetic Image denoising, real Image denoising, im-age demosaicing, and compression artifact reduction showthat our DAGL can produce state-of-the-art results withsuperior accuracy and visual quality.

3 The source code isavailable at IntroductionImage Restoration (IR) is typically an ill-posed inverseproblem aiming to restore a high-quality Image (IHQ)from its degraded measurement (ILQ) corrupted by vari-ous degradation factors. The degradation process can bedefined asILQ=HIHQ+n, whereHis a linear degra-dation matrix, andnrepresents additive noise [48, 55]. Ac-cording toH, IR can be categorized into many subtasks, , denoising, compression artifact reduction, demosaic-ing, super-resolution, compressive sensing [49, 53, 54, 46].The rise of deep Learning has greatly facilitated the de-This work was supported in part by National Natural Science Founda-tion of China (61902009).

4 (Corresponding author: Jian Zhang.)velopment of Image Restoration . Many deep Learning -basedmethods [49, 50, 51, 33] have been proposed to solve thisill-posed problem. Despite the remarkable success, mostmethods focus on Learning from a lot of external trainingdata without fully utilizing the internal prior in contrast, many classic model-based methods are imple-mented based on various priors, , total variation [26],sparse representation [9, 10, 47], and self-similarity [4, 7].The self-similarity assumes that similar content would re-cur across the whole Image , and the local content can berecovered with the help of similar items from other by [4], non-local neural networks [40] utilized self-similarity via deep networks, which are subsequently in-troduced to many Image Restoration tasks [20, 52].

5 How-ever, these pixel-wise non-local methods are easily influ-enced by noisy signals within corrupted images. [18, 19]were proposed to establish long-range correlations on patchlevel. Nevertheless, the patch matching step is isolatedfrom the training process. In N3 Net [29], a differentiableK-Nearest Neighbor (KNN) method was proposed. How-ever, restricted by the high complexity of channel-wise fea-ture fusion, N3 Net can only perform the non-local opera-tion within a small search region (10 10) and a smallnumber of matching very recent meth-ods [24, 23, 5] proposed more efficient patch-wise non-localmethods.

6 But they followed the same paradigm as existingnon-local methods to construct fully connected general, the repeatability of different Image contentis distinct, causing different requirements of non-local cor-relations in restoring different Image content. An earlywork [56] has well studied this property, finding that smoothimage contents recur much more frequently than compleximage details, and they should be treated convolutional network (GCN) is a special non-local method designed to process the Graph data by es-tablishing long-range correlations in non-Euclidean , the large domain gap limits the application of thisflexible non-local method in computer vision , few works [36, 35, 21] proposed to apply GCN toimage Restoration tasks.

7 Specifically, [36] and [35] are built4328based on Edge-Conditioned Convolution (ECC) [32] for im-age denoising. However, they constructed the long-rangecorrelations based on pixels and assigned a fixed number ofneighbors for each Graph node. In [21], a patch-wise GCNmethod is proposed for facial expression Restoration . Nev-ertheless, the adjacency matrix is predefined based on thefacial structure and isolated from the training process. Inaddition to ECC, Graph attention network (GAT) [38] is apopular Graph model combined with attention mechanismto identify the importance of different neighboring by GAT, in this paper, we propose a novel dy-namic Attentive Graph Learning model (DAGL) for imagerestoration.

8 In our proposed DAGL, the corrupted imageis recovered in an Image -specific and adaptive Graph con-structed based on local feature Related WorksOur model is closely related to Image Restoration algo-rithms, non-local attention methods, and Graph convolu-tional networks. Since in what follows, we give a brief re-view of these aspects and some most relevant Image Restoration ArchitecturesDriven by the success of deep Learning , almost all re-cent top-performing Image Restoration methods are imple-mented based on deep networks. Stacking convolutionallayers is the most well-known CNN-based strategy. Dongetal.

9 Proposed ARCNN [8] for Image Restoration with severalstacked convolutional layers. Subsequently, [49, 51, 50] uti-lized deeper convolutional architecture and residual learn-ing to further enhance Image Restoration performance. Re-cently, abundant novel models and function units were pro-posed. MemNet [33] utilized the dense connection in con-volutional layers for Image denoising. To enlarge the re-ceptive field, hourglass-shaped architecture [14, 43, 3, 44,17, 30], dilated convolution [50, 39], and very deep residualnetworks [53, 52] are often used. However, most methodsare plain networks and neglect to use non-local Non-local Prior for Image RestorationNon-local self-similarity is an effective prior that hasbeen widely used in Image Restoration tasks.

10 Some clas-sic methods [7, 4] utilized self-similarity for Image denois-ing and achieved attractive performance. Following the im-portance of self-similarity, some recent approaches [52, 20]utilized this prior based on non-local neural networks [40].Moreover, some patch-wise non-local methods [18, 19, 29]or transformer-based methods [5, 24] were proposed. Thesemethods performed matching and aggregation in a non-local manner can be generally defined as: xi=1ziXj Q (yi,yj)G(yj), i,(1)whereQrefers to the search region, andzirepresents thenormalizing constant calculated byzi=Pj Q (yi,yj).The function computes pair-wise affinity between queryitemyiand key a feature transformationfunction that generates a new representation ofyj.