Dynamic Attentive Graph Learning for Image Restoration

1 Dynamic Attentive Graph Learning for Image RestorationChong Mou , Jian Zhang , , Zhuoyuan Wu Peking University Shenzhen Graduate School, Shenzhen, China Peng Cheng Laboratory, Shenzhen, self-similarity in natural images has been ver-ified to be an effective prior for Image Restoration . However,most existing deep non-local methods assign a fixed numberof neighbors for each query item, neglecting the dynamicsof non-local correlations. Moreover, the non-local correla-tions are usually based on pixels, prone to be biased due toimage degradation. To rectify these weaknesses, in this pa-per, we propose a Dynamic Attentive Graph Learning model(DAGL) to explore the Dynamic non-local property on patchlevel for Image Restoration . Specifically, we propose an im-proved Graph model to perform patch-wise Graph convo-lution with a Dynamic and adaptive number of neighborsfor each node.

2 In this way, Image content can adaptivelybalance over-smooth and over-sharp artifacts through thenumber of its connected neighbors, and the patch-wise non-local correlations can enhance the message passing pro-cess. Experimental results on various Image restorationtasks: synthetic Image denoising, real Image denoising, im-age demosaicing, and compression artifact reduction showthat our DAGL can produce state-of-the-art results withsuperior accuracy and visual quality. The source code isavailable at IntroductionImage Restoration (IR) is typically an ill-posed inverseproblem aiming to restore a high-quality Image (IHQ)from its degraded measurement (ILQ) corrupted by vari-ous degradation factors. The degradation process can bedefined asILQ=HIHQ+n, whereHis a linear degra-dation matrix, andnrepresents additive noise [48, 55].

3 Ac-cording toH, IR can be categorized into many subtasks, , denoising, compression artifact reduction, demosaic-ing, super-resolution, compressive sensing [49, 53, 54, 46].The rise of deep Learning has greatly facilitated the de-This work was supported in part by National Natural Science Founda-tion of China (61902009). (Corresponding author: Jian Zhang.)velopment of Image Restoration . Many deep Learning -basedmethods [49, 50, 51, 33] have been proposed to solve thisill-posed problem. Despite the remarkable success, mostmethods focus on Learning from a lot of external trainingdata without fully utilizing the internal prior in contrast, many classic model-based methods are imple-mented based on various priors, , total variation [26],sparse representation [9, 10, 47], and self-similarity [4, 7].The self-similarity assumes that similar content would re-cur across the whole Image , and the local content can berecovered with the help of similar items from other by [4], non-local neural networks [40] utilized self-similarity via deep networks, which are subsequently in-troduced to many Image Restoration tasks [20, 52].

4 How-ever, these pixel-wise non-local methods are easily influ-enced by noisy signals within corrupted images. [18, 19]were proposed to establish long-range correlations on patchlevel. Nevertheless, the patch matching step is isolatedfrom the training process. In N3 Net [29], a differentiableK-Nearest Neighbor (KNN) method was proposed. How-ever, restricted by the high complexity of channel-wise fea-ture fusion, N3 Net can only perform the non-local opera-tion within a small search region (10 10) and a smallnumber of matching very recent meth-ods [24, 23, 5] proposed more efficient patch-wise non-localmethods. But they followed the same paradigm as existingnon-local methods to construct fully connected general, the repeatability of different Image contentis distinct, causing different requirements of non-local cor-relations in restoring different Image content.

5 An earlywork [56] has well studied this property, finding that smoothimage contents recur much more frequently than compleximage details, and they should be treated convolutional network (GCN) is a special non-local method designed to process the Graph data by es-tablishing long-range correlations in non-Euclidean , the large domain gap limits the application of thisflexible non-local method in computer vision , few works [36, 35, 21] proposed to apply GCN toimage Restoration tasks. Specifically, [36] and [35] are built4328based on Edge-Conditioned Convolution (ECC) [32] for im-age denoising. However, they constructed the long-rangecorrelations based on pixels and assigned a fixed number ofneighbors for each Graph node. In [21], a patch-wise GCNmethod is proposed for facial expression Restoration . Nev-ertheless, the adjacency matrix is predefined based on thefacial structure and isolated from the training process.

6 Inaddition to ECC, Graph attention network (GAT) [38] is apopular Graph model combined with attention mechanismto identify the importance of different neighboring by GAT, in this paper, we propose a novel dy-namic Attentive Graph Learning model (DAGL) for imagerestoration. In our proposed DAGL, the corrupted imageis recovered in an Image -specific and adaptive Graph con-structed based on local feature Related WorksOur model is closely related to Image Restoration algo-rithms, non-local attention methods, and Graph convolu-tional networks. Since in what follows, we give a brief re-view of these aspects and some most relevant Image Restoration ArchitecturesDriven by the success of deep Learning , almost all re-cent top-performing Image Restoration methods are imple-mented based on deep networks. Stacking convolutionallayers is the most well-known CNN-based strategy.

7 Dongetal. proposed ARCNN [8] for Image Restoration with severalstacked convolutional layers. Subsequently, [49, 51, 50] uti-lized deeper convolutional architecture and residual learn-ing to further enhance Image Restoration performance. Re-cently, abundant novel models and function units were pro-posed. MemNet [33] utilized the dense connection in con-volutional layers for Image denoising. To enlarge the re-ceptive field, hourglass-shaped architecture [14, 43, 3, 44,17, 30], dilated convolution [50, 39], and very deep residualnetworks [53, 52] are often used. However, most methodsare plain networks and neglect to use non-local Non-local Prior for Image RestorationNon-local self-similarity is an effective prior that hasbeen widely used in Image Restoration tasks. Some clas-sic methods [7, 4] utilized self-similarity for Image denois-ing and achieved attractive performance.

8 Following the im-portance of self-similarity, some recent approaches [52, 20]utilized this prior based on non-local neural networks [40].Moreover, some patch-wise non-local methods [18, 19, 29]or transformer-based methods [5, 24] were proposed. Thesemethods performed matching and aggregation in a non-local manner can be generally defined as: xi=1ziXj Q (yi,yj)G(yj), i,(1)whereQrefers to the search region, andzirepresents thenormalizing constant calculated byzi=Pj Q (yi,yj).The function computes pair-wise affinity between queryitemyiand key a feature transformationfunction that generates a new representation ofyj. Whilethe above operation aggregates adequate information for thequery item, the feature aggregation is restricted to be fullyconnected, involving all features within the search region,no matter how similar they are to the query Graph Convolutional Networks (GCN)By extending convolutional neural networks (CNN)from grid data, such as images and videos, to Graph -structured data, GCN has been attracting growing atten-tion from the computer vision community due to its ro-bust capacity of non-local feature aggregation.

9 Not thatwithout loss of generality, the commonly used non-localneural networks [40] can be viewed as a fully connectedgraph [12]. Recently, [21] utilized the predefined adjacencymatrix to perform Graph convolution for facial expressionrestoration. [36, 35] applied Edge-Conditioned Convolution(ECC) [32], a well-known GCN method, to Image denois-ing task. [27] further extended ECC to 3D denoising us consider a Graph that containsNnodes:G= (V,E),whereVis the set of Graph nodes, andEis the set of Rl1denote a Graph node andeij= (vi,vj) Edenote an edge pointing fromvjtovi. In ECC, there ex-ists a shared filter-generating networkF:Rs7 Rl2 an edge labeleij Rs, it outputs an edge-specificembedding matrix ij Rl2 l1. The aggregation processof ECC is an averaging operation embedded by the edge-specific embedding matrix, which can be formalized as: vi=1|Ni|Xj NiF(eij)vj+b=1|Ni|Xj Ni ijvj+b,(2)whereNi={j|(vi,vj) E}is the set of indexes of neigh-boring nodes ofvi, andb Rl2is a learnable bias.

10 Apartfrom ECC, Graph attention network (GAT) [38] is also apopular GCN method, and our proposed DAGL is inspiredby this method. Unlike ECC generating an embedding ma-trix through the edge label to perform embedding and aver-aging aggregation, GAT developed an attention weight foreach edge based on the self-attention mechanism [37]. Inthis way, each node can aggregate the information selec-tively from all its connected neighbors. The calculation ofthe attention weight is defined as: ij=exp(LeakyReLU(aT[Wvi||Wvj]))Pk Niexp(LeakyReLU(aT[Wvi||Wvk])),(3)whereW Rl2 l1anda R2l2 1refer to the learn-able weight matrixes of shared linear transformations, and||represents the concatenating operation. In the process of4329 Figure 1. Proposed Dynamic Attentive Graph Learning model (DAGL). The feature extraction module (FEM) employs residual blocks to ex-tract deep features.