1 From Beginner to Master: A Survey for Deep Learning ...

1 1 From Beginner to Master: A Survey for DeepLearning-based Single- image Super-ResolutionJuncheng Li , Zehua Pei , and Tieyong Zeng*Abstract Single- image super- resolution (SISR) is an important task in image processing, which aims to enhance the resolution ofimaging systems. Recently, SISR has made a huge leap and has achieved promising results with the help of deep Learning (DL). In thissurvey, we give an overview of DL-based SISR methods and group them according to their targets, such as reconstruction efficiency,reconstruction accuracy, and perceptual accuracy. Specifically, we first introduce the problem definition, research background, and thesignificance of SISR. Secondly, we introduce some related works, including benchmark datasets, upsampling methods, optimizationobjectives, and image quality assessment methods.

2 Thirdly, we provide a detailed investigation of SISR and give some domain-specificapplications of it. Fourthly, we present the reconstruction results of some classic SISR methods to intuitively know their , we discuss some issues that still exist in SISR and summarize some new trends and future directions. This is an exhaustivesurvey of SISR, which can help researchers better understand SISR and inspire more exciting research in this field. An investigationproject for SISR is provided in Terms image super- resolution , single- image super- resolution , SISR, Survey , INTRODUCTIONIMAGE super- resolution (SR), especially single-imagesuper- resolution (SISR), is one kind of image transforma-tion task and has received increasing attention in academicand industry. As shown in Fig. 1, SISR aims to recon-struct a super- resolution (SR) image from its degraded low- resolution (LR) one.

3 It is widely used in various computervision applications, including security and surveillance im-age, medical image reconstruction, video enhancement, andimage SISR methods have been studied long before, suchas bicubic interpolation and Lanczos resampling [1] whichare based on interpolation. However, SISR is an inherentlyill-posed problem, and there always exist multiple HR im-ages corresponding to one original LR image . To solve thisissue, some numerical methods utilize prior information torestrict the solution space of the reconstruction, such asedge-based methods [2] and image statistics-based meth-ods [3]. Meanwhile, there are some widely used learningmethods, such as neighbor embedding methods [4] andsparse coding methods [5], which assume that there exists atransformation between LR and HR , deep Learning (DL) [6] has demonstrated betterperformance than traditional machine Learning models inmany artificial intelligence fields, including computer vi-sion [7] and natural language processing [8].

4 With the rapiddevelopment of DL techniques, numerous DL-based meth-ods have been proposed for SISR, continuously promptingthe State-Of-The-Art (SOTA) forward. Like other imagetransformation tasks, the SISR task can generally be divided *: Corresponding author. : Contribute equally to this work and are co-first authors. J. Li, Z. Pei, and T. Zeng are with the Center for MathematicalArtificial Intelligence (CMAI), Department of Mathematics, The Chi-nese University of Hong Kong. (E-mails: 1. SISR aims to reconstruct a super- resolution (SR) image from itsdegraded low- resolution (LR) three steps: feature extraction and representation, non-linear mapping, and image reconstruction [9]. In traditionalnumerical models, it is time-consuming and inefficient todesign an algorithm satisfying all these processes.)

5 On thecontrary, DL can transfer the SISR task to an almost end-to-end framework incorporating all these three processes,which can greatly decrease manual and computing ex-pense [10]. Additionally, given the ill-posed nature of SISR which can lead to unstable and hard convergence on theresults, DL can alleviate this issue through efficient networkarchitecture and loss functions design. Moreover, modernGPU enables deeper and more complex DL models totrain fast, which show greater representation power thantraditional numerical is well known that DL-based methods can be di-vided into supervised and unsupervised methods. This isthe simplest classification criterion, but the range of thisclassification criterion is too large and not clear.

6 As a result,many technically unrelated methods may be classified intothe same type while methods with similar strategies maybe classified into completely different types. Different [ ] 29 Sep 20212 Fig. 2. The content and taxonomy of this Survey . In this Survey , we divide the DL-based SISR methods into four categories, which are classifiedaccording to their specific targets. Among them, the dark gray blocks are the focus methods in this SISR surveys [11], [12] that use supervision as theclassification criterion or introduce the methods in a pureliterature way, in this Survey , we attempt to give a compre-hensive overview of DL-based SISR methods and categorizethem according to their specific targets. In Fig. 2, we showthe content and taxonomy of this Survey . Obviously, wedivide the DL-based SISR methods into four categories:reconstruction efficiency methods, reconstruction accuracymethods, perceptual quality methods, and further improve-ment methods.

7 This target-based Survey has a clear contexthence it is convenient for readers to consult. Specifically,in this Survey , we first introduce the problem definition,research background, and significance of SISR. Then, we in-troduce some related works, including benchmark datasets,upsample methods, optimization objectives, and assessmentmethods. After that, we provide a detailed investigationof SISR methods and provide the reconstruction results ofthem. Finally, we discuss some issues that still exist in SISRand provide some new trends and future directions. Overall,the main contributions of this Survey are as follows:(1). We give a thorough overview of DL-based SISR methods according to their targets. This is a new perspec-tive that makes the Survey has a clear context hence it isconvenient for readers to consult.

8 (2). This Survey covers more than 100 SR methods andintroduces a series of new tasks and domain-specific appli-cations extended by SISR in recent years.(3). We provide a detailed comparison of reconstructionresults, including classic, latest, and SOTA SISR methods, tohelp readers intuitively know their performance.(4). We discuss some issues that still exist in SISR andsummarize some new trends and future PROBLEMSETTING Problem DefinitionImage super- resolution is a classic technique to improvethe resolution of an imaging system, which can be clas-sified into single- image super- resolution (SISR) and multi- image super- resolution (MISR) according to the number ofthe input LR images. Among them, MISR has graduallydeveloped into video super- resolution (VSR).

9 Comparedwith MISR/VSR, SISR is much more challenging sinceMISR/VSR have extra information for reference while SISR only has information of a single input image for the missingimage features the low- resolution image asIx Rh wand theground-truth high- resolution image asIy RH W, whereH > handW > w. Typically, in a SISR framework, theLR imageIxis modeled asIx=D(Iy; D), whereDisa degradation mapRH W Rh wand Ddenotes thedegradation factor. In most cases, the degradation process isunknown. Therefore, researchers are trying to model it. Themost popular degradation mode is:D(Iy; D) = (Iy ) s+n,(1)whereIy represents the convolution between the blurkernel and the HR imageIy, sis a subsequent down-sampling operation with scale factors, andnis usuallythe additive white Gaussian noise (AWGN) with standard3deviation.

10 In the SISR task, we need to recover a SRimageISRfrom the LR imageIx. Therefore, the task canbe formulated asISR=F(Ix; F), whereFis the SRalgorithm and Fis the parameter set of the SR , researches have converted the SISR into anend-to-end Learning task, relying on massive training datasand effective loss functions. Meanwhile, more and moreDL-based models have been proposed due to the powerfulrepresentation power of CNN and its convenience in bothforward and backward computing. Therefore, the SISR taskcan be transformed into the following optimization goal: F= arg min FL(ISR,Iy) + ( ),(2)whereLdenotes the loss function between the generatedSR imageISRand the HR imageIy, ( )denotes theregularization term, and is the trade-off parameter thatis used to control the percentage of the regularization 3.