arXiv:2111.12704v1 [cs.CV] 24 Nov 2021

1 Investigating Tradeoffs in Real-World Video super -ResolutionKelvin ChanShangchen ZhouXiangyu XuChen Change LoyS-Lab, Nanyang Technological University{chan0899, s200094, , on a Real-World this work, we investigate various tradeoffs caused by the complex and diverse degradationsin real-world VSR. Such tradeoffs are largely neglected in the literature. We propose simple yet effective solutions to the tradeoffs, and theresulting modelRealBasicVSRacts as a strong baseline for real-world VSR.(Zoom-in for best view)AbstractThe diversity and complexity of degradations in real-world video super -resolution (VSR) pose non-trivial chal-lenges in inference and training. First, while long-termpropagation leads to improved performance in cases of milddegradations, severe in-the-wild degradations could be ex-aggerated through propagation, impairing output balance the tradeoff between detail synthesis and arti-fact suppression, we found an image pre-cleaning stage in-dispensable to reduce noises and artifacts prior to propa-gation.}

2 Equipped with a carefully designed cleaning mod-ule, our RealBasicVSR outperforms existing methods inboth quality and efficiency (Fig. 1). Second, real-worldVSR models are often trained with diverse degradations toimprove generalizability, requiring increased batch size toproduce a stable gradient. Inevitably, the increased com-putational burden results in various problems, including1) speed-performance tradeoff and 2) batch-length trade-off. To alleviate the first tradeoff, we propose a stochas-tic degradation scheme that reduces up to 40% of trainingtime without sacrificing performance. We then analyze dif-ferent training settings and suggest that employing longersequences rather than larger batches during training al-lows more effective uses of temporal information, leading tomore stable performance during inference.

3 To facilitate faircomparisons, we propose the new VideoLQ dataset, whichcontains a large variety of real-world low-quality video se-quences containing rich textures and patterns. Our datasetcan serve as a common ground for benchmarking. Code,models, and the dataset will be made publicly available [ ] 24 Nov 20211. IntroductionIn real-world video super -resolution (VSR), we aim atincreasing the resolution of videos containing unknowndegradations. The diversity of degradations in this taskposes significant challenges in designing benchmarks andtraining settings, and hence earlier works assume either syn-thetic [4, 6, 36] or camera-specific [42] degradations and fo-cus on network designs. Although these works achieve re-markable success in restricted settings, the designs for theseover-simplified scenarios cannot generalize well to the com-plex degradations in the wild.

4 In addition, the complexityand diversity of degradations in real-world VSR introduceextra obstacles in both inference and training, including ar-tifact amplification and increased computational paper dives into the problems and tradeoffs in real-world VSR to share useful experiences in addressing is shown by Chanet al. [4] that long-term informationis beneficial to restoration. However, in real-world VSR,such information could also result in exaggerated artifacts,owing to error accumulation during propagation. This phe-nomenon leads to a tradeoff betweenenhancing detailsandsuppressing artifacts, since the synthesizing power of a net-work comes at the cost of amplifying noises and this work, we show that a simple solution can sufficientlyremedy this tradeoff.

5 In particular, we place animage clean-ingmodule prior to propagation for removing degradationsin the input images. The resulting modelRealBasicVSRavoids amplification of artifacts and achieves improved out-put quality while maintaining simplicity. We further de-velop a dynamic refinement scheme that repeatedly appliesthe cleaning module to remove excessive degradations inthe inputs. Our scheme allows a flexible tradeoff betweensmoothnessanddetailedness, which can be adjusted basedon a pre-defined threshold or user preference. A system-atic analysis of different combinations of losses and archi-tectures is conducted to demonstrate the significance of VSR models are generally trained with di-verse degradations to improve generalizability, and hencethey are often trained with increased batch size to ensurestable gradient.

6 As a result, real-world VSR usually re-quires a longer training time and more immense computa-tional resources than the non-blind counterpart. This workinspects two tradeoffs in real-world VSR to improve train-ing efficiency, hence shortening research , with increased batch size, training with long se-quences is prohibitive owing to the I/O bottleneck inducedby hardware limitations. The bottleneck is often alleviatedby reducing either the batch size or sequence length, whichresults in degraded performance. To ameliorate the prob-lem, we propose astochastic degradationscheme that ef-fectively reduces the I/O bottleneck without sacrificing theoutput quality. Notably, our degradation scheme yields upto40%reduction of training time in comparison to the con-ventional training , with a fixed computational budget, the increasedbatch size in real-world VSR inevitably decreases sequencelength during training.

7 We are interested in the tradeoff be-tween them with an aim to search for a more effective set-ting. To this end, we compare models trained with differ-ent combinations of batch sizes and sequence lengths. Weconclude that networks trained with longer sequences ratherthan larger batches could more effectively employ the long-term information in the input sequence, improving addition to the studies above, we introduce a newbenchmark for real-world VSR. Most existing bench-marks [26, 33, 41, 43] are constructed by contaminating thehigh-resolution (HR) videos with pre-defined most recent RealVSR dataset [42] exploits the dual-camera system in iPhone to capture paired data. Yet, theRealVSR dataset consists of only degradations specificallyfor the iPhone camera.

8 With only pre-defined degradations,the benchmarks mentioned above cannot accurately reflectthe generalizability of the models on real-world videos. Inthis work, we proposeVideoLQ, a real-world video datasetconsisting of diverse LR videos to cover various contents,resolutions, and degradations. Our dataset could serve as acommon benchmark for future methods. We test existingmethods on our datasets. Their quantitative and qualitativeresults and our dataset will be released for ease of Related WorkVideo existing VSR methods [2,4 6, 15 17, 19, 36, 40 42] are trained with pre-defineddegradations ( , either synthetic [26,33,41,43] or camera-specific [42]), and they deteriorate significantly when han-dling unknown degradations in reality. However, extendingfrom non-blind VSR to real-world VSR is non-trivial dueto various problems induced by the complex degradationsin the wild.

9 For example, artifact amplification during long-term propagation limits the performance of existing VSRmethods, and increased computational costs lengthen re-search cycles. In this work, we investigate the challengesin both inference and training, and provide respective solu-tions to the from syntheticsettings [3, 8 10, 38, 46],blindsuper-resolution [12, 14,18, 24, 25, 28, 40] assumes the inputs are degraded by aknown process with unknown parameters. The networksare trained with a pre-defined set of degradations with theparameters chosen at random. While the trained networksare able to restore images/videos with a range of degra-dations, the variation of degradations is often limited, andthe generalizability to real-world degradations is in VSRReal-World VSRI nputL = 1L = 5L = 100 InputL = 1L = 5L = 100 Figure of Long-Term employing long-term information leads to improved performance in non-blind VSR,propagation in real-world scenarios could lead to undesirable the sequence length.

10 (Zoom-in for best view)Two recent studies [37, 45] propose to employ more diversedegradations for data augmentation during training. By us-ing ESRGAN [39] with no changes in architecture, thesetwo methods demonstrate promising performance in real-world images. However, we find that such a direct extensionat the data augmentation level is not feasible in real-worldVSR as the network tends to amplify the noise and this work, we investigate the cause and propose a simpleyet effectiveimage cleaningmodule to remedy the with the cleaning module,RealBasicVSRoutper-forms existing works, including [37, 45], in both quality this study, we find that a seem-ingly trivial image cleaning module is critical to removedegradations prior to propagation and suppress artifacts inthe outputs.