IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4 ...

1 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4, APRIL 20041 IMAGE Quality Assessment: From Error Visibility toStructural SimilarityZhou Wang,Member, IEEE, Alan C. Bovik,Fellow, IEEEH amid R. Sheikh,Student Member, IEEE, and Eero P. Simoncelli,Senior Member, IEEEA bstract Objective methods for assessing perceptual im-age quality have traditionally attempted to quantify the vis-ibility of errors between a distorted IMAGE and a referenceimage using a variety of known properties of the humanvisual system. Under the assumption that human visualperception is highly adapted for extracting structural infor-mation from a scene, we introduce an alternative frameworkfor quality assessment based on the degradation of struc-tural information. As a specific example of this concept,we develop a Structural Similarity Index and demonstrateits promise through a set of intuitive examples, as well ascomparison to both subjective ratings and state-of-the-artobjective methods on a database of images compressed withJPEG and Error sensitivity, human visual system (HVS), IMAGE coding, IMAGE quality assessment, JPEG, JPEG2000,perceptual quality, structural information, structural simi-larity (SSIM).

2 I. IntroductionDigital images are subject to a wide variety of distor-tions during acquisition, processing, compression, storage,transmission and reproduction, any of which may resultin a degradation of visual quality. For applications inwhich images are ultimately to be viewed by human be-ings, the only correct method of quantifying visual im-age quality is through subjective evaluation. In practice,however, subjective evaluation is usually too inconvenient,time-consuming and expensive. The goal of research inob-jectiveimage quality assessment is to develop quantitativemeasures that can automatically predict perceived objective IMAGE quality metric can play a variety ofroles in IMAGE processing applications. First, it can beused to dynamicallymonitorand adjust IMAGE quality. Forexample, a network digital video server can examine thequality of video being transmitted in order to control andallocate streaming resources.

3 Second, it can be used tooptimizealgorithms and parameter settings of IMAGE pro-cessing systems. For instance, in a visual communicationThe work of Z. Wang and E. P. Simoncelli was supported by theHoward Hughes Medical Institute. The work of A. C. Bovik and Sheikh was supported by the National Science Foundation and theTexas Advanced Research Program. Z. Wang and E. P. Simoncelli arewith the Howard Hughes Medical Institute, the Center for Neural Sci-ence and the Courant Institute for Mathematical Sciences, New YorkUniversity, New York, NY 10012 USA (email: A. C. Bovik and H. R. Sheikh are with theLaboratory for IMAGE and Video Engineering (LIVE), Departmentof Electrical and Computer Engineering, The University of Texasat Austin, Austin, TX 78712 USA (email: of the proposed algorithm is availableonline ~lcv/ssim/.system, a quality metric can assist in the optimal design ofprefiltering and bit assignment algorithms at the encoderand of optimal reconstruction, error concealment and post-filtering algorithms at the decoder.))

4 Third, it can be usedtobenchmarkimage processing systems and IMAGE quality metrics can be classified accord-ing to the availability of an original (distortion-free) IMAGE ,with which the distorted IMAGE is to be compared. Mostexisting approaches are known asfull-reference, meaningthat a complete reference IMAGE is assumed to be known. Inmany practical applications, however, the reference imageis not available, and ano-referenceor blind quality as-sessment approach is desirable. In a third type of method,the reference IMAGE is only partially available, in the formof a set of extracted features made available as side infor-mation to help evaluate the quality of the distorted is referred to asreduced-referencequality paper focuses on full-reference IMAGE quality simplest and most widely used full-reference qualitymetric is the mean squared error (MSE), computed by aver-aging the squared intensity differences of distorted and ref-erence IMAGE pixels, along with the related quantity of peaksignal-to-noise ratio (PSNR).

5 These are appealing becausethey are simple to calculate, have clear physical meanings,and are mathematically convenient in the context of opti-mization. But they are not very well matched to perceivedvisual quality ( , [1] [9]). In the last three decades, agreat deal of effort has gone into the development of qualityassessment methods that take advantage of known charac-teristics of the human visual system (HVS). The majorityof the proposed perceptual quality assessment models havefollowed a strategy of modifying the MSE measure so thaterrors are penalized in accordance with their visibility. Sec-tion II summarizes this type of error-sensitivity approachand discusses its difficulties and limitations. In Section III,we describe a new paradigm for quality assessment, basedon the hypothesis that the HVS is highly adapted for ex-tracting structural information. As a specific example, wedevelop a measure of structural similarity that compares lo-cal patterns of pixel intensities that have been normalizedfor luminance and contrast.

6 In Section IV, we compare thetest results of different quality assessment models againsta large set of subjective ratings gathered for a database of344 images compressed with JPEG and TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4, APRIL 2004 Reference signal Distorted signal Quality/ Distortion Measure Channel Decomposition Error Normalization .. Error Pooling Pre- processing CSF Filtering .. Fig. 1. A prototypical quality assessment system based on error sensitivity. Note that the CSF feature can be implemented either as aseparate stage (as shown) or within Error Normalization .II. IMAGE Quality Assessment Based on ErrorSensitivityAn IMAGE signal whose quality is being evaluated canbe thought of as a sum of an undistorted reference signaland an error signal. A widely adopted assumption is thatthe loss of perceptual quality is directly related to the vis-ibility of the error signal. The simplest implementationof this concept is the MSE, which objectively quantifiesthe strength of the error signal.

7 But two distorted imageswith the same MSE may have very different types of errors,some of which are much more visible than others. Mostperceptual IMAGE quality assessment approaches proposedin the literature attempt to weight different aspects of theerror signal according to their visibility, as determined bypsychophysical measurements in humans or physiologicalmeasurements in animals. This approach was pioneeredby Mannos and Sakrison [10], and has been extended bymany other researchers over the years. Reviews on imageand video quality assessment algorithms can be found in[4],[11] [13].A. FrameworkFig. 1 illustrates a generic IMAGE quality assessmentframework based on error sensitivity. Most perceptualquality assessment models can be described with a simi-lar diagram, although they differ in detail. The stages ofthe diagram are as follows:Pre-processing. This stage typically performs a varietyof basic operations to eliminate known distortions from theimages being compared.

8 First, the distorted and referencesignals are properly scaled and aligned. Second, the signalmight be transformed into a color space ( , [14]) that ismore appropriate for the HVS. Third, quality assessmentmetrics may need to convert the digital pixel values storedin the computer memory into luminance values of pixels onthe display device through pointwise nonlinear transforma-tions. Fourth, a low-pass filter simulating the point spreadfunction of the eye optics may be applied. Finally, the ref-erence and the distorted images may be modified using anonlinear point operation to simulate light Filtering. The contrast sensitivity function (CSF)describes the sensitivity of the HVS to different spatial andtemporal frequencies that are present in the visual stim-ulus. Some IMAGE quality metrics include a stage thatweights the signal according to this function (typically im-plemented using a linear filter that approximates the fre-quency response of the CSF).

9 However, many recent met-rics choose to implement CSF as a base-sensitivity normal-ization factor after channel Decomposition. The images are typically sep-arated into subbands (commonly called channels in thepsychophysics literature) that are selective for spatial andtemporal frequency as well as orientation. While somequality assessment methods implement sophisticated chan-nel decompositions that are believed to be closely re-lated to the neural responses in the primary visual cortex[2],[15] [19], many metrics use simpler transforms such asthe discrete cosine transform (DCT) [20], [21] or separa-ble wavelet transforms [22] [24]. Channel decompositionstuned to various temporal frequencies have also been re-ported for video quality assessment [5],[25].Error Normalization. The error (difference) between thedecomposed reference and distorted signals in each channelis calculated and normalized according to a certain maskingmodel, which takes into account the fact that the presenceof one IMAGE component will decrease the visibility of an-other IMAGE component that is proximate in spatial or tem-poral location, spatial frequency, or orientation.

10 The nor-malization mechanism weights the error signal in a channelby a space-varying visibility threshold [26]. The visibilitythreshold at each point is calculated based on the energyof the reference and/or distorted coefficients in a neighbor-hood (which may include coefficients from within a spatialneighborhood of the same channel as well as other chan-nels) and the base-sensitivity for that channel. The normal-ization process is intended to convert the error into units ofjust noticeable difference (JND). Some methods also con-sider the effect of contrast response saturation ( , [2]).Error Pooling. The final stage of all quality metrics mustcombine the normalized error signals over the spatial extentof the IMAGE , and across the different channels, into a singlevalue. For most quality assessment methods, pooling takesthe form of a Minkowski norm:E({el,k}) =( l k|el,k| )1/ (1)whereel,kis the normalized error of thek-th coefficient inthel-th channel, and is a constant exponent typicallychosen to lie between 1 and 4.