IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4 ...

1 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4, APRIL 20041 IMAGE Quality assessment : From Error Visibility toStructural SimilarityZhou Wang,Member, IEEE, Alan C. Bovik,Fellow, IEEEH amid R. Sheikh,Student Member, IEEE, and Eero P. Simoncelli,Senior Member, IEEEA bstract Objective methods for assessing perceptual im-age quality have traditionally attempted to quantify the vis-ibility of errors between a distorted IMAGE and a referenceimage using a variety of known properties of the humanvisual system. Under the assumption that human visualperception is highly adapted for extracting structural infor-mation from a scene, we introduce an alternative frameworkfor quality assessment based on the degradation of struc-tural information.

2 As a specific example of this concept,we develop a Structural Similarity Index and demonstrateits promise through a set of intuitive examples, as well ascomparison to both subjective ratings and state-of-the-artobjective methods on a database of images compressed withJPEG and Error sensitivity, human visual system (HVS), IMAGE coding, IMAGE quality assessment , JPEG, JPEG2000,perceptual quality, structural information, structural simi-larity (SSIM).I. IntroductionDigital images are subject to a wide variety of distor-tions during acquisition, processing, compression, storage,transmission and reproduction, any of which may resultin a degradation of visual quality.

3 For applications inwhich images are ultimately to be viewed by human be-ings, the only correct method of quantifying visual im-age quality is through subjective evaluation. In practice,however, subjective evaluation is usually too inconvenient,time-consuming and expensive. The goal of research inob-jectiveimage quality assessment is to develop quantitativemeasures that can automatically predict perceived objective IMAGE quality metric can play a variety ofroles in IMAGE processing applications. First, it can beused to dynamicallymonitorand adjust IMAGE quality.

4 Forexample, a network digital video server can examine thequality of video being transmitted in order to control andallocate streaming resources. Second, it can be used tooptimizealgorithms and parameter settings of IMAGE pro-cessing systems. For instance, in a visual communicationThe work of Z. Wang and E. P. Simoncelli was supported by theHoward Hughes Medical Institute. The work of A. C. Bovik and Sheikh was supported by the National Science Foundation and theTexas Advanced Research Program. Z. Wang and E. P. Simoncelli arewith the Howard Hughes Medical Institute, the Center for Neural Sci-ence and the Courant Institute for Mathematical Sciences, New YorkUniversity, New York, NY 10012 USA (email: A.)

5 C. Bovik and H. R. Sheikh are with theLaboratory for IMAGE and Video Engineering (LIVE), Departmentof Electrical and Computer Engineering, The University of Texasat Austin, Austin, TX 78712 USA (email: of the proposed algorithm is availableonline ~lcv/ssim/.system, a quality metric can assist in the optimal design ofprefiltering and bit assignment algorithms at the encoderand of optimal reconstruction, error concealment and post-filtering algorithms at the decoder. Third, it can be usedtobenchmarkimage processing systems and IMAGE quality metrics can be classified accord-ing to the availability of an original (distortion-free) IMAGE ,with which the distorted IMAGE is to be compared.)

6 Mostexisting approaches are known asfull-reference, meaningthat a complete reference IMAGE is assumed to be known. Inmany practical applications, however, the reference imageis not available, and ano-referenceor blind quality as-sessment approach is desirable. In a third type of method,the reference IMAGE is only partially available, in the formof a set of extracted features made available as side infor-mation to help evaluate the quality of the distorted is referred to asreduced-referencequality paper focuses on full-reference IMAGE quality simplest and most widely used full-reference qualitymetric is the mean squared error (MSE), computed by aver-aging the squared intensity differences of distorted and ref-erence IMAGE pixels, along with the related quantity of peaksignal-to-noise ratio (PSNR).

7 These are appealing becausethey are simple to calculate, have clear physical meanings,and are mathematically convenient in the context of opti-mization. But they are not very well matched to perceivedvisual quality ( , [1] [9]). In the last three decades, agreat deal of effort has gone into the development of qualityassessment methods that take advantage of known charac-teristics of the human visual system (HVS). The majorityof the proposed perceptual quality assessment models havefollowed a strategy of modifying the MSE measure so thaterrors are penalized in accordance with their visibility.

8 Sec-tion II summarizes this type of error-sensitivity approachand discusses its difficulties and limitations. In Section III,we describe a new paradigm for quality assessment , basedon the hypothesis that the HVS is highly adapted for ex-tracting structural information. As a specific example, wedevelop a measure of structural similarity that compares lo-cal patterns of pixel intensities that have been normalizedfor luminance and contrast. In Section IV, we compare thetest results of different quality assessment models againsta large set of subjective ratings gathered for a database of344 images compressed with JPEG and TRANSACTIONS ON IMAGE PROCESSING, VOL.

9 13, NO. 4, APRIL 2004 Reference signal Distorted signal Quality/ Distortion Measure Channel Decomposition Error Normalization .. Error Pooling Pre- processing CSF Filtering .. Fig. 1. A prototypical quality assessment system based on error sensitivity. Note that the CSF feature can be implemented either as aseparate stage (as shown) or within Error Normalization .II. IMAGE Quality assessment Based on ErrorSensitivityAn IMAGE signal whose quality is being evaluated canbe thought of as a sum of an undistorted reference signaland an error signal. A widely adopted assumption is thatthe loss of perceptual quality is directly related to the vis-ibility of the error signal.

10 The simplest implementationof this concept is the MSE, which objectively quantifiesthe strength of the error signal. But two distorted imageswith the same MSE may have very different types of errors ,some of which are much more visible than others. Mostperceptual IMAGE quality assessment approaches proposedin the literature attempt to weight different aspects of theerror signal according to their visibility, as determined bypsychophysical measurements in humans or physiologicalmeasurements in animals. This approach was pioneeredby Mannos and Sakrison [10], and has been extended bymany other researchers over the years.