The SSIM Index for Image Quality Assessment

# Zhou Wang, Alan C. Bovik, Hamid R. Sheikhand Eero P. Simoncelli

Visit the webpage of SSIMplus, a novel perceptual video quality-of-experience (QoE) index designed for practical use and in many ways goes far beyond what SSIM can do.

The Structural SIMilarity (SSIM) index is a method for measuring the similarity between two images. The SSIM index can be viewed as a quality measure of one of the images being compared, provided the other image is regarded as of perfect quality. It is an improved version of the universal image quality index proposed before. Detailed description is given in the following paper:

1.      IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.

A Matlab implementation of the SSIM index (ssim_index.m) is available here. You can download it for free, change it as you like and use it anywhere, but please refer to its original source (cite the above paper and this web page).

Before using the code, please go through the suggested usage and demo tests below to get an idea on how to use it and how it works.

Suggested Usage

The above (ssim_index.m) is a single scale version of the SSIM indexing measure, which is most effective if used at the appropriate scale. The precisely “right” scale depends on both the image resolution and the viewing distance and is usually difficult to be obtained.  In practice, we suggest to use the following empirical formula to determine the scale for images viewed from a typical distance (say 3~5 times of the image height or width): 1) Let F = max(1, round(N/256)), where N is the number of pixels in image height (or width); 2) Average local F by F pixels and then downsample the image by a factor of F; and 3) apply the ssim_index.m program. For example, for an 512 by 512 image, F = max(1, round(512/256)) = 2, so the image should be averaged within a 2 by 2 window and downsampled by a factor of 2 before applying ssim_index.m.

The Matlab code (ssim.m) that includes the suggested downsampling process described above is given here:

The SSIM values are computed (using ssim.m) for 6 publicly available subject-rated image databases, including LIVE database, Cornell A57 database, IVC database, Toyama database, TID2008 database, and CSIQ database. When the images are in RGB color, an Matlab function “rgb2gray” is used to convert the images to gray scale. The results (in Matlab .mat format) are provided here, together with performance evaluations based on Spearman’s rank correlation coefficient (SRCC) and Kendall’s rank correlation coefficient (KRCC),  for future comparisons:

 Database Toyama SRCC KRCC LIVE 0.9479 0.7963 Cornell-A57 0.8066 0.6058 IVC 0.9018 0.7223 Toyama-MICT 0.8794 0.6939 TID2008 0.7749 0.5768 CSIQ 0.8756 0.6907

A tutorial paper is published at

2.      Z. Wang and A. C. Bovik, “Mean squared error: love it or leave it? - A new look at signal fidelity measures,” IEEE Signal Processing Magazine, vol. 26, no. 1, pp. 98-117, Jan. 2009.

Powerpoint and PDF figures of the above paper are available here for references:

MATLAB, C, C++, Java and LabView implementations of the SSIM index have also been made available online by other authors:

·         C++ implementation by Lefungus at http://perso.wanadoo.fr/reservoir/

·         C++ implementation by Mehdi Rabah at http://mehdi.rabah.free.fr/SSIM/

·         C++ implementation by Naru Kinjo at http://denshika.cc/open/20100630/, http://denshika.cc/open/20100630/batch.php

·         C++ OpenCV implementation (wrapper of Mehdi Rabah and Naru Kinjo’s work) by Norishige Fukushima at http://nma.web.nitech.ac.jp/fukushima/opencv/ssim/ssim-e.html

·         C implementation of RGBA SSIM for comparing PNG images by porneL at http://pornel.net/dssim

·         C implementation of SSIM and MS-SSIM by Tom Distler at http://tdistler.com/iqa/

·         Java implementation by Gabriel Prieto at http://rsb.info.nih.gov/ij/plugins/ssim-index.html and

·         Java implementation of Multi-scale SSIM by Gabriel Prieto at http://rsb.info.nih.gov/ij/plugins/mssim-index.html and http://www.ucm.es/gabriel_prieto/ms-ssim

·         LabView implementation at LIVE: http://live.ece.utexas.edu/research/quality/SSIM.llb

·         More efficient MATLAB implementation of SSIM by Adam Turcotte at http://ssim.rivetsforbreakfast.com/

·         Python implementation of SSIM by Antoine Vacavant at http://isit.u-clermont1.fr/~anvacava/code.html

·         Python implementation of SSIM (improved upon Antoine Vacavent’s version) by Helder C. R. de Oliveira at http://helderc.net/

Extensions and Related Papers

The following papers discuss the extensions and improvements of the SSIM index approach:

3.      Z. Wang, “On the mathematical properties of the structural similarity index,” IEEE Transactions on Image Processing, vol. 21, no. 4, pp. 1488-1499, Apr. 2012.

4.      Z. Wang and Q. Li, IEEE Transactions on Image Processing, vol. 20, no 5, pp. 1185-1198, May 2011. [Web and code]

5.      M. P. Sampat, Z. Wang, S. Gupta, A. C. Bovik and M. K. Markey, "Complex wavelet structural similarity: A new image similarity index," IEEE Transactions on Image Processing, vol. 18, no. 11, pp. 2385-2401, Nov. 2009.

6.      Z. Wang and Q. Li, Journal of the Optical Society of America A, vol. 24, no. 12, pp. B61-B69, Dec. 2007.

7.      Z. Wang and X. Shang, “Spatial pooling strategies for perceptual image quality assessment,” IEEE International Conference on Image Processing, Atlanta, GA, Oct. 8-11, 2006.

8.      Z. Wang and E. P. Simoncelli,Translation insensitive image similarity in complex wavelet domain,” IEEE International Conference on Acoustics, Speech and Signal Processing, vol. II, pp. 573-576, Philadelphia, PA, Mar. 2005.

9.      Z. Wang, L. Lu, and A. C. Bovik, “Video quality assessment based on structural distortion measurement,” Signal Processing: Image Communication, special issue on “Objective video quality metrics”, vol. 19, no. 2, pp. 121-132, Feb. 2004.

10.  Z. Wang, E. P. Simoncelli and A. C. Bovik,Multi-scale structural similarity for image quality assessment,” Invited Paper, IEEE Asilomar Conference on Signals, Systems and Computers, Nov. 2003. [Matlab Code] [Java Code 1, Java Code 2]

Other related papers:

11.  A. Rehman, Y. Gao, J. Wang, Z. Wang, “Image classification based on complex wavelet structural similarity,” Signal Processing: Image Communication, special issue on Biologically Inspired Approaches for Visual Information Processing and Analysis, vol. 28, no. 8, pp. 984-992, Sep. 2013.

12.  S. Wang, A. Rehman, Z. Wang, S. Ma and W. Gao, “SSIM-inspired divisive normalization for perceptual video coding,” IEEE Transactions on Image Processing, vol. 22, no. 4, pp. 1418-1429, Apr. 2013.

13.  A. Rehman and Z. Wang, “Reduced-reference image quality assessment by structural similarity estimation,” IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3378-3389, Aug. 2012.

14.  S. Wang, A. Rehman, Z. Wang, S. Ma and W. Gao, “SSIM-motivated rate distortion optimization for video coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 4, pp. 516-529, Apr. 2012.

15.  A. Rehman, M. Rostami, Z. Wang, D. Brunet and E. R. Vrscay, “SSIM-inspired image restoration using sparse representation,” EURASIP Journal on Advances in Signal Processing, special issue on Image and Video Quality Improvement Techniques for Emerging Applications, Jan. 2012.

16.  Z. Wang and E. P. Simoncelli, “Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities,” Journal of Vision, vol. 8, no. 12, Sept. 2008.

17.  Z. Wang and E. P. Simoncelli,An adaptive linear system framework for image distortion analysis,” IEEE International Conference on Image Processing, Genoa, Italy, Sept. 11-14, 2005.

18.  Z. Wang, A. C. Bovik and E. P. Simoncelli, “Structural Approaches to image quality assessment,” in Handbook of Image and Video Processing (Al Bovik, ed.), 2nd edition, Academic Press, June 2005.

19.  Z. Wang and E. P. Simoncelli,Stimulus synthesis for efficient evaluation and refinement of perceptual image quality metrics,” Human Vision and Electronic Imaging IX, Proc. SPIE, vol. 5292, Jan. 2004.

20.  Z. Wang, H. R. Sheikh and A. C. Bovik, “Objective video quality assessment,” in The Handbook of Video Databases: Design and Applications (B. Furht and O. Marqure, eds.), CRC Press, pp. 1041-1078, Sept. 2003.

21.  Z. Wang, A. C. Bovik and L. Lu, “Why is image quality assessment so difficult?IEEE International Conference on Acoustics, Speech, & Signal Processing, May 2002.

22.  Z. Wang, and A. C. Bovik, “A universal image quality index,” IEEE Signal Processing Letters, vol. 9, no. 3, pp. 81-84, March 2002.

Demonstration

In the following, all distorted images have roughly the same mean squared error (MSE) values with respect to the original image, but very different quality. SSIM gives a much better indication of image quality.

 Original, MSE = 0; SSIM = 1 MSE = 144, SSIM = 0.988 MSE = 144, SSIM = 0.913 MSE = 144, SSIM = 0.840 MSE = 144, SSIM = 0.694 MSE = 142, SSIM = 0.662

The following example illustrates the maximal/minimal SSIM images synthesized along the equal-MSE hypersphere in the space of all images. All images along the hypersphere have the same MSE values with respect to the reference image, but have drastically different perceptual quality.

The following example demonstrates the SSIM index map, which provides a measurement of local image quality over space. Note that severe JPEG compression produces annoying pseudo-contouring effects (in the sky region) and blocking artifacts (along the boundaries of the building), which are successfully captured by the SSIM index map, yet poorly predicted by the absolute error map (in both distortion/quality maps, brighter indicates better quality).

 Original image JPEG compressed image Absolute error map SSIM index map

Test on LIVE Image Database

The SSIM indexing algorithm has been tested on the LIVE database created at the Lab for Image and Video Engineering (LIVE) at the University of Texas at Austin. The database is subject-rated and available for free download.

First Stage LIVE database test

In the first stage, the database contained 460 images, where 116 were original images and the rest 344 are JPEG and JPEG2000 compressed. Two sample images (cropped from 768X512 to 256X192 for visibility) are shown below. Note that quantization in JPEG and JPEG2000 algorithms often results in smooth representations of fine detail regions (e.g., the tiles in the upper image and the trees in the lower image). Compared with other types of regions, these regions may not be worse in terms of pointwise difference measures (as shown in the absolute error map). However, since the structural information of the image details are nearly completely lost, they exhibit poorer visual quality. Close piece-by-piece comparison of the SSIM index and the absolute error maps, we observe that the SSIM index is more consistent with perceived quality measurement. Note: in both distortion/quality maps, brighter means better quality.

 Original Images JPEG/JPEG2000 Compressed Images Absolute Error Map SSIM Index Map

The scatter plots of the subjective measurement (mean opinion score, MOS) versus the objective predictions (PSNR and MSSIM) are shown below, where each point represent one test image. Clearly, MSSIM is much better in predicting the perceived image quality.

 PSNR vs. MOS MSSIM vs. MOS

Full LIVE database test

The LIVE database was later developed to contain more diverse distortion types, including JPEG2000 compression, JPEG compression, Gaussian noise contamination, Gaussian blur, and JPEG2000 compressed images undergoing fast fading channel distortions. The full database contains 982 images, with 203 original and 779 distorted images. The SSIM index was computed with the full database and the SSIM values for all images are provided in the following file (in Matlab .mat format) for future comparisons:

Created Feb. 2003

Last updated Nov. 9, 2014