Human Vision and Electronic Imaging IX, Proc. SPIE, vol. 5292, Jan. 2004
Stimulus Synthesis for Efficient Evaluation and Refinement of Perceptual Image Quality Metrics
Howard Hughes Medical Institute, Center for Neural Science and Courant Institute of Mathematical Sciences, New York University, New York, NY 10003
Abstract: Evaluation and comparison of perceptual image quality metrics is an important issue that has become more critical given the recent increase in the number of proposed metrics. A standard solution is to compare the metrics with ratings by human subjects on a large database of images. In order to reduce the number of subjective comparisons, the form of distortion is usually highly restricted. Thus, despite the substantial time involved in gathering psychophysical data, there is no guarantee that the test results on these restricted databases provide a sufficient test for a “general-purpose” image quality metric. In this paper, we propose to synthesize image stimuli that best differentiate two candidate quality metrics. Given two image quality metrics, we start from an initial distorted image and iteratively search for the best/worst images in terms of one metric while constraining the other metric to remain fixed. We then repeat this, reversing the roles of the two metrics. Subjective evaluation on the quality of pairs of these images generated at different initial distortion levels provides a strong indication of the relative strength and weaknesses of the quality metrics being compared. This methodology also provides an efficient way to further refine the definition of an image quality metric, and may potentially be generalized to a much broader range of psychophysical studies for performance evaluation of computational perceptual models.