Human Vision and Electronic Imaging IX, Proc. SPIE, vol. 5292, Jan. 2004
Stimulus Synthesis for
Efficient Evaluation and Refinement of Perceptual Image Quality Metrics
Zhou
Wang and Eero P. Simoncelli
Howard Hughes Medical Institute, Center for Neural
Science and Courant Institute of Mathematical Sciences, New York University,
New York, NY 10003
Abstract: Evaluation
and comparison of perceptual image quality metrics is an important issue that
has become more critical given the recent increase in the number of proposed
metrics. A standard solution is to compare the metrics with ratings by human
subjects on a large database of images. In order to reduce the number of subjective
comparisons, the form of distortion is usually highly restricted. Thus, despite
the substantial time involved in gathering psychophysical data, there is no
guarantee that the test results on these restricted databases provide a
sufficient test for a “general-purpose” image quality metric. In this paper, we
propose to synthesize image stimuli that best differentiate two candidate
quality metrics. Given two image quality metrics, we start from an initial
distorted image and iteratively search for the best/worst images in terms of
one metric while constraining the other metric to remain fixed. We then repeat
this, reversing the roles of the two metrics. Subjective evaluation on the
quality of pairs of these images generated at different initial distortion levels
provides a strong indication of the relative strength and weaknesses of the
quality metrics being compared. This methodology also provides an efficient way
to further refine the definition of an image quality metric, and may
potentially be generalized to a much broader range of psychophysical studies
for performance evaluation of computational perceptual models.