Maximum Differentiation (MAD) Competition

Zhou Wang and Eero P. Simoncelli

MAD_fixmse_iters

MAD competition is an efficient methodology for comparing computational models of a perceptually discriminable quantity.  Rather than pre-selecting test stimuli for subjective evaluation and comparison to the models, stimuli are synthesized so as to optimally distinguish the models.  Specifically, we first synthesize a pair of stimuli that maximize/minimize one model while holding the other fixed.  We then repeat this procedure, but with the roles of the two models reversed.  Subjective testing on pairs of such synthesized stimuli provides a strong indication of the relative strengths and weaknesses of the two models.  Careful study of the stimuli may, in turn, suggest potential ways to improve a model or to combine aspects of multiple models. We demonstrate the methodology using two examples: contrast perception and perceptual image quality assessment.

Demonstration:

MAD competition between two full-reference image quality assessment models: Mean squared error (MSE) and Structural Similarity Index (SSIM)

Synthesizing maximum/minimum SSIM images along the equal-MSE contour in the image space:

mad2.gif

 

Synthesizing maximum/minimum SSIM images at different MSE levels:

 

fixmse.gif

Reference:

Zhou Wang and Eero P. Simoncelli, "Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual discriminability," Journal of Vision, vol. 8, no. 12, pp. 1-13, Sept. 2008.

 

Created Nov. 21, 2007, Updated Sept. 25, 2008