A Quality-of-Experience Index for Streaming Video



Abstract

With the rapid growth of streaming media applications, there has been a strong demand of Quality-of-Experience (QoE) measurement and QoE-driven video delivery technologies. Most existing methods rely on bitrate and global statistics of stalling events for QoE prediction. This is problematic for two reasons. First, using the same bitrate to encode different video content results in drastically different presentation quality. Second, the interactions between video presentation quality and playback stalling experiences are not accounted for. In this work, we first build a streaming video database and carry out a subjective user study to investigate the human responses to the combined effect of video compression, initial buffering and stalling. We then propose a novel QoE prediction approach named Streaming QoE Index (SQI) that accounts for the instantaneous quality degradation due to perceptual video presentation impairment, the playback stalling events, and the instantaneous interactions between them. Experimental results show that the proposed model is in close agreement with subjective opinions and significantly outperforms existing QoE models. The proposed model provides a highly effective and efficient meanings for QoE prediction in video streaming services.

Paper
JSTSP 2016 paper (to appear)
Citation
Bibtex
@article{duanmu2016sqi,
    author    = {Duanmu, Zhengfang and Kai, Zeng and Ma, Kede and Rehman, Abdul and Wang, Zhou}, 
    title     = {A Quality-of-Experience Index for Streaming Video}, 
    journal   = {IEEE Journal of Selected Topics in Signal Processing},
    year      = {2016}
}
Dataset

The Waterloo Quality-of-Experience database consists of 20 RAW HD reference videos covering diverse content. Using aforementioned sequences as the source, each video is encoded into three bitrate levels (500Kbps, 1500Kbps, 3000Kbps) with x264 encoder to cover different quality levels. The choices of bitrate levels are based on commonly-used parameters for transmission of HD videos over networks. A 5-second stalling event is simulated at either the beginning or the middle point of the encoded sequences. The stalling indicator was implemented as a spinning wheel. In total, we obtain 200 test samples that include 20 source videos, 60 compressed videos, 60 initial buffering videos, and 60 mid-stalling videos. The study involved over 25 subjects. Each video in the Waterloo Quality-of-Experience database was assessed in a single-stimulus continuous quality evaluation (SSCQE) with hidden reference.

Results

- Evaluated Algorithms
Algorithm Reference
PSNR Peak signal to noise ratio
SSIM Wang et al. Image quality assessment: from error visibility to structural similarity. TIP. 2004.
MS-SSIM Wang et al. Multi-scale structural similarity for image quality assessment. Asilomar. 2003.
SSIMplus Rehman et al. Display device-adapted video quality-of-experience assessment. HVEI. 2015.
FTW2013 Hoβfeld, et al. Internet video delivery in YouTube: from traffic measurements to quality of experience. TMA. 2013.
Mok2011 Mok et al. Measuring the quality of experience of HTTP video streaming. IM. 2011.
VsQM Rodriguez, et al. Quality metric to assess video streaming service over TCP considering temporal location of pauses. TCE. 2012.
Xue2014 Xue et al. Assessing quality of experience for adaptive HTTP video streaming. ICME. 2014.

- Algorithm Summary

- Performance Comparison

- Scatter Plot