With the rapid growth of streaming media applications, there has been a strong demand of Quality-of-Experience (QoE) measurement and QoE-driven video delivery technologies. Most existing methods rely on bitrate and global statistics of stalling events for QoE prediction. This is problematic for two reasons. First, using the same bitrate to encode different video content results in drastically different presentation quality. Second, the interactions between video presentation quality and playback stalling experiences are not accounted for. In this work, we first build a streaming video database and carry out a subjective user study to investigate the human responses to the combined effect of video compression, initial buffering and stalling. We then propose a novel QoE prediction approach named Streaming QoE Index (SQI) that accounts for the instantaneous quality degradation due to perceptual video presentation impairment, the playback stalling events, and the instantaneous interactions between them. Experimental results show that the proposed model is in close agreement with subjective opinions and significantly outperforms existing QoE models. The proposed model provides a highly effective and efficient meanings for QoE prediction in video streaming services.

The Waterloo Quality-of-Experience database consists of 20 RAW HD reference videos covering diverse content. Using aforementioned sequences as the source, each video is encoded into three bitrate levels (500Kbps, 1500Kbps, 3000Kbps) with x264 encoder to cover different quality levels. The choices of bitrate levels are based on commonly-used parameters for transmission of HD videos over networks. A 5-second stalling event is simulated at either the beginning or the middle point of the encoded sequences. The stalling indicator was implemented as a spinning wheel. In total, we obtain 200 test samples that include 20 source videos, 60 compressed videos, 60 initial buffering videos, and 60 mid-stalling videos. The study involved over 25 subjects. Each video in the Waterloo Quality-of-Experience database was assessed in a single-stimulus continuous quality evaluation (SSCQE) with hidden reference.

@article{duanmu2016sqi,
    author    = {Duanmu, Zhengfang and Kai, Zeng and Ma, Kede and Rehman, Abdul and Wang, Zhou}, 
    title     = {A Quality-of-Experience Index for Streaming Video}, 
    journal   = {IEEE Journal of Selected Topics in Signal Processing},
    volume    = {11},
    number    = {1},
    pages     = {154--166},
    year      = {2016},
    month     = {Feb.}
}

Algorithm	Reference
PSNR	Peak signal to noise ratio
SSIM	Wang et al. Image quality assessment: from error visibility to structural similarity. TIP. 2004.
MS-SSIM	Wang et al. Multi-scale structural similarity for image quality assessment. Asilomar. 2003.
SSIMplus	Rehman et al. Display device-adapted video quality-of-experience assessment. HVEI. 2015.
FTW2013	Hoβfeld, et al. Internet video delivery in YouTube: from traffic measurements to quality of experience. TMA. 2013.
Mok2011	Mok et al. Measuring the quality of experience of HTTP video streaming. IM. 2011.
VsQM	Rodriguez, et al. Quality metric to assess video streaming service over TCP considering temporal location of pauses. TCE. 2012.
Xue2014	Xue et al. Assessing quality of experience for adaptive HTTP video streaming. ICME. 2014.

Rights to all videos are retained by the photographers. For researchers and educators who wish to use the videos for non-commercial research and/or educational purposes, we can provide access under the following terms:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
2. University of Waterloo makes no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
3. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify University of Waterloo, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
4. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
5. University of Waterloo reserves the right to terminate Researcher's access to the Database at any time.
6. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.