The dynamic adaptive streaming over HTTP provides an inter-operable solution to overcome the volatile network conditions, but its complex characteristic brings new challenges to objective video quality-of-experience (QoE) measurement. To test the generalizability and to facilitate the wide usage of QoE measurement techniques in real-world applications, we establish a new database named Waterloo Streaming QoE Database III (SQoE-III). Unlike existing databases constructed with hand-crafted test sequences, the SQoE-III database, so far the largest and most realistic of its kind, consists of a total of 450 streaming videos created from diverse source content and diverse distortion patterns, with six adaptation algorithms of diverse characteristics under 13 representative network conditions. All streaming videos are assessed by 34 subjects, and a comprehensive evaluation is conducted on the performance of 15 objective QoE models from four categories with regards to their efficacy in predicting subjective QoE. Detailed correlation analysis and statistical hypothesis testing are carried out. The results of this paper shed light on the future development of adaptive bitrate streaming algorithm and video QoE monitoring system.

The Waterloo Streaming Quality-of-Experience Database-III consists of 20 RAW HD reference videos and 450 simulated streaming videos, and of an average duration of 13 seconds. In order to generate meaningful and representative test videos, we conducted a set of DASH video streaming experiments, recorded the relevant streaming activities, and reconstructed the streaming session using video processing tools. The streaming sessions generated by 6 adaptive streaming algorithms including rate-based, BBA, AIMD, Elastic, QDASH, and FESTIVE under 13 wide-ranging bandwidth conditions are recorded and evaluated by 34 subjects. Subjects score the quality of each video sequence according to the 0-100 numerical quality scale.

@article{duanmu2018sqoe,
  author = {Duanmu, Zhengfang and Rehman, Abdul and Wang, Zhou}, 
  title = {A Quality-of-Experience Database for Adaptive Video Streaming}, 
  journal = {IEEE Transactions on Broadcasting},
  volume = {64},
  number = {2},
  pages = {474-487},
  year = {2018},
  month = {June}
}

Algorithm	Reference
FTW	Toβias et al. Internet video delivery in YouTube: From traffic measurements to Quality of Experience. Data Traffic Monitoring and Analysis. 2013
VQM	Pinson et al. A new standardized method for objectively measuring video quality. TBC. 2004.
Mok2011	Mok et al. Measuring the Quality of Experience of HTTP video streaming. IIM. 2011.
Kim2012	Kim et al. QoE assessment model for video streaming service using QoS parameters in wired-wireless network. Int. Conf. Advanced Communication Technology. 2012.
SSIMplus	Rehman, et al. Display device-adapted video Quality-of-Experience assessment. Proc. SPIE. 2015.
VsQM	Rodriguez, et al. Quality metric to assess video streaming service over TCP considering temporal location of pauses. TCE. 2012.
Liu2012	Liu et al. A case for a coordinated internet video control plane. SIGCOMM. 2012.
ATLAS	Bampis et al. Learning to predict streaming video QoE: Distortions, rebuffering and memory. TIP. 2018.
Xue2014	Xue et al. Assessing quality of experience for adaptive HTTP video streaming. ATLAS. 2014.
NARX-QoE	Bampis et al. Continuous prediction of streaming video QoE using dynamic networks. SPL. 2017.
Bentaleb2016	Bentaleb et al. SDNDASH: Improving QoE of HTTP Adaptive Streaming Using Software Defined Networking. MM. 2016.
Liu2015	Liu et al. Deriving and Validating User Experience Model for DASH Video Streaming. TBC. 2015.
Yin2015	Liu et al. A control-theoretic approach for dynamic adaptive video streaming over HTTP. SIGCOMM. 2015.
SQI	Duanmu et al. A Quality-of-Experience Index for Streaming Video. JSTSP. 2017.
P.NATS	Robitza et al. A modular HTTP adaptive streaming QoE model-Candidate for ITU-T P. 1203 ("P. NATS"). QoMEX. 2017.

Rights to all videos are retained by the photographers. For researchers and educators who wish to use the videos for non-commercial research and/or educational purposes, we can provide access under the following terms:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
2. University of Waterloo makes no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
3. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify University of Waterloo, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
4. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
5. University of Waterloo reserves the right to terminate Researcher's access to the Database at any time.
6. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.