Quality of Experience Management Research

The field Quality of Experience (QoE) Management comprises two major sub-areas: the measurement and modeling of QoE and the application of the QoE models in QoE-aware system design. The former is challenging because users' perceptions are not directly observable and massively multi-dimensional, and they may be affected by immeasurable external factors. The latter is even more challenging because solutions rely on a solid QoE model with consideration of highly-dimensional parameters that may influence the performance of networking and multimedia systems. A brief overview of our research in this area is as follows.

Quality of Experience Measurement and Modeling

Session-Time-Based QoE Modeling

We have proposed an innovative approach to measure and quantify QoE based on large-scale, passive observation of user behavior in real life. Our framework quantifies the relationship between user satisfaction and system-level performance metrics, such as network delay, processing delay, and bandwidth, by statistical methods. To evaluate its effectiveness, we have applied the framework to VoIP services [1] and online games [2,3,4,5], and demonstrated the use of resulting models in QoE provisioning. Our research pioneers in similar studies and has received intensive attention from the research community.
  1. "Quantifying Skype User Satisfaction," Kuan-Ta Chen, Chun-Ying Huang, Polly Huang, and Chin-Laung Lei, Proceedings of ACM SIGCOMM 2006, Oct., 2006. [web | paper]
  2. "Effect of Network Quality on Player Departure Behavior in Online Games," Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei, IEEE Transactions on Parallel and Distributed Systems, Vol. 20, No. 5, pp. 593--606, May, 2009.
  3. "A Generalizable Methodology for Quantifying User Satisfaction," Te-Yuan Huang, Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei, IEICE Transactions on Communications, Vol. E91-B, No. 5, pp. 1260--1268, May, 2008.
  4. "How Sensitive are Online Gamers to Network Quality?," Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei, Communications of the ACM (special issue on Entertainment Networking -- Recreational Use of IP Network), November, 2006. [web | paper]
  5. "On the Sensitivity of Online Game Playing Time to Network QoS," Kuan-Ta Chen, Polly Huang, Guo-Shiuan Wang, Chun-Ying Huang, and Chin-Laung Lei, Proceedings of IEEE INFOCOM 2006. [web | paper]

QoE Measurement and Collection Methodology

Mean Opinion Score (MOS) is commonly used to measure the QoE (i.e., user satisfaction) with network and multimedia systems. However, the approach is inefficient for three reasons: 1) It is difficult to rate content quality by giving an absolute score; 2) it is not economic in terms of experiment time and cost; and 3) the experiments need to be conducted under the limitations of working hours, computer hardware, and place.

To save the nuisance of MOS experiment procedures and costs, we proposed a framework called OneClick [1] to capture users' perceptions in a lightweight approach, which only requires a subject to click a dedicated key whenever he/she feels dissatisfied with the quality of the application in use. The framework is particularly effective because it is intuitive, lightweight, efficient, time-aware, and application-independent. Furthermore, we proposed the first cheat-proof crowdsourceable experiment framework [2,3,4]. By using paired comparison and an algorithm to verify the consistency of each participant's input, the proposed framework enables researchers to outsource their experiments to an Internet crowd without sacrificing the quality of the results and obtain a higher level of participant diversity at a lower monetary cost.

  1. "OneClick: A Framework for Measuring Network Quality of Experience," Kuan-Ta Chen, Cheng Chun Tu, and Wei-Cheng Xiao, Proceedings of IEEE INFOCOM 2009. [web | paper]
  2. "Quadrant of Euphoria: A Crowdsourcing Platform for QoE Assessment," Kuan-Ta Chen, Chi-Jui Chang, Chen-Chi Wu, Yu-Chun Chang, and Chin-Laung Lei, IEEE Network, March, 2010. [web | paper]
  3. "A Crowdsourceable QoE Evaluation Framework for Multimedia Content," Kuan-Ta Chen, Chen-Chi Wu, Yu-Chun Chang, and Chin-Laung Lei, Proceedings of ACM Multimedia 2009. [web | paper]
  4. "Online Game QoE Evaluation using Paired Comparisons," Yu-Chun Chang, Kuan-Ta Chen, Chen-Chi Wu, Chien-Ju Ho, and Chin-Laung Lei, Proceedings of IEEE CQR 2010. [web | paper]

QoE Visualization

There is not yet a proper visualization tool to map the many-to-one relationship between QoS metrics and QoE, leaving researchers speechless in the cacophony of traditional two-dimensional diagrams. We have proposed that the radar chart, with a few tweaks, is surprisingly suitable for the purpose. In [1,2], we present our adaptation of the radar chart, and demonstrate in a Voice-over-IP context its use in single- and cross-application performance analysis, application recommendation, and network diagnosis.
  1. "Radar Chart: Scanning for Satisfactory QoE in QoS Dimensions," Yu-Chun Chang, Chi-Jui Chang, Kuan-Ta Chen, and Chin-Laung Lei, to appear in IEEE Network. [web | paper]
  2. "Radar Chart: Scanning for High QoE in QoS Dimensions," Yu-Chun Chang, Chi-Jui Chang, Kuan-Ta Chen, and Chin-Laung Lei, Proceedings of IEEE CQR 2010.


Quality of Experience Provisioning

System Auto-reconfiguration in Run Time

It is an extremely challenging control-theoretic problem to automatically adjust system parameters (which are usually highly-dimensional) to ensure optimal user satisfaction at all times. Taking Voice over IP (Internet Telephony) as an example, as in [1,2,3], we have proposed an efficient algorithm based on a statistical regression approach to adjust multi-dimensional system parameters. The evaluation results showed that the proposed algorithm was able to provide users with decent VoIP experience under any condition by adjusting the system parameters intelligently.
  1. "Can Skype be More Satisfying? -- A QoE-Centric Study of the FEC Mechanism in the Internet-Scale VoIP System," Te-Yuan Huang, Po-Jung Wang, Kuan-Ta Chen, and Polly Huang, IEEE Network, March, 2010. [web | paper]
  2. "Tuning the Redundancy Control Algorithm of Skype for User Satisfaction," Te-Yuan Huang, Kuan-Ta Chen, and Polly Huang, Proceedings of IEEE INFOCOM 2009.
  3. "Towards User-Centric Rate Adaptations for VoIP Traffic," Te-Yuan Huang, Chih-Ming Chen, Kuan-Ta Chen, Polly Huang, ACM SIGCOMM 2007 (poster).

VoIP System Configuration Analysis

VoIP playout buffer dimensioning has long been a thorny optimization problem, as the buffer size has to be maintained on the balance between conversational interactivity and speech quality. Although a great deal of research has been devoted to the solutions, whether the results are applied in practice is unclear. Motivated by the question, we originated an investigation on the playout buffer dimensioning algorithms used in three popular VoIP applications: Skype, Google Talk, and MSN Messenger [1]. We found that the buffer sizes of Google Talk and MSN Messenger were not properly adjusted, while the buffer of Skype was not adjusted at all. Moreover, none of the applications adapted their buffer sizes to the network loss rate, which was supposed to be considered as to ensure optimal QoE provisioning.
  1. "An Empirical Evaluation of VoIP Playout Buffer Dimensioning in Skype, Google Talk, and MSN Messenger," Chen-Chi Wu, Kuan-Ta Chen, Chun-Ying Huang, and Chin-Laung Lei, Proceedings of ACM NOSSDAV 2009. [web | paper]

Remote Diagnosis

It is very difficult, if not impossible, to know the load of a particular host on the Internet without cooperation facility at the host. In [49], we proposed a novel approach based on peer-to-peer relaying to measure the load of a certain computer on the Internet. The proposed methodology can be adopted to measure the processing delays at any relay node on the Internet as well as to avoid VoIP quality degradation by measuring and predicting the workload of relay nodes.
  1. "Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes," Kuan-Ta Chen and Jing-Kai Lou, Proceedings of IEEE/IFIP DSN 2008. [web | paper]


Sheng-Wei Chen (also known as Kuan-Ta Chen)
http://www.iis.sinica.edu.tw/~swc 
Last Update April 28, 2017