How Sensitive are Online Gamers to Network Quality?

Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei
(To appear in Communications of the ACM, Special Issue on Entertainment Networking - Recreational Use of IP Networks, November, 2006.)
26 April, 2004

PDF Version | Contact Us
Network quality is generally considered critical to real-time, interactive, online game playing. Yet, even though today's best-effort Internet does not provide quality-of-service (QoS) guarantees, online gaming have been becoming increasingly popular. This may be because QoS is not really a "must-have," or because players are simply accustomed to struggling with unfavorable network conditions.
Ironically, despite the growing popularity of online games, complaints about high "ping-times" or "lags" continue to surge in game player forums5. It would appear that most players see network latency and loss as major hindrances to enjoyable gaming experiences. This article investigates whether players are really sensitive to network quality as they claim. If so, we would like to answer: "Do players quit games earlier because of unsatisfactory network conditions?"
A number of studies have evaluated the effect of network quality on online gamers [HB03,[OH03,[Arm03,[NC04,[BCL+04,[ZA04,[YII05,[CHW+06]. In most studies, a series of games is played in a controlled network environment, where the gaming experience of subjects is graded either subjectively or objectively. Subjective tests require participants to report their feelings about playing games with different levels of network QoS. While this approach might capture human perceptions effectively in a controlled situation, it cannot measure how players react to poor game quality in real life, because other factors may outweigh gamers' QoS-intolerance and affect their decisions. Besides, it is very costly for such studies to scale up by including more human subjects. Meanwhile, objective evaluations are usually based on user performance in a specific context, such as the number of kills in shooting games, the time taken to complete each lap in racing games, or the capital accumulated in strategy games. However, game scores are highly dependent on player skills, game design, and content, so the results are not comparable and generalizable across different games.
From a psychological perspective, the very pleasing sensation that players experience in online games is analogous to being in the flow state after taking mood-changing substances like marijuana. A study by Said et al. found that the most significant factors related to the flow experience in online gaming are skill, challenge, and involvement [IML03]. The game playing experience can be described as a pleasurable and exciting activity that makes gamers unaware of the time that passes while they stay in the game. According to the theory, if the feeling of involvement in the virtual world is diminished by network lags, users will be more conscious of the real world and their sense of time distortion will be mitigated. Furthermore, players may simply decide to quit a game as soon as they detect unacceptable lags. Therefore, we conjecture that game playing time is affected, to some extent, by the network quality that users experience.
There are certainly exogenous reasons other than the quality of network conditions that could affect gamers' decisions to continue with a game or leave it. For example, players may leave a game due to their schedule constraints or mental condition. On the other hand, they may be tied by social bonds, no matter how unstable the game play is. Although the behavior of individual players is highly variable and unpredictable, we show that, overall, there is a consistent and strong link between players' departure decision and network QoS.

Network quality vs. game playing time

Our conjecture is validated by real-life traces taken from a commercial MMORPG, ShenZhou Online [UT]. MMORPGs (Massively Multiplayer Online Role-Playing Games), such as EverQuest and World of Warcraft, are networked computer role-playing games in which a large number of players interact with one another in a virtual world. While many genres of games are round- or stage-based, a design that allows players to regain consciousness of the real world, the adventure in role-playing games is continuous and there is no explicit mechanism that forces players to take a break. The uninterrupted sense of immersion in a virtual environment is considered one of the main attractions of MMORPGs because it entices players into a flow experience; the other major attractions are the elaborate reward cycles and the social networks [Yee]. This implies that network quality, a key factor in maintaining a perfect sense of immersion, strongly influences the flow experience of MMORPG players; therefore, the game playing time should be shorter if network conditions that users experience is unsatisfactory.
Another reason for investigating MMORPGs is that they are relatively slow-paced compared to other popular genres, such as first-person shooting games, that usually require players to make split-second decisions. In addition to the pace of a game, there is a great deal of difference in how players control their virtual characters. In fast-action genres, such as shooting games, players must instruct characters "what" actions to take and "how" to perform those actions; for example, to move a character to a new location, a player must control each step it takes (such as three steps west followed by five steps north). On the other hand, in slow-action genres, such as MMORPGs and war strategy games, players only instruct characters "what" to do, i.e., they only need to point out the location the character should move to, and it will automatically head toward the destination via a route that is either pre-determined by game designers or computed on-the-fly. By any standard, MMORPGs are slow-action games that undoubtedly have less stringent service requirements than fast-action games. Therefore, they could be seen as a baseline of real-time interactive online games such that if network QoS frustrates MMORPG players, it should also affect players of other game genres.
ShenZhou Online is a commercial, medium-size, fantasy MMORPG, in which thousands of players are online at any time. To play, participants must pay a monthly subscription fee at a convenience store or online. As is typical of MMORPGs, players can engage in fights with random creatures, train themselves for particular skills, partake in marketplace commerce, or take on a quest. With the help of the ShenZhou Online staff, we monitored all inbound/outbound game traffic at the server side. A total of 15,140 game sessions were recorded over two days. The network performance for each session was extracted based on the sequence number and flags in the TCP packet header, where the network latency was computed by the difference between the acknowledgement and delivery time of non-retransmitted game packets, and the network loss rate was inferred based on the receipt (or not) of TCP acknowledgements. The observed players were spread over thirteen countries and hundreds of autonomous systems, which manifests the heterogeneity of network path characteristics and therefore the generality of this trace. On average, players stayed for 100 minutes after joining a game. This is consistent with the statistics of Korean MMORPGs played in Japan [4Ga], which show that the average session time is between 80 and 120 minutes. However, the differences in individual game playing time are very large; for example, the shortest 20% of sessions span less than 40 minutes, but the top 20% of players spend more than 8 hours continuously in a game.
We use graphical plots to illustrate the difference in game playing time of sessions which experienced different levels of network quality. Fig. 1 depicts the association of game playing time with network latency, network delay variation, and network loss rate, respectively. All three plots indicate that the more serious the network impairment a player experienced, the sooner he/she left the game. The changes in game playing time are surprisingly significant. For instance, gamers who experienced 150 ms latency played for 4 hours on average, but those who experienced 250 ms latency only played for 1 hour on average-a high ratio of 4:1. Moreover, variations in network delay and network loss induce more variation in game playing time (note the range of the y-axis of the graphs).
Now that we have demonstrated that players are not only sensitive, but also reactive, to network quality; however, how to define "good quality" or "bad quality" remains a problem. For example, given two QoS configurations: "low latency with moderate variation" and "moderate latency with low variation," which one is better? To answer this problem, we propose a model that describes the changes in game playing time due to network QoS. This allows us to grade the overall quality of game playing based on a set of network performance metrics, such as latency and loss, in terms of user satisfaction.
cor_mag.png
Figure 1: The relationship between game session time and network QoS factors

Modeling of game playing time

We find that the survival model, which is often used in the medical field to describe the relationship between patients' survival time and the treatment they receive, provides a good fit to describe the reactions of game players to network quality [CHW+06]. The derived model takes network QoS factors as the input, and computes the departure rate of online players as the output. The regression equation is defined as follows:
departure rate
∝ exp(
1.6×network latency+
9.2×network delay variation+
0.2×log(network loss rate)).
(1)
This equation illustrates that the player departure rate is roughly proportional to the exponent of the weighted sum of certain network performance metrics, where the weights reflect the impact of each type of network impairment. The coefficients can be interpreted by the ratio of the departure rates of two sessions. For example, suppose two players, A and B, join a game at the same time and experience similar levels of network loss and delay variations, except that their network latency is 100 ms and 200 ms respectively. The ratio of their respective departure rates can then be computed by exp((0.2 − 0.1)×1.6) ≈ 1.2, where 1.6 is the coefficient of network latency. That is, at every moment during the game, the probability that A will leave the game is 1.2 times the probability that B will leave.
Given the strong relationship between game playing time and network QoS, we can "predict" the former if the latter is known. Forecasting when a player will leave a game could provide useful hints for system performance optimization and resource allocation. To assess the feasibility of forecasting game playing time, in Fig. 2, we compare the actual time and model-predicted time. On the graph, sessions are sorted and grouped by their risks, defined according to Eq. 1, to estimate the level of player intolerance of poor network conditions. We observe that, at the macro level, the prediction is rather close to the actual time observed, which suggests that a service provider can predict how long a player will stay when he/she joins a game and optimize resource allocations accordingly.
eval_mag.png
Figure 2: Actual vs. model-predicted game playing time for session groups sorted by their risks
Please note that while the methodology we presented can be readily applied on all kinds of online games, the exact equation about players' QoS-sensitivity may depend on specific game design, such as the transport protocol used, and vary between different games.

Applications of Player Sensitivity to Network QoS

We have shown that unsatisfactory network conditions discourage users from continuing with a game. Even though we are unable to change the fact that the Internet today is not QoS-guaranteed, which may leave players unsatisfied due to poor network conditions, we can still improve matters by exploiting players' sensitivity to network QoS.
Improving user satisfaction. Given that we can quantify the risk of players leaving a game due to unsatisfactory QoS, systems could be designed to automatically adapt to network quality in real-time in order to improve user satisfaction. For example, we could enhance the smoothness of game playing in high-risk sessions by increasing the packet rate (if the high risks are caused by long propagation delay or random loss on a noisy link rather than transient congestion) or the degree of data redundancy so that the players would be less likely to leave prematurely. Scarce resources, such as the processing power (for handling player requests and computing game states), or the network bandwidth (for dispatching the latest game states), could be allocated more appropriately to sessions based on their risk scores. More specifically, based on the risk scores, resource allocation could be deliberately biased toward high-risk sessions that experience poor network quality, while simultaneously maintaining a reasonable level of player satisfaction in low-risk sessions.
Optimization of network infrastructure. Based on Eq. 1, network quality can be assessed by a single value so that different levels of quality can be compared in terms of user satisfaction. For example, according to our scoring, players prefer a setting of 200 ms latency and 0.1% loss rate rather than a setting of 100 ms latency and 1% loss rate, because packet loss is less tolerable than network latency. As different performance metrics are usually design trade-offs, this score can be used to optimize overall user satisfaction. For instance, as our model indicates that players are less tolerant of large delay variations than high latency, providing a smoothing buffer at the client side, which would incur additional latency but smooth the pace of game playing, would be a plus as it improves the overall gaming experience.
Network troubleshooting. To provide continuous high-quality game services, providers monitor network conditions between servers and customer networks to detect problems in real-time before customers' complaints flood the customer service center. However, considering the large number of online users (hundreds of thousands is not uncommon for popular games), it would be prohibitively expensive, even unpractical, to monitor the status of network paths between all inter-communicated networks in real time. Instead, we can track users' gaming time, which is much more cost-effective. Since online gamers are sensitive to network conditions, a series of unusual (premature) departures within a short period might indicate abnormal network conditions and appropriate remedial action could be triggered automatically.

Conclusion

In recent years, QoS provisioning for real-time interactive applications, such as online gaming, has been actively discussed because of the popularity of such applications and the complexity of the problem. One of the main obstacles to provide satisfactory user experience is that, unlike system-level performance metrics, such as bandwidth or latency, user satisfaction is intangible and unmeasurable. The key to this problem is to measure users' opinions about network performance objectively and efficiently. We have shown that game playing time is strongly related to network QoS and is thus a potential indicator of user satisfaction. Nevertheless, users' perceptions are inevitably influenced by the design and implementation of an application. How to generalize the measure of user experience so that the satisfaction measures for different applications can be normalized and compared with each other remains an open question.

References

[4Ga] 4Gamer.net. Gametrics weekly Korea MMORPG population survey. http://www.4gamer.net/specials/gametrics/gametrics.shtml.
[Arm03] Grenville Armitage. An experimental estimation of latency sensitivity in multiplayer Quake 3. In 11th IEEE International Conference on Networks (ICON), 2003.
[BCL+04] Tom Beigbeder, Rory Coughlan, Corey Lusher, John Plunkett, Emmanuel Agu, and Mark Claypool. The effects of loss and latency on user performance in Unreal Tournament 2003. In Proceedings of NetGames'04, pages 144-151. ACM Press, 2004.
[CHW+06] Kuan-Ta Chen, Polly Huang, Guo-Shiuan Wang, Chun-Ying Huang, and Chin-Laung Lei. On the Sensitivity of Online Game Playing Time to Network QoS. In Proceedings of IEEE INFOCOM'06, Barcelona, Spain, April 2006.
[HB03] Tristan Henderson and Saleem Bhatti. Networked games: a QoS-sensitive application for QoS-insensitive users? In RIPQoS '03: Proceedings of the ACM SIGCOMM Workshop on Revisiting IP QoS, pages 141-147. ACM Press, 2003.
[IML03] Said Ila, Dick Mizerski, and Desmond Lam. Comparing the effect of habit in the online game play of australian and indonesian gamers. In Proceedings of the Australia and New Zealand Marketing Association Conference, 2003.
[NC04] James Nichols and Mark Claypool. The effects of latency on online madden NFL football. In Proceedings of NOSSDAV'04, pages 146-151. ACM Press, 2004.
[OH03] Manuel Oliveira and Tristan Henderson. What online gamers really think of the internet? In Proceedings of NetGames'03, pages 185-193. ACM Press, 2003.
[UT] UserJoy Technology. ShenZhou Online. http://www.ewsoft.com.tw/.
[Yee] Nick Yee. The Daedalus project. http://www.nickyee.com/daedalus.
[YII05] Takahiro Yasui, Yutaka Ishibashi, and Tomohito Ikedo. Influences of network latency and packet loss on consistency in networked racing games. In Proceedings of NetGames'05, pages 1-8. ACM Press, 2005.
[ZA04] Sebastian Zander and Grenville Armitage. Empirically measuring the QoS sensitivity of interactive online game players. In Australian Telecommunications Networks and Applications Conference (ATNAC), 2004.

Footnotes:

1. This work is supported in part by the National Science Council under the Grants No. NSC 94-2218-E-002-074 and NSC 95-3114-P-001-001-Y02, and by the Taiwan Information Security Center (TWISC), National Science Council under the Grants No. NSC 94-3114-P-001-001Y and NSC 94-3114-P-011-001.
2. Kuan-Ta Chen (ktchen@iis.sinica.edu.tw) is an assistant research fellow in the Institute of Information Science at Academia Sinica, Taipei, Taiwan.
3. Polly Huang (phuang@cc.ee.ntu.edu.tw) is an associate professor in the Department of Electrical Engineering and Director of the Network and Systems Laboratory at National Taiwan University, Taipei, Taiwan.
4. Chin-Laung Lei (lei@cc.ee.ntu.edu.tw) is a professor in the Department of Electrical Engineering and Director of the Distributed Computing and Network Security Laboratory at National Taiwan University, Taipei, Taiwan.
5. http://www.gamedev.net/community/forums/


Sheng-Wei Chen (also known as Kuan-Ta Chen)
http://www.iis.sinica.edu.tw/~swc 
Last Update September 28, 2019