On Video Games and Missing Data
In the copious amount of free time that ignoring impending deadlines allows me to have, I play my share of video games. Considering the most popular post to date on this blog has been about an online zombie plague, this should come as a surprise to few.
The game I play the most is World of Warcraft, and beyond writing about zombie plagues, it intersects my life in curious ways. Why? Because, when it comes down to it, its a numbers game. I’m sure someone will be upset with me for saying this, but it doesn’t take a tremendous amount of skill to do well in the game. What it does take is a willingness to take apart the game’s mechanics, or more typically, the willingness to read forums and websites made by those who do.
This type of simulation modeling (which is almost all deterministic, I’d kill to see a stochastic model if anyone runs across one) is oft refered to as “Theorycrafting”. For skilled players, a huge amount of decision making is based on these simulation models – how to equip your character with weapons and gear, what combination is best for killing various and sundry creatures and taking their stuff, even what sequence to press buttons in.
Its a fun way to apply what I do for a living to something slightly less serious. Dead dragons instead of sick babies and the like. Recently, it’s started to be used to actively evaluate claims the developers have made about the game – whether the numerical feedback the players are getting (and there is a *tremendous amount*) reflects what the developers have said they’ve implemented. Sometimes it does, and sometimes…it doesn’t. And Blizzard seems to very clearly not like this. A typical response to player driven “So I tested this and my numbers suggest the system doesn’t seem to be working…”:
“A very nice collection of data and good write-up, Vashardjor. Your numbers appear correct and based on these alone we can see how you (and others) may conclude that the system is not working correctly. However, due to the new system being hidden (kind of the idea, really) there are pieces which you couldn’t include in your analysis, and which would likely have led to an entirely different conclusion. ”
This gets to me. While I’ll be the first to admit that there are times when the Theorycraft community makes some unwarranted assumptions (usually they drastically overestimate how “big” a sample they need to make something meaningful), they are usually very good at using validated data pooled over a large number of players.
Epidemiology has problems with missing data every day. Its a big deal. Statistical inference based on incomplete data can be irrepressibly biased, broken beyond belief, and generally worthless. The problem? Most of this data isn’t hidden. The theorycrafters getting brushed off by forum admins don’t have a missing data problem – because all that matters is the player experience.
The players see everything that their character does. And everything their cohorts do. And everything that’s done to them. Anything else Blizzard collects isn’t missing data – it’s irrelevant data. And I’d like people to stop using legitimate (and complicated) statistical outs to dismiss legitimate customers pointing out their changes are only partially working, if that. These people are doing you a massive favor – dedicated, careful, transparent analysis of your system. For free.
I almost wish they’d go back to their old “Working as intended” explanation – while less than helpful, at least it was intellectually honest.
Filed under: General, Soapbox | Leave a Comment