Quasi-Experimental Design in Gaming? Instrumental Variables to the Rescue

By Eunki Min At Happy Elements, our team strives to enhance the user experience through continuous innovation — and sometimes we have to be crafty about it. Experiments and A/B tests are the keys to evaluating the causal effect of new features. But sometimes it’s just not feasible to include fully randomized treatments, either due to cost or for other logistical reasons. When that happens, we turn to causal inference methods and use observational or quasi-experimental data to test out new features in our games For example, there are times when we cannot fully control the determinants of outcome variables other than our treatments. In those cases, our A/B tests or Ordinary Least Squares (OLS) regressions become biased. Fortunately, there is a remedy! If some variables affect the treatment status but do not directly change the outcome variable, we call them Instrumental Variables (IV). By varying the values of the IV, we can create variation in treatment that is independent of the uncontrolled determinants of outcome variables. In these “quasi-experiments,” we use this variation to study the causal effect of the treatment on the outcome.The use of IVs as a tool is widespread in economics and social science, but also in data science. In all of these fields, it’s a pretty common occurrence that we cannot fully randomize a treatment. As much as we’d like to channel our inner Captain Picards and shout “Make it so!”  (come on, somebody out there must still love Star Trek: TNG?), we are at the mercy of the user/subject/participant to opt in to the treatment.  Except that we can still nudge users to take the treatment by providing eligibility or incentives. And if the user ultimately decides to take the treatment, we build these random nudges into our model as instrumental variables to avoid bias, and everyone’s a winner.

To paint a more vivid picture of how this could play out at Happy Elements, imagine a scenario where we are considering making gifts available to users that can help them play our games. We want to find out how (if at all) these gifts will improve our user experience metrics. In this scenario, we cannot control the decision to redeem the gifts; that is up to our users. We can, however, control eligibility–so we use eligibility as our IV. By using these IVs in our estimation model, we are able to optimize the gaming experience with better gift allocation and gift content. Making Happy Element users even happier–and us, too, in the process.