The Halo Effect and RPGs
I recently finished The Halo Effect by Phil Rosenzweig. It’s a business book that talks about some psychologically-difficult-to-avoid errors of perception and analysis that undermine business books, especially books which purport to find simple rules that lead to successful business results. The biggest point he makes is related to the Halo Effect. Basically, he explains that if a certain factor seems like it ought to contribute to an overall result, then there is a strong tendency in humans to have their perception of the overall result also impact their perception of the individual factor (so, for example, if a company is financially successful people will tend to rate it as highly “customer focused”, but if it’s having trouble they won’t, even if there’s no difference in the company’s actual customer interactions). It’s a well-written, interesting, and thought-provoking book.
As is my tendency lately, I immediately started mapping the ideas in the book to the field of roleplaying game design. The most obvious thing here is that many of the factors that would seem to feed into an overall “fun session” are strongly subject to the Halo Effect. Things like “a chance to hang out with friends with the same geeky hobbies”, “fictional content I enjoy”, and “good game mechanics” all seem like they ought to feed into “fun session”, so if you have a fun session you are likely to remember the game mechanics as being good even if they weren’t (I listen to a lot of AP podcasts, and it still amazes me how often people will say that they enjoy a game after I’ve listened to an entire session of them struggling with it). This is yet another reason that designers need to observe objectively when playtesting to be able to identify what truly does or doesn’t work since self-reporting will be strongly influenced by the Halo Effect. (It also means that people with an established fanbase can produce poor games and have them called “good” by their fans, which is more common than I would like the RPG hobby).
I think it also ought to raise some questions about the “playtest at conventions” strategy that some indie designers have committed to over the past few years. Does the high-energy environment of a con cloud useful data? Does the weird microcelebrity “I played with the designer!” thing pollute your results? I’ve never been to a con so I’ve been reluctant to share my thoughts about this stuff, but I’ve never been comfortable with the idea of mixing advocacy and testing, which seems to be a lot of what the “playtest at cons” movement is about.
I can see the value of applying Halo Effects to playtesting, but also the dangers. For example, aren’t “friends with the same hobbies” and “good game mechanics” connected to “fictional content I enjoy”? Can we meaningfully separate those out?
Similarly, why should the high-energy environment of a con cloud “useful data”? Should we only test games in non-fun environments?
There’s a danger, I think, of over-quantifying games and playtesting. They’re complex social process that, probably, we don’t understand well enough to quantify.
Graham, I think we can separate those out as concepts, or at the very least we should be careful about mixing them up with each other, even though they’re not completely independent. The point I was trying to make is that the Halo Effect can make it easy to believe you’re measuring X when you might be measuring Y. A game might seem more fun in a high-energy convention context than in a normal home-play environment, which could lead a designer to misplaced conclusions if they think they’re only measuring the impact of the mechanics on play.
I agree with you that there’s a danger of over-quantifying, but that doesn’t mean we should abandon all efforts to understand complex things (that’s similar to the basic thesis of the book that inspired the original post: that running a business is complicated, and looking at it scientifically is hard, but that doesn’t mean we should be content with not analyzing at all or settling for pseudo-science).