Designing, RPG

Some RPGs are more popular than others, and it can be difficult to figure out why. This is not unusual, many “culture industry” products work this way: some movies are hits and others are flops, some songs top the charts and others are ignored, some books hit the best-seller lists and others are never read. It’s reasonable to want an audience for the things you create, but it is difficult to engineer a hit. One notable factor is that social influence plays a role: we humans can use the opinions of the people around us to help us identify good things, since personally investigating each and every song, book, movie, RPG, etc., is an intractable problem.

Recognizing that this is a pattern in multiple contexts, some social scientists have attempted to study it. In one study, the experimenters set up a website in which subjects could listen to, rate, and download songs. However, they arranged the experiment so that different subjects ended up in different “worlds” from each other. In the control group the participants had no idea how well-liked the songs were, but in other worlds people were able to see the “charts” that indicated how good/popular the songs were with their peers. One thing you might expect is that some songs are just better than others, so the best songs would tend to bubble up to the top in a Darwinian fashion. Another thing you might expect is that, due to social influence, a “rich get richer” effect would come into play as people gravitated towards songs that were lucky enough to be identified as good early on. Running in parallel like this let the experimenters compare the how the “worlds” developed. As their abstract summarizes:

Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.

What I think we should take away from this is that there may be things that are in our control and things that are not in our control when we create things that will compete in a cultural market. The quality is under our control: the best songs rare did poorly, and the worst rarely did well. But any other result was possible because in a chaotic social environment we have little ability to predict with certainty how well something will be received: not every flap of a butterfly’s wings causes a hurricane. When we’re designing an RPG we can control things like whether the rules are fully explained, whether we’ve included proper examples, whether the information is presented concisely, and whether people have fun when they play the game properly. We’re far less able to control whether the game will hit the zeitgeist and have whatever combination of features make something the “new hotness”.

Now, some might argue that even though it’s difficult you need to craft your product so that it appeals to an audience. And perhaps you can use testing to see what does or doesn’t appeal. The problem with this approach is that it’s very difficult to extrapolate from thin data. Anyone who follows politics can tell you that asking around among your friends is a terribly unreliable way to judge how the population as a whole feels about an issue. Statisticians and pollsters are very attuned to the issues of sample size and nonrandom samples. The field of statistics is built upon the idea of making inferences about the world from limited data. There are mathematical rules that can tell you how confident you should be that the results of querying a random sampling of a particular size would generalize to the overall population. On the one hand this can be pretty impressive, political pollsters are able to make reasonable predictions about the entire voting population of a country as big as the United States from samples of hundreds or a few thousand. But even though that’s a small number when compared to the population of a country, it’s a big number when compared to the customer reach of independently-published roleplaying games. Furthermore, size isn’t the only thing that matters: it’s common for political bloggers to have “web polls” of their readership but it would be foolish to extrapolate from them no matter how many people responded because the tools of statistical inference only work from a random sample. If your sample is biased it is incredibly difficult to extrapolate from it. So, if you’ve floated your RPG but haven’t gotten a positive response from an audience, should you conclude that no one will like it? No one will stop you from extrapolating from a small, biased sample if you want to, but that’s generally not the way that data-driven decisions are supposed to work (of course there’s also the question of effect size: extremely strong reactions may be easier to detect with smaller samples). Measuring your success at finding playtesters for your game is not a very reliable tool for figuring out if people will want to play a game, but if actual playtesting is done well it can be an excellent too for determining if the game is fun when people do play it.

But the product itself isn’t the only thing that you’re creating. What about things like marketing campaigns, establishing brands or reputations, or building notoriety in related fields? Interestingly, some of the same authors from the study I described above ran a very similar experiment in order to observe what would happen when subjects were exposed to intentionally manipulated market information. What they discovered is that there was indeed a “self-fulfilling prophecy” effect of songs that were presented as popular continuing to be popular (and similarly for those presented as unpopular to remain unpopular), but the effect wasn’t overwhelming. Some element of the songs’ intrinsic properties must have an impact, because the artificially popular/unpopular songs didn’t seem to match the trajectories in conditions where the songs achieved that status “naturally”.

Interestingly, they observed an overall effect on the manipulated markets:

A final and unexpected consequence of the inversion was a substantial reduction in the overall number of downloads. … [S]ubjects in all social influence worlds tended to listen to the songs that they thought were more popular. In the inverted worlds, however, the songs that appeared to be more popular tended to be of lower appeal; thus, subjects in the inverted world were more exposed to lower appeal songs. For example, in the unchanged world, the 10 highest appeal songs had about twice as many listens as the 10 lowest appeal songs, but in the inverted worlds this pattern was reversed with the 10 lowest appeal songs having twice as many listens. As a consequence, subjects in the inverted worlds left the experiment after listening to fewer songs and were less likely to download the songs to which they did listen. … Together, these effects led to a substantial reduction in downloads: 2,197 and 2,160 in the inverted worlds, compared with 2,898 in the unchanged world.

The combination of increased success for some individual songs … on the one hand, and decreasing overall downloads, on the other hand, suggests that the choice to manipulate market information may resemble a social dilemma, familiar in studies of public goods and common-pool resources … but less evident in market-oriented behavior. [Some of this data] suggests that any individual band could expect to benefit by artificially inflating their perceived popularity, regardless of their true appeal or the strategies of the other bands; thus all bands have a rational incentive to manipulate information. When too many bands employ this strategy, however, the correlation between apparent popularity and appeal is lowered, leading to the unintended consequence of the market as a whole contracting, thereby causing all bands to suffer collectively.

Obviously there are differences between the experimental conditions and real-world markets, but it does suggest that the strategy of intentionally “hyping up” your own product may seem smart in a zero-sum world but can contribute to a “tragedy of the commons” when everybody commits to that strategy. It seems to me that the wiser course would be for everyone to make the best products they can and try to operate in as fair a market as possible so they are most likely to get the results that they deserve (including players getting to play the best games). To the extent that our markets are imperfect, I believe it’s most important to care about removing barriers to entry so that potentially-good games get at least one bite at the apple, and we don’t need to worry so much about “promoting” games/people who are already getting a chance to let their games speak for themselves.