So it all began with a debate that Norman, Mash and I had one Wednesday. We were discussing the various issues and rumours about
Kapcon and it's prize giving. Kapcon is a highly regarded and very enjoyable roleplaying convention held here in Wellington every January. It has quite a reasonable turn-out each year and a very high calibre of games run.
The issue that has plagued it over the years has been the prizegiving. Some people mutter that the awards tend to be a popularity contest where many winners get their awards due to friends being involved in their games and thus marking them high. Now I'm not entirely convinced that this is actually the case, but since I started statistics and research methods, I have noticed something worrying - the current scoring system has some very serious flaws in it that could lead to results being more like a random drawing from a hat than any measure of ability or success.
Further, I began to suspect that it is possible to "rig" your chances of success because of the vagueness of the system.
This was concerning. Add to this the increasing popularity of shared narrative games like Primetime Adventures, Lucky Jones' and Badass Space Marines - which have built into them certain aspects that suggest an advantage in such scoring system as Kapcons... there was reason for us to discuss possible solutions.
So the topic was broached on
NZRAG with much debate and a small degree of snarkiness.
Over the course of this discussion I have given a lot of thought about my initial concerns and where I may have made some slip ups in my analysis. I've come to the following thoughts about the system, shared narrative games and fun in general.
Fun = Engagement
The most major assertion about this whole debate was that Kapcon and the prizegiving was about fun. Now I have a bit of an issue with measuring fun because it really isn't an objective thing. Fun in and of itself is highly subjective - this is because it is born from a variety of variables that are combined in unique ways to every person. So measuring fun itself is almost an exercise in futility.
We can't really measure what is fun, but we can measure when people are having fun. This is different because we're not measuring the degree of fun itself, we are measuring the frequency of a positive response. I have chosen to call this engagement. Why? Because in my experience most people report that they have had fun when they have been engaged in something.
So to turn this to roleplaying, a fun game is one that engages the players in some way - be it as a player, character or audience to the game. Now I'm not about to go into the scoring aspect of this yet, I want to first explore the idea of engagement...
Think about the last fun game you played in - what was it that you were doing when you were enjoying the game? I feel pretty confident in saying that you were paying attention to what was going on and interacting in some way with the game as it played. Be that laughing at a funny moment, being scared when something horrible was happening, experiencing sorrow as your character... in some way you were drawn into the game.
A Game Master's job is to ensure that engagement above all other things, I feel. In most games it is through a variety of tricks and techniques that you become engaged in the game - and a good GM knows how to keep the players focused on the game, both interacting with it and enjoying the experience.
How games deal with engagement
So getting to the crux of the initial debate, I've noticed a considerable difference between shared narrative games and traditional roleplaying games when it comes to engagement and the GM. See, the thing about games like PTA is that they have engagement built into the system - players are made to interact with the game as a whole from the set up of the setting to the running of the storyline, even when their characters are not in a scene.
Essentially, the GM doesn't need to work as hard to get the players interacting with the game because the system is already doing this and sharing a lot of the load. Thus, it is reasonable to assume that shared narrative games are more likely to produce a fun experience. This is because the players are more involved in the game and have a vested interest in staying engaged with it.
Now this isn't to say that shared narrative games are
better - on the contrary, there is also a factor where disagreements between players arise and the GM must facilitate them. This can then cause players to disengage from the game, because their input is not involved.
However, in the issue of "fun" - the engagement in the game is coming from the players rather than the GM. The good thing about this is that it allows the GM to then no longer worry about keeping the players interested, instead s/he is able to enjoy the game in much the same way as the players, only having to facilitate the occasional conflict of personalities that may arise.
These games are
designed with the idea of lightening the GMs load so that they can get on with the game.
In a traditional game, a GM is required to be much more skilled at engaging the players - now systems in traditional games have their own ways of handling this, but they tend to provide tools for the GM to use rather than just cutting past the GM.
As the old saying goes - a good GM can make any game fun.
Scoring Values, Kapcon and Variables
So what is the issue with Kapcon and it's scoring system. Well, the issue is based around the "Best GM" scoring method. In Kapcon there are two methods used for scoring - one is a very simple nomination system where the most nominations are used - attendees of the convention simply state who they are nominating and why. These are then seperated out into a number of categories and counted up - most nominations winning. Simple and not really requiring much in the way of statistical knowledge.
However the GM voting uses a scoring system that has players state the game that they played and give it a ranking taken from a range of 1 through to 7.
Now the Kapcon committee hasn't released exactly how they then calculate these values - but the assumed method is to take the average of all scores. Whoever scores 7 at the end of it all, with the most votes, wins.
How ties are decided remains somewhat of a mystery too - but one would assume that number of votes and sessions would come into play.
So what is the problem with this system? Well from a survey/scoring viewpoint it has a very glaring and serious flaw - what do the values mean?
The official line of Kapcon is that individuals are invited to place whatever value they want to the numbers, with the understanding the high is good and low is bad.
In common research scales this is often done. No doubt many of you have done a test where you had 1= strongly disagree and 7= strongly agree.
The problem is that you can't base it off one question, and you'll notice in all those tests that you tend to be asked a number of questions that are used to establish a pattern, so that the staticians looking at the data know how you are valuing each number.
Furthermore, you are given clear indication of what each number represents.
Random Variables
The issue facing Kapcon is that no value or scale is actually given. Players are left to decide what each number represents. This is a problem because different people will interpret the numbers in very different ways. This means that at the statistical end of things you are getting results back that are based on wildly different values - meaning that two ranks of 6 wont necessarily mean the same thing.
So, in turn, you are getting random variables disrupting your data set with each game you take scores for. Because much of the placing in a game is done with the drawing of names from a hat, this is a further randomisation of how the scoring is going out. Meaning that the results have even less to do with anything happening in the game - because the values are getting constantly mixed and disrupted.
It is somewhat naive to assume that this randomisation evens out the variance caused by players using different values.
To better clarify this - consider player X and Player Y. X is a bit critical, and decides that a rank of "7" is unreachable. Basically the GM will have to blow X's socks off and bring about the second coming to score a 7. However, 1 is very possible because some games can blow so much that X will quite readily give a 1. So this means that the actual range of scores that X will give could be all over the place. Y, on the other hand, is a real softie and has decided that everyone is trying their best. So Y is willing to give a 7 to games that are enjoyable and engaging, a 6 to those that try hard and Y manages to have a good laugh in, and a 5 to anything that wasn't that grand. This means that Y is really marking out of a range of 3, not 7. It also means that any GM lucky to have Y will be regularly scoring high regardless of ability or game.
Now I have met people like X and Y at Kapcon, so it is very reasonable to consider these examples.
So what you have happening here is Kapcon ranking GMs in a particular set of values that are not being shared by the attendees. Essentially, X's 6 and Y's 7 have the same "fun" value to these attendees - however Kapcon's scoring system will treat Y's 7 as more valuable.
Given that each game has only 6 people involved - a game that ranks low but scores Y's 5 will be unnaturally knocked up the rankings. It gets more concerning at the upper end of the scale. Consider two GMs who get high rankings - the GM with X in their game is going to be disadvantaged.
So before we even consider the fun issue, we have a problem with the scoring.
Why there is an issue
Bringing it all together now, the issue that arises is that attendees are being asked to vote which games were the most fun for them. This essentially means that the more engaging a game is, the more likely it will score high. The assumption behind this is that a good GM will produce a fun game. The better the GM, the more people in the game will have fun.
This is a reasonable assumption. After all, a good GM will be able to make someone who doesn't normally enjoy his chosen system to still have a good time and rank it highly. Right?
Except that there is no fixed scale - so the player who doesn't normally enjoy said system is potentially still going to rank the game lower due to system issue and a lack of engagement.
Now consider the shared narrative issue - players are more likely to rank a shared narrative game as fun simply because the system engages them and is more likely to produce a vested interest in enjoying the game. So while a good GM will produce an excellent gaming experience, a moderately skilled GM with an excellent group of players also will rank high as well.
What this means is a GM who choses to use a traditional game and works that much harder to engage the players so that they enjoy the game just as much is automatically disadvantaged by choice of system - which shouldn't be allowed to affect the GM's ranking.
So what is the solution?
Possible solutions for competitive gaming
Well there are number. Firstly, I propose that we need to keep from seperating or banning games. System should not be an issue. So it comes down to producing a scoring system that best accounts for the imbalance in games and focuses on how well a GM is getting the players to engage in the game. After all, if fun is the point of the scoring- then we need to be able to effectively identify when it is the GM's skill that is producing the fun in the game as opposed to luck of the draw with players.
Now we can't do this perfectly, obviously, but there are a number of possible ways to better lessen the chance of a random win.
Firstly, a fixed scale. It's the first rule of building a scale such as Kapcon uses, and it really ought to have been instigated from the get go. It clearly lays out to the attendees how the counters are interpreting the numbers and lessens the X and Y players issue. It wont perfectly, fix it - there is still a chance that X will rank lower and Y will rank higher - but it does protect the convention from accusations of a faulty system and puts the responsibility clearly and unarguably in the attendees hands. If they truly enjoyed a game ranked it as "very good" and not "excellent" then they have no reason to gripe if that GM doesn't win.
Secondly, adjust the scoring to better identify GM skill and remove the player variable. There are a number of ways of doing this. One way is to simply ask "Did the GM achieve the following?" asking players to tick each correct answer, totalling the ticks to get a score. Another is to have a scale of numbers with 1 = I wasn't interested to 7 = the GM got everyone involved and enjoying the game.
Another solution that doesn't remove the player variable, but still provides a clearer idea of how to score while better dealing with the X Y issue is to rank the game in words, applying a numerical value secretly that only the organisers need to use to identify a winner. This would be simply: Was the game - Poor (numerical value = 1) Average (numerical value = 2) Good (numerical value =3) Excellent (numerical value =4). Then you can have a commentary section below or a second "tie-breaker" question of "mark off the words that represent things the GM did to make the game fun..."
Ideally it is a combination of these systems that can produce a very simple, quick to fill in, sheet that is both fair and a more accurate scoring system.
As of this date it is unlikely that Kapcon will change its system dramatically, although there has been comment of making the scale have fixed values - which is a good first step. However I suspect it wont resolve all the issues because it doesn't really account for the XYplayer issue or the imbalance of how a GM is engaging the players versus the players producing their own fun and scoring a game high because of their work rather than the GMs.
Love and Huggles
Conan
Currently Reading: Nothing at the moment...
Currently Playing: Unknown Armies - To Go; Mage: The Awakening - Threshold
Mood: Taking on unfairness with reasonable force!
