The question for day 21 of #AprilTTRPGMaker is “How many playtests?” This is a big question, and as I've said before the topic of playtesting is near and dear to my heart.
My position is that it's unethical for a game designer to release an unplaytested game to the public. This is just a specific case of my general view that nobody should release untested products. Obviously that's a much bigger deal in a field like pharmaceuticals where problems in your product could actually kill people, but I think an ethical game designer should test their product before release to be sure that it does what reasonable people generally expect a game to do.
The fact is that many people have released unplaytested games. My view contains the implicit criticism that people shouldn't have done that, which is something those who have done it don't like to hear. I believe I've pissed off several people who have worked as freelance designers for mainstream publishers and some prominent indies by holding this position in social media conversations. Nevertheless, I think it's more important to hold views that are right than views that are popular. Obviously it's possible to be lucky and have your untested product work fine, but I still think that's a bad practice. I've also experienced people assuming that I hold this view in hypocritical bad faith and that I only express it to “gatekeep” people from getting started. Nope, as already mentioned I hold this view even for people who have been “successful”, and more importantly my views are obviously not binding on anyone except me, so no one is more “held back” by my view than I am. There's not a huge amount of harm caused by releasing untested games, so even if I had the power to do something like “drum people out of the industry” for doing it I probably wouldn't, but I think the world would be a better place if proper playtesting was a norm that everyone followed. I think it's probably in everyone's best interests to engage in good pre-release testing, anyway – the games will be better for it which is good for designers and good for players.
There's a potential trap with testing that it can always seem like more is better, so you can get yourself into a pattern where you always want to do more. If you do that it pushes your release off to infinity. I don't think that's wise. If your playtesting practice also includes an element of social engagement, such as being able to show off your game at game conventions, it might be even easier to fall into this trap. I think the key to remember is that playtesting is done for a purpose, which is generally to get all the problems in a game identified so they can be ironed out and it can be released (or, occasionally, to discover that a game is so fundamentally flawed that it should be abandoned).
I think this is hard question to answer, and “how many” may not be the ideal metric, even though that's the most easily measured. First of all, if you start counting from zero you're not taking into account how much the game needs to change based on information gained via testing. If you need to completely revamp large systems you're probably going to need more subsequent tests than games that were lucky enough to have no major problems. In my experience in testing microprocessors we tended to assume that there would be diminishing returns on our bug-finding efforts – you tend to find the easy-to-find stuff first (in an RPG context, maybe something like “the rules said to roll for X, but you didn't specify what kind of dice to roll”) and then more subtle and hard-to-find stuff later.
To some degree you can use that as a basis for determining when you're done: if you've been doing good testing and haven't needed to make any revisions to the game for a while it's probably ready for release. Because I haven't gotten to a stage where I feel I've had adequate external testing with my games I can't say with certainty, but I think that if two different independent groups were able to play one of my games to completion without problems I'd probably say it's ready for release.
In my microprocessor validation job we had a concept of “coverage” to track how well our testing was covering the important interactions in the device-under-test. Basically you need to apply some engineering judgment to the thing you're testing to understand what sort of factors need to be varied. For example, most games let players choose whatever names they want for their characters, but usually this doesn't have any meaningful impact on play, so you probably don't need to worry that you've tested a broad variety of character names. On the other hand, if you've got a class-based system where the players need to operate as a combat team then you probably do want to make sure that you've tested with parties that have several different combinations of character classes because that often has a big impact on how a game plays at the table. If you run a million tests but each one only exercises 5% of your design then you've still got 95% of the design left to test. If your game is straightforward enough that there's not a lot of variance needed to exercise the full scope of possibilities then you may not need many tests.
(From Kira Magrann's twitter)
