Posted by sigma on August 30th, 2009 |
4 commentsI’ve been searching for a good reference article on testing methodologies and how to ensure a test is a good discriminator. I haven’t had much luck. I know that this discipline exists but I’m not sure what it’s called – “testing theory” perhaps?
Let me start by sucking the fun right out of pinball and describing each game as a test. It’s a really enjoyable test, most of the time, but it’s testing your skills and giving you a score, which you can then compare against others on the same machine under the same conditions. A machine is a good discriminator of skill if it rewards a variety of approaches to play, has balanced scoring, and is neither too hard nor too easy.
It’s fairly obvious that if a machine has one easy shot that is worth more than anything else, then play on that machine becomes an exercise in repeating that shot – more a test of endurance than skill. At PAPA 6, the Video Mode on Junkyard comes to mind, as does Riverboat Gambler at Pinball Expo. But what’s less obvious is the “too hard / too easy” requirements.
Imagine that the entrance exam to the world’s most prestigious college is insanely difficult, and for the purposes of discussion, imagine that it’s all multiple-choice questions. If the test is so difficult that a bright, qualified applicant might only know 5% of the answers, but there are 5 choices for each question, then simple guessing will tend to yield a score of 20%, making the 5% of actual knowledge much harder to discern in the results. In fact, you’ll be admitting a lot of lucky people who may actually be morons.
In pinball, the issue is luck versus skill. If the game is so hard that nobody can keep the ball in play, then the skill factor is replaced by luck – whoever gets the lucky bounce will get the most points. Conversely, if a game is too easy, you’re back to the endurance issue (and your tournament – oh, that’s right, we run a tournament and that’s why I’m talking here – won’t run very efficiently).
Obviously, this is some of the reasoning behind game selection and setup in each division at PAPA. Sometimes we might make a game a little too hard in A, but that doesn’t hurt too much – players can pick their machines, and decide if they want to take a chance (higher luck factor) on an unusually difficult machine.
