Designers provide researchers with clear hypotheses, ready to be tested. In two distinct publications (Schuh et al. 2008; Romero 2008), the Microsoft Game Studios user research team detailed how designer intents were tested in the “Time Trials” game mode of the racing title Forza Motorsport 2. The intent was that that particular game mode had to be challenging but not overly frustrating. From the outset the designers themselves proposed a clear hypothesis to test: “approximately 80% of the target users should be able to complete any particular time trial (…) after ten laps” (Schuh et al. 2008; p. 252). User researchers invited players from outside the team and tested all of the 25 challenges included in the game. They measured the percentage of players who actually passed the challenge. Results made it clear that the designers’ intents were far too optimistic. For example, Time Trial number 3 on the track “Tsukuba Short” had 0% of players succeeding: the target time of 45.7 seconds was too low. After analyzing the lap times of the real players, researchers suggested raising the target lap time from 45.7 to 50.2 seconds. In the end, the game design team decided to set the target time at 48.8: they knew the cars used by playtesters could be upgraded in the game and thus became faster. This is a great example of how the feedback from user researchers can complete and improve the work from the design team.
Schuh, E., Gunn, D. V., Phillips, B., Pagulayan, R. J., Kim, J. H., & Wixon, D. (2008). TRUE Instrumentation: Tracking real-time user experience in games. In K. Isbister & N. Schaffer (Eds.), Game Usability: Advice from the experts for advancing the player experience. San Francisco, CA, USA: Morgan Kaufmann.
Romero, R. (2008). Successful instrumentation: Tracking attitudes and behaviors to improve games. In Proceedings of Game Developers Conference GDC 2008. San Jose, CA, USA. Retrieved here (Audio Track)