Modern Game Testing E-Book

Nikolina Finska

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 415

Veröffentlichungsjahr: 2023

Beschreibung

Few things are more annoying for gamers than encountering a buggy new game. This often leads to negative reviews, and in turn, you’ll find that demand for your games declines. The solution lies in better quality assurance (QA) – and Modern Game Testing will show you how to achieve just that. Whether you’re a new tester, developer or producer, the QA testing techniques shown in this book, using modern methodologies and the latest technology, will have you releasing quality games that are on time and, most importantly, on budget.
The book begins by introducing you to QA and the various types of tests that are performed on games. You’ll then explore test cases and bug reporting, building tests for different platforms (even consoles and PCs), and LiveOps and test management. As you advance, you’ll build a QA team from scratch and work with remote QA testers. The chapters help you take a more traditional approach to learning lessons, enabling you to examine the modern agile approach and various testing strategies that you can then adopt. All angles are covered with oodles of examples, so you’ll have everything you need to implement QA strategies in your organization.
By the end of this book, you’ll have a clear understanding of the modern methodologies of QA testing for games, and be able to build efficient, reliable, and long-lasting QA teams.

Leseprobe

Modern Game Testing

Learn how to test games like a pro, optimize testing effort, and skyrocket your QA career

Nikolina Finska

BIRMINGHAM—MUMBAI

Modern Game Testing

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Rohit Rajkumar

Publishing Product Manager: Vaideeshwari Muralikrishnan

Senior Content Development Editor: Feza Shaikh

Technical Editor: Joseph Aloocaran

Copy Editor: Safis Editing

Project Coordinator: Aishwarya Mohan

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Vijay Kamble

Marketing Coordinator: Anamika Singh, Namita Velgekar, and Nivedita Pandey

First published: July 2023

Production reference: 1220623

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80324-440-2

www.packtpub.com

This book is dedicated to the late Milan Rusmir, an exceptional engineer and extended family member. During my formative years, Milan always found time to support and nurture my curiosity and entertain a young girl’s inquisitive mind. He influenced my interest in computers and games from an early age, shaping the path that led me to the fulfilling career I’m so proud of today. Milan’s kindness and encouragement were a big source of inspiration, showing me the transformative power of mentorship and the importance of fostering curiosity in others.

-Nikolina Finska

Contributors

About the author

Nikolina Finska is a gaming industry veteran and QA guru, with over 10 years of expertise in the game sector and another 10 in the software industry. She was nominated for the Tester of the Year award by the Finnish Testing Association for her contributions to QA methods.

Nikolina worked on over 20 games and oversaw QA on various Angry Birds franchise games. She has done practically every type of QA in the past, from localization to live OPS, on a variety of platforms, including massive databases, the web, and every imaginable gaming platform. Nikolina is a popular guest lecturer on QA and games production at Finnish universities. Besides QA, Nikolina teaches game production, entrepreneurship, and general leadership.

I want to thank my spouse, Jorma, for his support, advice, and encouragement while writing this book. Big thanks go to my late dogs, Kara and Tron, who were always by my side and made long writing hours easier to bear.

About the reviewer

Juha Pomppu is a process and software quality professional with so many hats – QA lead, auditor, trainer, coach, mentor, community builder, conference manager, and team leader.

He started his IT life with home computers, coding low-level demos with the MOS 6510 and Motorola 680x0 assembly languages. He doesn’t code anymore, but this hobby gave him a deep understanding of how computers work.

Today, his professional passion is in quality management systems, modern software development processes, and software quality assurance – and, of course, people and teams, because together we can create systems with high quality.

Table of Contents

Preface

Part 1: Game Testing Foundation

1

Setting the Stage – Introduction to QA for Modern Games

Understanding the evolution of modern game testing

Exploring the differences between software and game testing

Why is QA important for games, especially within the agile process?

When and how should QA testing for games be performed?

When should QA for games be performed?

How should QA testing for games be performed?

Summary

2

All Engines Go – The Basics of Game QA

What is tested in games?

Stability

First-time user experience (FTUE)

Core game loops

Level progression

Game physics

Game logic

AI behavior

Usability flow (UX)

What is the most important thing to test?

Game QA in practice

Game QA challenges

Frequent changes in technology

Changes in platform guidelines and regulations

Conflicting priorities

Limited time for QA

Considered an entry-level role but with lots of requirements

Working with the development team

Live bugs

Agile practices and game QA

Scrum

Kanban

Summary

3

A Deeper Look – Types of Testing in Games

Functional testing

How do we carry out functional testing?

Compliance testing

Mobile (iOS and Google Play)

Gaming consoles (PlayStation, Nintendo, and Xbox)

PC and Mac

Localization QA

Different levels of localization

Things we look out for during localization QA

Regression testing

Differences between regression and functional testing

Different approaches to regression testing

Other types of testing

Basic acceptance testing/acceptance testing/smoke testing

Stress testing/load testing

Playtesting

Ad hoc testing

Beta testing

Summary

4

Deeper Look – Testing on Various Gaming Platforms – Mobile, PC, and Console

Platform relevance

Testing on Google Play

Testing on the Apple App Store

Testing on other mobile platforms

Testing on consoles

Gameplay testing

Achievements

Legal compliance

Specifics of console testing

Testing on PC and other platforms

Summary

5

It Must Be Hardware: Testing Hardware in Modern Game QA

Is hardware important in modern game QA?

Test sets – how to build one

Hardware testing beyond mobile

Console hardware testing

PC hardware testing

Summary

Part 2: Test Strategy and Execution

6

Friend or Foe – Test Cases

What are test cases, and do we need them?

How to write great test cases

Test case alternatives

Test scenarios

Use cases

Test charter

Summary

7

It Works on My Machine: Bug Flow

The importance of bug flow in game teams

How to set up a good bug flow

Bug flow statuses and transitions

Summary

8

I Thought I Fixed That: How to Write Efficient Bug Reports

Why bug reports matter

How to write excellent bug reports

Headline or title

Description

Screenshot/video or similar

Version tested

Reproduction rate

Platform and OS version

Comment

Assign to

Severity versus priority

Severity

Priority

Bug reporting best practices

Reliability

Objectivity

Clarity

Timeliness

Bug examples

Summary

9

It Works, but It Hasn’t Been Tested: Testing Approach

Lessons learned from the waterfall model

Agile approach – embedded QA

How to pick the right testing focus?

Testing strategies in live ops

Types of testing strategies

Summary

10

Eat, Sleep, Test, Repeat: Test Methodology

Risk-based testing

Risk identification

Risk analysis

Risk prioritization

Developing a test strategy based on risks

Monitoring and managing risks

Exploratory testing

Equivalence partitioning and boundary value analysis

Decision tables

Strategies for dealing with new code

Summary

Part 3: Test Management and Beyond

11

Are You on the Right Version? Live Ops and QA

The difference between dev and live ops

How to test new features

Testing new content

New features

Dealing with submissions

Backend pushes

Working with live bugs

Summary

12

Beyond Testing – Introduction to Test Management

The test management role

A test plan

Methods to help you estimate testing efforts

The Delphi technique

Work Breakdown Structure (WBS)

3-point estimation

A functional point measure

Test planning and execution coverage

Studio policies and ways of working

The available budgets and resources

The perceived importance of the game we are working on

Existing contracts and other contractual dependencies

The development methodology

The game’s complexity and genre

The perceived importance of QA in the team

How to plan test coverage

Summary

13

There Are No BUGS Without U – QA and the Game Team

Building QA teams

Working with remote QA teams

Test outsourcing process

Measuring outsourcing effectiveness

A career in game QA

Automated testing

The future of game testing

Summary

Index

Other Books You May Enjoy

Preface

It is surprising how difficult it is to find relevant information about how modern game testing works. While we can find lots of information about how to test more traditional, premium games, somehow, more detailed guidelines into modern games QA, especially on mobile and in live ops, are almost impossible to find. My goal in writing this book was to give you a deep insight not only into how modern games are tested today, but also into how to work efficiently with agile methodologies, flat team organization, and the unique challenges of free-to-play games. We will go into great detail about the best QA practices that will ensure your games are high quality, on budget, and released on time. While we will briefly discuss automation testing, please note that the focus of this book is primarily on manual testing. Modern Games Testing was written with the intent to provide plenty of practical examples, rooted in personal experience, helping you get an inside look at the fascinating world of games QA.

Who this book is for

This book is aimed primarily at game testers but also game producers, game developers, testing managers, and other QA professionals who want to learn more about modern approaches to game QA and use them to build more efficient and cost-effective QA teams and products. It is desirable that you have prior professional testing experience, either in software or games testing, and/or experience working in the gaming industry. Basic familiarity with agile working practices such as scrum is needed to fully understand all concepts explained in the book. A basic understanding of the gaming industry ecosystem will help you understand the covered topics in more depth.

What this book covers

Chapter 1, Setting the Stage – Introduction to QA for Modern Games, discusses the importance of QA and the main differences between games and software QA.

Chapter 2, All Engines Go – The Basics of Game QA, examines what we test in games and the main challenges of games QA.

Chapter 3, A Deeper Look – Types of Testing in Games, explores the different types of testing in games and how to execute them.

Chapter 4, Deeper Look – Testing on Various Gaming Platforms – Mobile, PC, and Console, delves into the specifics of QA on different gaming platforms.

Chapter 5, It Must Be Hardware: Testing Hardware in Modern Games QA, covers the importance of hardware and how to create optimal test sets.

Chapter 6, Friend or Foe – Test Cases, discusses how to write great test cases and their alternatives.

Chapter 7, It Works on My Machine: Bug Flow, explores how to set up efficient bug flow and statuses.

Chapter 8, I Thought I Fixed That: How to Write Efficient Bug Reports, examines in detail how to write great bug reports.

Chapter 9, It Works, but It Hasn’t Been Tested: Testing Approach, delves into agile methodology and its approach to testing.

Chapter 10, Eat, Sleep, Test, Repeat: Test Methodology, covers the most commonly used QA methodologies with practical examples.

Chapter 11, Are You on the Right Version? Live Ops and QA, examines the details of how QA works in live ops.

Chapter 12, Beyond Testing – Introduction to Test Management, explores in detail how to make a test plan and efficient testing estimation.

Chapter 13, There Are No BUGS without U – QA and the Game Team, gets you familiar with a career in QA and explores the future of QA.

To get the most out of this book

Please get yourself familiar with the basics of agile methodology. Familiarity with games business models and genres will help you understand the material in more depth.

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/LG6tq.

Conventions used

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles.

There are a number of text conventions used throughout this book.

Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "We can also see that the bug has a Repro rate value of 10/10. That affected the Priority value, which is set to Highest.".

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Modern Game Testing, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803244402

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Game Testing Foundation

In this part of the book, we will get familiar with the basics of games QA, including the different types of testing, working with gaming platforms, and how to utilize hardware in game testing.

This part has the following chapters:

Chapter 1, Setting the Stage – Introduction to QA for Modern GamesChapter 2, All Engines Go – The Basics of Game QAChapter 3, A Deeper Look – Types of Testing in GamesChapter 4, Deeper Look – Testing on Various Gaming Platforms – Mobile, PC, and ConsoleChapter 5, It Must Be Hardware: Testing Hardware in Modern Games QA

1

Setting the Stage – Introduction to QA for Modern Games

At its core, quality assurance (QA) in game development isn’t much different from QA in other types of software. However, there are some QA testing aspects that are specific to games.

But, let’s first start by introducing how QA is done in modern games, when, and by whom? How is it organized in this extremely fast-paced industry? These are some of the questions we are going to answer in this chapter.

In this chapter, we will first discover the main differences between the testing of games and the testing of other types of software. Then, the reader will learn more about the importance of QA. Finally, we will go through a couple of real-world scenarios that showcase what can happen when testing goes wrong in the gaming industry.

By the end of this chapter, you will have good insights into the basics of game QA and its importance.

In this chapter, we will cover the following topics:

Understanding the evolution of modern game testingExploring the differences between software and game testingWhy is QA important for games, especially within the agile process?When and how should QA testing for games be performed?

Understanding the evolution of modern game testing

Today, QA is one of the key components of any modern software development process. It is unimaginable to release software to users without testing it first. Users now have so many choices with regard to apps, games, and digital tools, and if you release software that does not work properly or has usability issues, you risk losing many of your users. Even worse, you risk your reputation as a developer if players discover something in your game that doesn’t work and publish this information online on game forums and social media.

Therefore, QA is an important component of the development process.

QA and testing are interchangeable terms. Throughout history, humans have striven to provide quality of execution in their work – from the ancient pyramids through medieval fortresses to modern software. At its core, modern QA has its roots in medieval professional guilds, such as those for tailors, merchants, and smiths. To ensure that the quality of their products met the required quality standards, guilds implemented strict peer control that in many ways is similar to testing today. They set quality standards that guild members had to meet in order to become part of and stay in the guild. These parameters ensured that guild members everywhere provided a high level of service and in return, drove more business to them.

Modern testing is not too far from that: we test software to ensure that it meets the required quality standards and includes all implemented and approved features. Of course, these days, we have replaced the quality standards set by guilds with ones set by product owners and end users.

Modern game testing has developed along with the growth of the gaming industry. Games became widely popular in the 1980s, and they kept on evolving to various new platforms: first consoles, then home PCs, and, in the 21st century, mobile and other handheld devices. It doesn’t look like the industry is going to slow down anytime soon either. The gaming industry in 2021 was globally worth more than 180 billion USD, more than the music, TV, and film industries put together (https://www.thc-pod.com/episode/the-gaming-industry-is-now-bigger-than-movies-and-music-combined). The biggest money makers are mobile and free-to-play (F2P) games, and the top earners among them bring in over a billion USD per year (https://newzoo.com/key-numbers; https://www.statista.com/statistics/263988/top-grossing-mobile-ios-gaming-apps-ranked-by-daily-revenue/).

Games are a big business today and quality is more important than ever. With that said, there are no common standards for game QA across the industry. Every gaming studio is different and even within the same company, different teams follow different QA practices. The differences are even more significant when testing for different gaming platforms – while testing on consoles hasn’t changed too much since the beginning of game development, mobile game testing is embracing the latest trends in QA to be able to support more fast-paced development.

The terminology used in game testing is not unified (a particular term may mean completely different things in different studios), and even QA jobs might have different levels of responsibilities or completely different job descriptions from studio to studio. Taking into account all those differences, certain things remain the same. Testers generally spend months testing games repeatedly, using different approaches and shifting focus to different parts and characteristics of the game.

What would we consider to be QA today? There are many different definitions out there, but at its core, game QA is a set of testing activities, including test execution, exploration, and verification, that aims to ensure that games meet design specifications, technical quality, platform regulations, and player expectations. As we can see just from the definition, QA in gaming entails a lot of responsibilities and requires a whole range of skills. A game tester is a person who must fully understand the product’s vision, is familiar with technical risks and dependencies, can juggle conflicting priorities, knows the game better than anybody, and represents the players’ interests.

Now that you have a basic understanding of the prerequisites and best practices of QA for games, let us try and understand what makes game QA different from normal software testing. Knowledge of these differences, especially in terms of the unique aspects we test for in game development, is essential for us to master QA testing and meet the end user’s expectations within the gaming industry.

Exploring the differences between software and game testing

Regardless of whether our software is used in medical devices, spaceships, or the games we play, the testing methodology, if not the same, is remarkably similar. But, even if testing medical software might seem more demanding, it doesn’t mean that testing games is going to be any easier. It’s challenging to adequately perform QA on time and within budget. Usually, if development encounters problems and misses milestones, QA testing, which generally occurs toward the end of the development cycle, will also be postponed. Unfortunately, software release dates are rarely moved forward. Hence, the more delayed development is, the shorter the time span allowed for QA checks. However, the scope for testing stays the same or even grows wider. That’s why the profession of game testing is somewhat notorious for its long working hours and high stress levels.

As per their definition, games also comprise software, just like the app you use on your phone to track your steps, the software that assists pilots to fly planes more efficiently, and even the software you use to read this book! All these digital products have their differences – in terms of complexity, the programming language they use, the target audience, user experience, and so on. However, they also have lots of similarities.

Software that is used in airplanes, medical devices, and military applications is considered life-critical. That means that if such software fails for any reason, it can lead to the loss of human life. When we compare this to game software, the worst consequences of bugs in game systems are a loss of progress or, in the case of F2P games, a loss of money. While unpleasant for users, failures in game software are much less impactful.

Looking at the difference through the lens of QA, life-critical software testing is usually more rigid, takes longer, and has strict, well-defined requirements. When we talk about non-life-critical software, testing practices will very much depend on the internal company processes and software development methodology in use. Although some aspects of software testing are company- or industry-specific, the following aspects are common:

StabilityScalabilityFunctionality of featuresUser interface (UI) look and functionUser flow and usability (UX)Performance under stressFirst-time user experience (FTUE)Localization

When testing games, in addition to the aforementioned aspects, we also test for certain specific aspects that are generally not tested in any other category of software development. These include the following:

Fun factorArtificial intelligence (AI) in gamesGame physicsEvaluation of game rulesLevel progressionGame difficulty balancingMultiplayer functionsPlaythroughRealismConsistencyGame levelsAchievementsSound and musicVoice-overs

We can see from the preceding list that although there are many crossover areas in testing, gaming software has its own specific requirements that require mastering. In this book, we will cover in detail the methodologies and practices that will help you do that.

Now that you know what makes game QA unique, let us understand why testing these unique factors is necessary to launch and maintain a successful game in the market.

Why is QA important for games, especially within the agile process?

By now, it’s obvious that QA is an essential part of ensuring the quality of a product. But, just how important is it? What will happen if QA is not done, or if it hasn’t been done well? There are quite a few examples of software failures that have had a big impact on end users and caused significant damage and even loss of human life.

One of the more well-known examples is the case of Stanislav Petrov, an air defense officer in the USSR, who potentially managed to avert a third world war. On September 26, 1983, while on duty, Officer Petrov received a notification from the nuclear early warning system, showing that the US had launched its nuclear missiles, attacking the USSR. Officer Petrov realized in time that the system was malfunctioning and that the alarm was false.

Gaming examples are less scary, but one that really showcases how bad game bugs can get is the case of the World of WarcraftHakkar bug. The final boss in the Zul’Gurub raid, the Blood God Hakkar, was designed to spray Corrupted Blood when killed on enemies close to him, potentially killing weaker heroes. Unfortunately, that caused pets in the game to become poisoned and they quickly managed to spread the plague and kill thousands of players. Of course, all those deaths were digital and Blizzard, the game’s developer, reacted fast and released a patch that revived most of the slain characters. To learn more about the Hakkar bug and the impact it had on the game and the players, see https://en.wikipedia.org/wiki/Corrupted_Blood_incident.

From these couple of examples, we can already see how important role QA plays in software testing. Without skilled, efficient, and timely QA, we can lose users, revenue, and even human lives. But modern QA is important for other reasons as well. With the rise of the free-to-play (F2P) business model and the huge importance of live operations (live ops), game development had to change its methodology to accommodate fast-paced, iterative development. Most of the game development industry has now switched from the waterfall model to agile methodology.

For QA testers in gaming, that was a significant change in terms of when and how testing was executed. Instead of being done at the end of the development, QA now plays a more active part in development teams, with testers representing the players’ perspective and working closely with designers and coders as part of the team. QA testers verify issues that are highlighted by the player support team, do early tests of new features, and learn about technical risks from developers. From being purely quality gatekeepers and critics, in modern game development, QA testers are part of a team that has an accurate understanding of how users will really interact with the game.

While having QA functions embedded into the team will not replace proper playtests, it does give developers early insights into how a player will perceive certain features or changes in the game. This can make product development more efficient and help avoid making expensive design and balancing mistakes that can cause lots of harm to the end product. And while not all agile teams will have embedded QA, the ones that do usually make sure that they take full advantage of the intimate player knowledge that the QA specialists hold, as well as their deep insight into game strengths and weaknesses. This reliance on QA specialists or teams not only makes QA important to prevent games going live with bugs such as Hakkar but also helps make for more fun, engaging, and at the end of the day, more profitable games.

When and how should QA testing for games be performed?

There are lots of discussions about when QA is executed. Traditionally, QA was always done toward the end of development. This was especially true when using a waterfall system. Following the waterfall development rules, QA testing is performed only after development has concluded and the finished software is handed over to the testing team. The following figure shows how the waterfall model follows specific steps in a particular order:

Figure 1.1 – Waterfall system

In a waterfall system, testing is done only when development is finalized. However, with the rise of agile methodology, gaming studios have gradually started to adopt this method of development as well. Furthermore, with the explosion of mobile gaming and F2P games, it became obvious that agile development was necessary for efficient live ops. This required that we change the way we test games. Handing the product to the testing team when “done” became impossible, because modern F2P games never end, and hence development is never finished.

When should QA for games be performed?

Game development has evolved into an iterative model, where we develop smaller chunks of code that are tested as they are developed. As we can see in the following figure, agile development is iterative – small, usable pieces of code are frequently released and tested:

Figure 1.2 – Iterative development

The aim of agile methodology is to develop minimum viable features that can be continuously improved in future iterations.

There are several unique aspects of testing in an agile environment:

Testing is continuous: a QA tester still holds the position of gatekeeper and has the responsibility of validating whether a feature or the whole build is of good-enough-quality to be released to the target audience. However, besides that, QA testers also test early features, validate usability flow, develop test cases based on use case scenarios, and perform many other tasks. If your game is live, the QA team will additionally handle live testing and verification of bugs coming from player support, forums, and your game’s social media channels.To be able to support continuous testing, QA is part of the agile team, rather than a separate unit. It is impossible to get full insight into agile development if QA is not part of the team. In modern game development, we still often use outsourced testers for the QA-heavy stages: first-time releases, major updates, or significant changes such as the introduction of multiplayer. However, very often we have situations where we have one full-time QA team member and a scalable external QA partner.Testers must adapt their approach to testing based on the development stage and feature readiness. It doesn’t help to report minor bugs while testing the first version of a major feature. This can be particularly challenging for QA testers. These professionals need to have experience, skills, and great collaboration with the rest of the team to fully understand the product priorities and focus their testing effort on the right thing at the right time. In the following table, you can find a high-level overview of what the testing focus should be in each stage of game development.

MILESTONE

TESTING FOCUS

HOW TO TEST

PRE-PRODUCTION

User stories, scalability, game design flow

Document reviews, decision tables

ALPHA

Core gameplay, fun factor, stability

On the selected principal device, emulators

BETA

UI, UX, game flow, FTUE, monetization, coverage

On a wide range of target devices

RELEASE CANDIDATE

Balancing, meta game, localization, game polish, shop and monetization, coverage

On the main supported devices

Table 1.1 – Testing focus per development phase

Testers represent the player in the team. It’s their job not only to test the code, but also to collaborate in the development process, putting themselves in the players’ shoes. This approach requires testers to have an in-depth understanding of the target audience, its preferences, its likes and dislikes, and a solid knowledge of competitors’ games. Most of the time, the QA specialist closely collaborates and communicates with player support; testers can consequently gain precious insights into players’ gaming patterns, game preferences, and the things they find most frustrating.

Based on everything we have discussed up to here, we can freely conclude that optimally, QA testing should start as soon as we start development. Many studios still hold on to old ways of working and test only at the end of the development cycle, as it’s perceived to be cheaper. This type of testing is usually very stressful as there is rarely enough time to ensure adequate testing coverage and many serious bugs can easily creep into the live game. By having QA performed in a more iterative way, we decrease that risk. This approach is potentially much cheaper than having to fix the damage of a failed product launch.

By now, we have a better understanding of when games should be tested. But, how should the testing be undertaken?

How should QA testing for games be performed?

When we start thinking about testing, it’s important to have some kind of understanding of what the desired outcome of our test is. That will allow us to compare the actual result of the test with the desired outcome. For some tests, this might be obvious – for example, the test case open settings and mute the sound has the obvious desired outcome that the game is muted.

For other scenarios, however, it’s not so simple. If we investigate a test case such as check whether game loads, it might seem deceivingly obvious: of course, the game should load! But, how long should the loading process take? Should there be any sort of loading indicator? Will the audio be on or off while loading? As we can see, it’s not necessarily that simple to determine what the desired outcome is. This test case is something my team and I had to experience ourselves to find the answer to. It took our team almost a week, while also researching competitors, to finally agree on what the expected outcome of that particular test case should be. The desired outcome of a test case is also sometimes called the test oracle.

More often than not, we also use test cases for testing. Even if they are not a mandatory part of game testing, they are very frequently used across various gaming studios. We will study test cases in depth later in the book, but for now, it’s sufficient to become familiar with their general definition. Test cases are written instructions that list specific steps for how testing should be performed.

Besides test cases, testers can execute tests using different types of documentation. Many teams create their own product test plans, which are detailed, carefully planned documents describing all aspects of testing that will be undertaken for a specific game. Test plans usually don’t contain test cases themselves, but they specify test case tools, who created them, how, when, and based on what. The plan might also contain links to the test case repository.

If your team has fully embraced agile development practices, you might also use test charters. Test charters are less formal documents compared to test cases. They generally contain information about what the testing goal is and some ideas and approaches regarding how to reach it.

Lastly, let’s talk about who is doing the testing. By now, you have probably realized that we are using tester and QA specialist/tester as interchangeable terms. There are really no differences between the two, but, as we don’t have a unified industry terminology, there might be some differences between the two, depending on how a particular studio interprets testing tasks. However, besides professional QA personnel, there are other people who perform testing as well:

DevelopersMembers of your development team apart from developers (designers, artists, etc.)The marketing teamProduct managersPlayers

It’s part of the developer’s job to do testing as well, although they use a very different methodology from testers. Developer testing focuses on what is called white-box testing. This implies testing mostly focused on unit tests and code reviews. While these tests are exceptionally useful, they have slightly different purposes compared to the tests performed by QA teams.

In modern agile teams, most of the team performs at least some type of testing. Designers review the logic and validity of their designs. Artists check how their work looks and feels in the game. UX designers review any changes in the user flow. The producer will have an overall look at all the components, trying to assess the level of completion and early build quality. All of these help the QA team to get a better picture of where the specific risks are and focus their efforts in the right place at the right time.

When we talk about marketing, we primarily talk about player support. It is wise to keep the player support team informed well in advance about any game developments and provide them with design documents and access to early builds. That will help them understand the game better and address customer comments and complaints properly. As they are the people spending the most time with the players, very often, you get valuable feedback from those early play sessions.

Lastly, players themselves are also testers, whether we like it or not. They will play the game in all the possible ways they see fit and report bugs through player support, forums, or social media. You might ask, if they test it themselves, why would we bother paying specialists to do it? This is because, usually, when a player notices bugs in the game, it’s already too late, especially if the bug is a serious one. We might already have lost part of our players or a significant amount of revenue.

However, there are ways of utilizing players’ testing in a more formal way. We do that through beta testing or playtests. In these types of tests, we allow smaller, often specifically selected groups of players to play early builds of the game. The primary goal of these tests is to get an idea of how players will receive the game and their likes and dislikes, but they also very often yield useful bugs.

Summary

In this chapter, we learned about some of the basics of modern game testing. This will help you to easily transition into modern, fast-paced game teams and adapt more quickly to agile teams. You also learned about the main differences between testing software and games, which lays the foundation for what comes next: practical QA steps and the challenges that come with it.

2

All Engines Go – The Basics of Game QA

In this chapter, we are going to dive deep into the core of game QA and examine in detail the elements that should be tested in games. We will learn how QA works in practice and what the main challenges of game QA are. At the end, we will touch upon modern agile practices and how game QA fits within these practices.

The goal of this chapter is to provide you with practical information on modern game testing that will allow you and your team to optimize their efforts and focus on the most important elements. You will also learn more about how game QA functions within the modern agile development framework.

By the end of this chapter, you will have a good understanding of the following:

What is tested in games?What is the most important element to test?How to prioritize in QAThe basics of QA in practiceWhere QA fits in the modern agile methodology

What is tested in games?

By now, we already know that there are lots of similarities between general software and games. In practice, this means that games must go through the same vigorous testing as any other software. We have also established that games have their own unique characteristics. Looking at games as a whole, they lie at the intersection of technology, art, and business. This makes them a unique type of software that requires us to consider a broader picture when talking about testing games.

The gaming industry is notorious for its lack of documentation, and very often testers need to figure out themselves what to test and how. It’s advisable to get any design documents or use cases to help you determine the scope of testing and the expected results. In general terms, we are testing the following:

StabilityFirst-time user experience (FTUE)Core game loopsLevel progressionGame physics (if the game has physics)Game logicAI behaviorUsability flow (UX)

We will go through each of these aspects in the following subsections.

Stability

Stability is the core element of any application and forms the foundation for all other QA efforts.

So, the primary aspect that we want to test in a game is its stability, which is an indicator of whether the following have been achieved:

The game can be installed on a target platformThe game’s installation process is not too longThe game runs on the target platformThe game runs on multiple versions of an operating systemThe game runs at appropriate framerate and doesn’t have delays or glitchesThe game is stable and doesn’t crashThe game doesn’t freezeThe game doesn’t kick a player out of the game

As we can see from this list, there are quite a few aspects of stabilitythat we need to consider. Another thing we need to pay attention to is that some of these requirements are ambiguous. When we say the installation process is not too long, what does that mean? Is there a standard optimal time for this? The answer is not straightforward. Some complex games can have long loading times; this is something that would not be acceptable for casual games. Ultimately, the right answer is determined by the actual players. Will players accept this long installation process? If the answer is no, then the installation takes too long. Another more practical way to set specific values for this requirement is to check the loading times of similar games that can be considered your competition. You probably want to stay in the same range as your competitors, or do better.

Of course, when we talk about premium PC games, this particular requirement is not that important. Players already paid for the game, and they are committed to waiting until the game is loaded. They might not be too happy about it, but this will not affect your game sales. On the other hand, if we talk about a free-to-play mobile game, this same problem can spell disaster for your game. Players have low levels of commitment as they didn’t spend money on it in advance and there are many competitors offering a similar gaming experience, also for free. If it takes too long for your game to load, they might just abandon it and move on to the next best thing. Even worse, they might give you a one-star review in the store and say that the game doesn’t work, which will discourage other players from downloading your game.

In the example above, we can see how the specifics of the platform and business model directly affect the importance of QA.

Another reason why it is important to test game stability first is the fact that this is a precondition for all other tests you will run. You will struggle with testing progression or game logic if the game is unstable, and you won’t be able to test anything if the game doesn’t load. These tests are always part of Basic Acceptance Testing (BAT) and are usually run first.

After stability, the next thing we want to test is whether the game does what it is supposed to do.

First-time user experience (FTUE)

FTUE is similar to a tutorial. We test how the player is onboarded to the game: is the gameplay explained well enough? Is it paced properly? Does the player have enough chances to practice what they are taught? Is it too long? Is it correct? Is it misguiding? Asking these types of questions gives us practical guidance on how to test the FTUE. We do have to keep in mind though that these answers will vary from game to game. As with many other things, often, the answers depend on the specific game genre and the target audience. While role-playing games (RPGs) and many strategy games are known for their steep learning curves, most casual and puzzle games have more detailed explanations on how to play the game and don’t become too challenging early on.

Core game loops

Core game loops are the main game design mechanics. These are the main actions that the player carries out in a game; these actions strongly depend on the game genre. For example, in match-three games, the player will be taken to a level where they need to match three or more elements to create strings. Matching the elements clears the board and the player is rewarded with a specific number of points for each match. When the player reaches a certain number of points, the player will win the level and be able to move on to the next level. This is the core game loop of a match-three game; we can see many items that we can test here:

Can you start a level?Does each level start at zero points?Can you do the matching as intended?Does each match give you the appropriate number of points?Does the physics of matching work (do matching items disappear, explode, etc.)?Are matched elements removed from the board?Does a level really end when the target is reached?Can you move to the next level when the current level is won?Do points reset with each new level?

There are many more things we can test on each level but asking these questions will tell us if the core game loop works as intended.

Note on testing more complex games

More testing is required for more complex games, such as strategy games, where a player needs to get currency to build a building (generator) that generates goods that are used to build an army, and an army in turn is needed to conquer more territory, as more territory brings more currency. As we can see, these game loops are much more complex with more co-dependencies, and as such, will take more time to test fully.

Level progression

We already mentioned level progression when we spoke about core game loops. Many games have levels – sequential stages of the game that showcase that a player’s skills and fortunes are growing with the time they spend in the game. They act as a motivator for the player and are one of the reasons why they keep playing the game. Levels can be obvious and part of the gameplay: they may be displayed on a map or similar and numbered. But they can also be incorporated more deeply into the gameplay and shown as the player’s skill level, settlement level, or similar. We can find great examples of embedded levels by looking at games in the 4X genre such as State of Survival. Regardless of how levels are displayed, we need to test the following:

Can you start the level?Does it start with appropriate resources/counters?Can you finish the level when the targets are met?Can you pass the level?Can you start the following level?Are leveling-up rules clearly visible?

Besides the technicalities of moving to the next level, here we also need to pay attention to the difficulty of the level. This is one of the parts of testing that is very specific to the gaming industry. How can we determine whether a level is too difficult or too easy? Usually, with free-to-play games, we can adjust the degree of difficulty relatively easy. But, with premium and boxed games, it’s often impossible or exceptionally hard. One of the best ways to decide what is too hard/too easy is to try to put yourself in the player’s shoes. Who is playing your games? What level of gameplay difficulty will they expect? As a tester, it’s always recommended to play similar games yourself, to get a sense of how difficulty is balanced. It is also advisable to have discussions with level designers and get a better understanding of the desired balance.

Game physics

Some of the most successful games out there are physics based. Angry Birds, Goat Simulator, Portal, and Half Life are all examples of games that rely heavily on the physics gameplay element. By physics, we mean mimicking the behavior of objects in real life: for example, heavy objects will fall faster, or if you hit a ball with more force, it will fly further. Of course, games are supposed to be fun, and sometimes the laws of physics are bent or even broken. In GTA, your car has a flying car mode; in some games, you can jump really high; or, for example, in Madden you can get an acceleration boost that allows you to run extremely (unnaturally) fast.

Note on testing physics in games

When testing physics in a game, it is very important to understand how objects in the game are supposed to behave: are physics in the game a copy of the real world or does the game design allow for certain freedom? After we are clear on that, we should focus on testing physics consistency: that flying car mode should really enable us to fly the car, or tapping the character twice should allow him to jump extra high every single time when physics apply in the game.

Game logic