29,99 €
In today’s world, software is everywhere—from entertainment apps to mission-critical systems that support our health, finance, and infrastructure. Testing plays a vital role in ensuring these systems work reliably. Whether you're a software developer, hobbyist, or IT professional, this book will guide you in mastering the art of testing. It’s about asking the right "What if?" questions, uncovering vulnerabilities, and ensuring software performs as expected throughout its lifecycle.
Testing isn't just about automation; it’s a human-driven, creative process that requires skill, and a deep understanding of software behavior. With practical examples and expert insights, this book helps you craft your own test strategies and explore novel approaches to problem-solving in the testing world. With its help, you’ll hone your testing skills with techniques and methodologies rather than tool-based solutions.
Authored by experts Matt Heusser and Michael Larson, the book provides valuable strategies for making testing both effective and engaging. Matt is known for his leadership in project rescue initiatives, while Michael’s work in accessibility testing has helped shape industry standards.
By the end of this book, you’ll be equipped to enhance your testing practices and ensure high-quality software in an ever-evolving tech landscape.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Veröffentlichungsjahr: 2023
Software Testing Strategies
A testing guide for the 2020s
Matthew Heusser
Michael Larsen
BIRMINGHAM—MUMBAI
Copyright © 2023 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Group Product Manager: Gebin George
Publishing Product Manager: Kunal Sawant
Senior Editor: Rounak Kulkarni
Technical Editor: Rajdeep Chakraborty
Copy Editor: Safis Editing
Associate Project Manager: Deeksha Thakkar
Project Coordinator: Manisha Singh
Indexer: Pratik Shirodkar
Production Designer: Joshua Misquitta
DevRel Marketing Coordinator: Shrinidhi Manoharan and Sonia Chauhan
Business Development Executive: Samriddhi Murarka
First published: December 2023
Production reference: 1231123
Published by Packt Publishing Ltd.
Grosvenor House
11 St Paul’s Square
Birmingham
B3 1RB, UK
ISBN 978-1-83763-802-4
www.packtpub.com
For Kathleen, Rachel, and Juliana Heusser. I’ll always be your dad, for anything. Also, I asked Michael, “How can I write chapter 14 and not mention Jesus, my Lord?” He said, “Put it in the dedication, where it belongs.” And so, it is. Finally, to every worker at Excelon Development, past, present, and future. Your work gave me a little free time for this research and to write this book without sacrificing time with Julie, who is still in school. I am incapable of expressing all that that means to me.
– Matthew Heusser
To Christina, Nicholas, Karina, and Amber Larsen, you are all the reason I do what I do. It’s an honor and a privilege to be your husband and father. Thank you for putting up with me. Additionally, thank you to the many people who have worked with me over the years and given me a chance to help make the software we use and the interactions we have with it a little better, more usable, more inclusive, and available to as many people as possible.
– Michael Larsen
Matthew Heusser is the 2014 Winner of the Most Influential Agile Professional Person Award (MAITPP) at Agile Testing days in Potsdam, Germany. In 2015 he received the most popular online contributor to Agile award at the Agile Awards, Marble Arch in London. In 2023, his company, Excelon Development, was listed as an INC 5000 award recipient; one of the fastest-growing privately held companies in the United States. In the same year, they were also listed as a top ten software test provider in CIO Applications magazine. He won awards through 30 years of dedicated service to the field of software delivery, including serving as lead organizer for the initial Great Lakes Software Excellence Conference, member of the board of directors of the association for software testing, senior editor of “How to Reduce the Cost of Software Testing”, and organizer of a half-dozen peer workshops on software delivery – all while making active contributions on software projects.
I’ve tried to credit people in the book for their testing ideas, but special thanks go to Homer Austin, Kathleen Shannon, and Dean Defino for teaching me a love of logic and proof. My father, Roger Heusser, taught me perseverance and commitment to a profession. It was my friends, Mike Pettee, Joe Cerven, Mary Colborn, Nikki Johnson, and others who helped me pick up after a life setback and re-orient myself with new goals. Asimov, Heinlein, Clarke, and Gygax taught me to love reading and want to write, but it was Eric Matthews and Steve Hoek who encouraged me to think deeply about software delivery, and Esther Schindler who encouraged me to write about it.
Michael Larsen has been working in software testing, one way or another, since 1991. He has been involved with a variety of companies and products over the years. He lives in the San Francisco Bay Area and has been an advocate for quality since launching his TESTHEAD blog in 2010 (https://mkltesthead.com/). He has served as a member of the Board of Directors and President of the Association for Software Testing (AST), and the Marketing Chair for the Pacific Northwest Software Quality Conference (PNSQC).
I owe a huge debt of gratitude to Shannah Miller and Marcia Bednarcyk, who took a chance on a young musician looking for work 30+ years ago and said, “Hey, why not?”
Cansu Akcay is a Software Test Engineer based in London with 10 years of industry experience. She is the founder of CA Software Testing LTD (http://casoftwaretesting.co.uk/). Cansu has worked as a Senior Software Test / QA Engineer, collaborating with various QA teams in manual and automation testing on multiple types of applications. She has also worked with large, medium, and small (start-up) local and global companies. Cansu manages an Instagram account (@ca.softwaretesting) where she shares tips and the latest updates on testing.
Stephen Spady is a Principal Software Engineer in Test with over 30 years of experience in software testing and development. Throughout his extensive career, he has held various roles, including software test engineer, software development engineer, lead, manager, software architect, test architect, and systems engineer. Notably, he dedicated 13 years to Microsoft, where he played a pivotal role in developing test automation and continuous integration/continuous delivery (CI/CD) systems for Microsoft Word, Office, RichEdit, Netdocs, and CRM.
Over the course of his career, Stephen has actively contributed to the advancement of AI technologies, conducting research and development for their application in software design, development, and testing. Stephen holds a Bachelor of Science in Computer Engineering from the University of Nebraska–Lincoln.
We live in an age where software is everywhere. It is inescapable at this point. Some of it is trivial, meant for entertainment or passing time, while some software is the most mission-critical possible, maintaining the delicate balance between life and death for a person. Most software that we will interact with will fall somewhere within that continuum. It may be on the web, on our phone, on our watch, or measuring the weight of our workout water bottle, reminding us to hydrate ourselves at important times. Even if we don’t interact with it directly, software runs in many areas of our lives that we don’t even consider, including our financial institutions, our power plants, medical imaging systems, or in running continuous trials to find the best way to synthesize chemical reactions or fight deadly viruses.
What do all of these areas of software interaction have in common? Someone has to create them and deploy them but perhaps most importantly, someone has to test them. Over the past couple of decades, there has been a move away from that “someone” and more towards “something”, meaning automated tests in some capacity doing all of the testing work. Surely, software can do the work of a thousand testers, right? Yet it has come back in the news media and in high profile cases that maybe, just maybe, having people who truly understand testing is important, necessary, and should be prepared to do that work. That person does not need to have the title of “tester” to do testing. They can be a software developer, a lone hobbyist working on a passion project, or someone working with a server farm running systems, making sure they are operable, and people are able to access them. Testing happens at many levels and over the entire software development life cycle.
At its most basic level, testing is the actual experimentation that takes place in the scientific method. It’s the asking of “What if?” questions. It’s the joy of being sneaky and difficult with software, to try to find areas where the software is vulnerable and open to attack, where the person doing the testing can either support or refute the ideas at hand. Is the software fit for use, or is there a problem here?
Our goal with this book, Software Testing Techniques, is to help you, our esteemed reader, take hold of the fun and adventure that is software testing (and yes, it most certainly can be fun, and it is often quite literally an adventure). We want to give you skills, processes, techniques, and perhaps some novel ways of looking at the puzzle pieces of software testing. If that sounds like fun to you, buckle in and join us.
Patrick Bailey, a professor at Calvin College, once did a research report where he asked people their role and, if they could only do one form of testing, what would it be? Bailey found, strikingly, that people tend to associate the most valuable kind of testing with their role. Programmers found unit testing valuable, internal customers found User Acceptance Testing, Business Analysts found system testing, and so on more valuable.
Project Managers, for example, tend to see work as an assembly line. Anyone with a mantra like “plan the work, work the plan” and a documentation bias is going to like the idea of writing the tests down and then executing them. When we have seen that tried, the results, are, frankly, lower value. So, the company criticizes the button-pushers, maybe laughing at them. Thomas Moore wrote about this in 1551 in his book Utopia. To borrow from that author: First, we create bad testers and then we punish them.
That may be the first mistake, which used to be the most common. Today, we see people who saw that first mistake, know it is foolish, and view all human testing as that sort of low-value, scripted exercise. This group sees testing as something else – automation of the GUI, lower-level unit checks, and perhaps API tests. All those have their part and place in this book, but they also fail to capture the improvisational element of good testing. We’ll cover that improvisational effort in Chapter 1, at the GUI level, where it is most obvious, and then try to maintain that spirit throughout additional chapters.
In other words, this book is not limited to “just” button-pushing testing, but there is something that happens in the hands of someone skilled that needs to be studied and applied at all levels, throughout the process.
The scope of our book is about all the ways to find out the status of the software, quickly, by exercising the code. We’ll be the first to admit that is not a complete picture of software quality. It does not include how to create good requirements, or how to perform code inspections or pair programs. We see testing as a feedback mechanism, it is not the only one, and there is more to quality than that feedback.
“Just testing”, we thought, was more than enough for one book. That feedback is important. It is often neglected; it is often done poorly. The second part of the book covers test integration into a delivery process. The third part covers “Practicing Politics”, on how to give feedback that can be effectively used by the organization.
If you’ve ever heard “Why didn’t we test that”, “Why didn’t we find that bug”, or, perhaps worst of all, “Okay, you found that bug and prioritized it as must fix and we insisted it could be delayed, but why didn’t you advocate more effectively?”, then this book is for you. This book is also for you if:
If you keep seeing bugs in software that seem obvious that you wish other people caught.If you want to get better at finding information quickly and expressing it well to change the outcome of the process.If you want to come up with ways to use the information you find to reduce the bug injection rate.If you want to be able to diagnose and explain how you made key risk/reward tradeoffs.Hopefully, by now you realize that testing is serious, grown-up, risk-management work. Any beginner can follow the process, and any mid-level tester can tie up most software teams in delays waiting to fix things. The real value in testing is beyond that, in making smart risk/reward decisions, to invest limited resources in risk management.
You can think of this as three levels of testing. On level one, you go through the basic motions of using the application in the most simplistic way. This is the “happy path.” The second level of tester is on a bug hunt. The person viewing testing views their job as to find bugs, or, as one person once put it, to “cackle with glee” as they “make developers cry.” Where the first level is probably overly agreeable, the second level can be actually adversarial to development. It is on the third level that we look at how much time will be invested in what risks in order to “not be fooled” about quality while making minimal disruptions to delivery speed. In some cases, finding problems with the product early can increase delivery speed.
Not too long ago or far away, one of the authors, Matthew, was the testing steward at a Health Insurance Company in West Michigan. The insurance company had contracted with a local but nationally known consultancy to come in and work on some software maintenance. One of the consultants, a brilliant coder, shoved the keyboard away in frustration, shouting, “This is totally untestable!”
Matt picked up the keyboard and mouse and started to exercise the software, saying, “Sure it is, watch!” This was followed by, “Oh, you don’t mean the code is untestable, you mean you have no way to have a computer run it through pre-defined exercises and check the results. You mean it isn’t … test-automate-able, maybe?”
This is another example of what Pat Bailey was talking about in his research at Calvin. To the programmer, “tests” were automated pieces of code that checked the functionality. To Matt, the employee of the insurance company, testing was the process of figuring out information about the software.
At almost the exact same time as this story, one of our mentors, Dr Cem Kaner, was giving a talk on “Testing as A Social Science”, (https://kaner.com/pdfs/KanerSocialScienceTASSQ.pdf). In that presentation, Dr Kaner defined software testing this way.
Software Testing is:
A technical investigationConducted to provide quality-related informationAbout a software productTo a stakeholderThis book tends to use that as an operating definition. Testing is where you try to find out if the thing you built will actually do the things you are asking it to do. While you can fool yourself with a schedule, a bad design, and code that doesn’t work, Testing is the first process designed to make sure that we are not fooled. Kaner went on to make this argument, “Much of the most significant testing work looks more like applied psychology, economics, business management (etc.) than like programming”.
This will probably upset a lot of programmers, and maybe some DevOps people too. After all, the holy grail of testing in the 2020s is to automate all the checking of things. In general, that means running all the checks, quickly, and reporting results automatically with No Humans Involved, or NHI.
People seem to stop and forget that the holy grail seems to be fiction, with a great deal of energy wasted searching for a legend. The yellow brick road led to the Emerald City headed by a wizard that was a lie. People forget that Zeitgeist, the spirit of the age, implies not truth but a sense of fashion that will go out of style.
It’s not that we are against automation or tooling. Our problem is the dearth of information on how to do testing well.
Given a test framework (or exercise) …
What test should we run (or write) first?
What does that test result tell us?
When should we stop?
When we stop, what do we know?
This sounds like an interactive exercise, where the results of the first test inform the second. It is possible to select some subset of that exercise, turn it into an algorithm, write it up in code, and run it routinely to see if what worked today runs differently tomorrow. Some people call this regression testing. It is even possible to create automated checks before the code, sometimes called Test Driven Development (TDD). Even if the tests are automated, the process of figuring out what to run and what that tells us is the process we are more interested in.
Some people make a distinction between the institutionalized, run-it-every-time bits of code and the feedback-driven process of exploration. In particular, Michael Bolton uses the term “Checks” to be the algorithmic unthinking comparison, and ‘Testing” to be the more expansive activity that often includes checking. We find this helpful in that “automated testing” loses some of the flavor of Dr Kaner’s definition that we saw earlier. To that, we would add Heusser’s First Observation: After the first time you run an automated test to see if it passes, it really ceases to be a true test. Instead, it becomes automated change detection – and what programmers do is create change!
Our goal for the reader is to be able to do more than memorize the material and more than perform an analysis of software according to a set of rules to come up with test ideas. We believe in this book we have synthesized a large amount of material that superficially disagrees with itself and is inconsistent. We’ve interpreted that through our own ideas, laying out some test approaches that have the most value, and then giving advice on how to balance them. When you’re done with this book, you should be able to look at a user interface or API, identify many different ways to test it and have the tools to slice up your limited time to determine how much time to spend on what test approaches, describing them to someone else.
Whew. It’s finally here. The book is final. It is ready to be out.
This book is about our (Matthew Heusser and Michael Larsen) stories. We are proud of it. It includes how we test, why we test, what influenced our thinking about testing, and a few exercises for you. To do that we had to do a few unconventional things, like change our “person” between I, you, we, Matt, and Michael. We had to write about opinions, which you can disagree with, and experiences, which might not be relevant for you. One of our biggest challenges was what to cut, to understand what would be the most valuable to you, the reader. After doing hundreds of interviews on podcasts, consulting broadly, and attending a hundred or so conferences, we think we have some idea of what that might be. Yet a book takes our potential audience for feedback even wider. We look forward to hearing from you, the reader, about what we could have phrased more carefully, and what we should add, subtract or change. For today though…
On with the show.
This book is for anyone who wants to understand the process and mindset of software testing, to put into practice the ideas and methods that will help them better test and understand the systems they work with, and to advocate for those who can’t speak for themselves in the software development process. Note, that does not presuppose that the person reading this book is a dedicated software tester. We encourage everyone who has any involvement in software development, including programmers, product and project managers, business analysts, systems administrators, and anyone who has a vested interest in the software their organization creates to be as capable and robust as possible.
Chapter 1, Testing and Designing Tests, this chapter introduces testing as a risk management activity, focusing on powerful test ideas for important software parts. It discusses the theory of error, unscripted testing, and various methods including model-driven and soak testing.
Chapter 2, Fundamental Issues in Tooling and Automation, addresses common pitfalls in test automation and shares lessons from years of experience. It covers concepts like the minefield regression problem and the battleship problem, concluding with solutions to these automation challenges.
Chapter 3, Programmer-Facing Testing, focuses on developer testing and covers unit testing, test-driven development, and testing web APIs, among other topics. It concludes with a practical exercise in creating unit tests, using Ruby as an example.
Chapter 4, Customer-Facing Tests, explores the nuances of customer-facing test automation, discussing GUI automation patterns, specification by example, and low-code/no-code tools, aiming to enable readers to analyze and optimize user interface testing.
Chapter 5, Specialized Testing, delves into specialty areas of testing and covers performance and load testing, security, accessibility, internationalization, and regulated testing, each with its unique challenges and methodologies.
Chapter 6, Testing Related Skills, expands beyond test design and execution and focuses on skills like recognizing bugs, communicating problems, planning and documenting work, and metrics, and influencing change in testing processes.
Chapter 7, Test Data Management, addresses the test data management problem and provides techniques for creating, storing, editing, deleting, and restoring data states to drive efficient application testing and reliable test tools.
Chapter 8, Delivery Models and Testing, broadens the scope of how testing interacts with software delivery models like Waterfall, Scrum, and DevOps. Through this, this chapter helps readers understand and optimize the interaction between testing and these models.
Chapter 9, The Puzzle Pieces of Good Testing, breaks down the components of testing, like recipes, coverage, and defects, and encourages readers to reassemble these elements into a cohesive test strategy tailored to their needs.
Chapter 10, Putting Your Test Strategy Together, builds on the previous chapter, to guide readers through analyzing current testing states, prioritizing risks, and communicating and implementing a comprehensive test strategy.
Chapter 11, Lean Software Testing, introduces Lean Software Testing and combines test and operations management techniques. It covers topics like the Seven Wastes, flow, constraints, and a lean approach to metrics and measurement.
Chapter 12, Case Studies and Experience Reports, uses case studies and experience reports, and offers real-life lessons and strategies from the field, providing practical insights into testing challenges and solutions.
Chapter 13, Testing Activities or a Testing Role?, explores the nuances of who should perform testing activities, discussing cultural conflicts, risk mitigation teams, and various testing models like shift-left and continuous testing.
Chapter 14, Philosophy and Ethics in Software Testing, ventures into the philosophical and ethical dimensions of testing. This chapter examines the limitations of testing, the value of ethical reasoning, and the importance of clear communication in testing processes.
Chapter 15, Words and Language About Work, focuses on communication and emphasizes the importance of precise language and context in testing, exploring different testing schools of thought and the distinction between process and skill.
Chapter 16, Testing Strategy Applied, applies the book’s concepts to practical scenarios, including a reference implementation for a mobile test strategy and a critical examination of AI in software testing, offering a comprehensive view of testing strategy application.
Most of the approaches and techniques described in this book are less tool-specific and more person-specific. There are, however, a few examples in the book that will benefit from downloading software or hardware and installing them. Most of the tools and techniques are platform agnostic or have distributions that work with all platforms. Some of the examples in this book are based on Ruby, so having a working version of Ruby on your system would be beneficial. In the Specialized Testing section there is an example based around JMeter, so installing JMeter and its components would likewise be worthwhile.
Software/hardware covered in the book
Operating system requirements
Ruby
Windows, macOS, or Linux
JMeter
Windows, macOS, or Linux
If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.
You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Software-Testing-Strategies. If there’s an update to the code, it will be updated in the GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/Software-Testing-Strategies_ColorImages.pdf.
Disclaimer
With the intention of the Publisher and Author, certain graphics included in this title are displaying large screen examples where the textual content is not relevant to the graphic example. We encourage our readers to download the digital copy included in their purchase for magnified and accessible content requirements.
There are a number of text conventions used throughout this book.
Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “It will create a Fizzbuz_02 object in memory, call the method, and expect the output.”
A block of code is set as follows:
if !defined?(count_to) or count_to<1 puts "Use: FizzBuzz01.rb (count)\n" puts "Where count is a round number of value one or higher" abort(""); endAny command-line input or output is written as follows:
Finished in 0.001213s, 2473.2069 runs/s, 2473.2069 assertions/s. 3 runs, 3 assertions, 0 failures, 0 errors, 0 skipsBold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “ All the test does is go to Excelon Development’s main page, then click on Consulting, then Contact Us, and fill in the Contact Us form”
Tips or important notes
Appear like this.
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Once you’ve read Software Testing Strategies, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.
Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.
Thanks for purchasing this book!
Do you like to read on the go but are unable to carry your print books everywhere?
Is your eBook purchase not compatible with the device of your choice?
Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.
Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.
The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily
Follow these simple steps to get the benefits:
Scan the QR code or visit the link belowhttps://packt.link/free-ebook/9781837638024
2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly
This part is about how we test. It starts with a real working example of jumping into a project to test without any background. Our goal with this part is to give the reader both an overview of all the areas of risk that people can reduce by inspection, and also to give practical, hands-on advice on how to test software with little to no introduction, little to no documentation, under conditions of time pressure and uncertainty, while doing work that stands up to scrutiny. This includes all kinds of testing, including developer testing and test tooling. We’ve tried to present a balanced approach to test tooling to optimize your chances of success while being realistic.
This section has the following chapters:
Chapter 1, Testing and Designing TestsChapter 2, Fundamental Issues in Tooling and AutomationChapter 3, Programmer-Facing TestingChapter 4, Customer-Facing TestsChapter 5, Specialized TestingChapter 6, Testing Related SkillsChapter 7, Test Data Management