Managing the Testing Process E-Book

Rex Black

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 1177

Veröffentlichungsjahr: 2011

Beschreibung

New edition of one of the most influential books on managing software and hardware testing In this new edition of his top-selling book, Rex Black walks you through the steps necessary to manage rigorous testing programs of hardware and software. The preeminent expert in his field, Mr. Black draws upon years of experience as president of both the International and American Software Testing Qualifications boards to offer this extensive resource of all the standards, methods, and tools you'll need. The book covers core testing concepts and thoroughly examines the best test management practices and tools of leading hardware and software vendors. Step-by-step guidelines and real-world scenarios help you follow all necessary processes and avoid mistakes. * Producing high-quality computer hardware and software requires careful, professional testing; Managing the Testing Process, Third Edition explains how to achieve that by following a disciplined set of carefully managed and monitored practices and processes * The book covers all standards, methods, and tools you need for projects large and small * Presents the business case for testing products and reviews the author's latest test assessments * Topics include agile testing methods, risk-based testing, IEEE standards, ISTQB certification, distributed and outsourced testing, and more * Over 100 pages of new material and case studies have been added to this new edition If you're responsible for managing testing in the real world, Managing the Testing Process, Third Edition is the valuable reference and guide you need.

Leseprobe

Table of Contents

Title Page

Copyright

About the Author

Credits

Acknowledgments

Introduction

The Focus of This Book

Canon or Cookbook?

The Tools You Need

The Resources You Need

On Context

Using This Book

What's New and Changed in the Third Edition?

Chapter 1: Defining What's on Your Plate: The Foundation of a Test Project

What You Might Test: The Extended Test Effort

What You Should Test: Considering Quality

What You Can Test: Schedule, Resources, and Budget

Case Study

Exercise

Chapter 2: Plotting and Presenting Your Course: The Test Plan

Why I Write Test Plans

How Many Test Plans?

Using Drafts to Stimulate Discussion

Using a Test Plan Template

The IEEE 829 Template: Compare and Contrast

Selling the Plan

Clarity, Pertinence, and Action

Case Study

Exercises

Chapter 3: Test System Architecture, Cases, and Coverage

Test System Architecture and Engineering

It's Not Saint Paul's, But…Principles for Test System Architecture

The Bricks and Mortar of the System: Test Cases

Avoiding the Dreaded Test Escape: Coverage and Regression-Test Gaps

“There's a Lesson to Be Learned Here…”: Test Case Incremental Improvement

You Can't Do It All: Deciding What Not to Do

Case Study

Bonus Case Study

Bonus Case Study

Exercises

Chapter 4: An Exciting Career in Entomology Awaits You: A Bug-Tracking Database

Why Bother? The Case for a Formal Bug-Tracking System

So, What Seems to Be the Problem? The Failure Description

Flexible Reporting: Beginning to Construct a Database

The Vital Few and the Trivial Many: Ranking Importance

Putting the Tracking in Bug Tracking: Adding Dynamic Information

Finishing Touches: Capturing Bug Data for Analysis

The IEEE 829 Standard

Extracting Metrics from the Bug-Tracking Database

Managing Bug Tracking

Case Study

Exercises

Chapter 5: Managing Test Cases: The Test Tracking Spreadsheet

Building a Minimalist Test Tracking Spreadsheet

Making Enhancements

Putting the Test Tracking System in Motion

The IEEE 829 Test Log

Extracting Metrics from the Test Tracking Spreadsheet

IEEE 829 Test Reporting: Interim, Level, and Master

Case Study One

Case Study Two

Exercises

Chapter 6: Tips and Tools for Crunch Mode: Managing the Dynamic

Do Sweat the Details: Staying on Top of Everything

A Spider's Web of Connections: Managing Test Hardware and Software Configuration Logistics

Expect the Unexpected: A Change Management Database

Case Study

Exercises

Chapter 7: Stocking and Managing a Test Lab

Do You Need a Test Lab?

Selecting and Planning a Lab Area

The Test Lab Inventory

Security and Tracking Concerns

Managing Equipment and Configurations

Keeping the Test Environment Clean

Human Factors

Case Study

Exercises

Chapter 8: Staffing and Managing a Test Team

The Right Person for the Job: What Kind of People Make Good Test Engineers

Defining the Test Team: How Many Whos Do What?

Specialists or Project Resources? Organizational Models

Hiring Testers

Giving a Damn: Motivating Your Test Team

Extending Your Talent: Using Temporary Experts and Implementers

Case Study

Exercises

Chapter 9: The Triumph of Politics: Organizational Challenges for Test Managers

Don Quixote, Champion of Quality: What's Your Job, Anyhow?

Where You Fit: The Test Group in the Organization

What Else Fits? Adding Other Functions to Test

Working with Other Managers: Directions of Test Management

Testing in the Dark: Should You Proceed without Documentation?

Pink Slips: Layoffs and Liquidation

Presenting the Results: The Right Message, Delivered Properly

“You Can Tell the Pioneers…”: The Effect of Early Adoption on Test

Exercises

Chapter 10: Involving Other Players: Distributed Testing, Outsourcing, and Related Topics

Choosing Your Partners

Planning a Distributed Test Effort

Managing a Distributed Test Effort

How Outsourcing Affects Testing

Case Study 1

Case Study 2

Bonus Case Study: People Are Not Widgets!

Conclusion

Exercises

Chapter 11: Economics of Testing: Fiscal Context

Is Quality Free? The Economic Justification for the Testing Investment

Case Study

Exercises

Chapter 12: Testing Implications of Project and Process: Situational Context

Where Testing Fits into the Project Life Cycle

Process Improvement

Improving Your Test Process

Managing the Testing Process: A Retrospective Conclusion

Case Study 1: Agile Testing Challenges

Case Study 2: Maturity and ROI

Exercises

Appendices

Appendix A: Hardware Testing Fundamentals: An Introduction for Software Testing Professionals

Test Management

Basic Functionality and Self Tests

Electrical Testing

Environmental Tests

Mechanical Life

Thermal Tests

Reliability

Packaging Tests

Acoustics

Safety

Radiation

Standards and Regulations

Components and Subsystems

Integrated Software

Supplier Quality Engineering

Pilot Testing

Case Study

Appendix B: Omninet: The Internet Everywhere Marketing Requirements Document

1 Scope

2 Required release date

3 Description of requirements

Appendix C: Omninet: The Internet Everywhere System Requirements Document

Functionality System Requirements

Reliability System Requirements

Usability System Requirements

Efficiency System Requirements

Maintainability System Requirements

Portability System Requirements

Design Models

Appendix D: Bibliography, Related Readings, and Other Resources

Bibliography and Related Readings

Online and Hard-Copy Publications

Contacting RBCS

Glossary

Index

Managing the Testing Process: Practical Tools and Techniques for Managing Software and Hardware Testing, Third Edition

Published by

Wiley Publishing, Inc.

10475 Crosspoint Boulevard

Indianapolis, IN 46256

www.wiley.com

Copyright © 2009 by Rex Black. All rights reserved.

Published by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-0-470-40415-7

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone]. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Web site may provide or recommendations it may make. Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Control Number: 2009929457

Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc. is not associated with any product or vendor mentioned in this book.

About the Author

With a quarter-century of software and systems engineering experience, Rex Black is President of RBCS (www.rbcs-us.com), a leader in software, hardware, and systems testing. For more than a dozen years, RBCS has delivered services in consulting, outsourcing, and training for software and hardware testing. Employing the industry's most experienced and recognized consultants, RBCS conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation, and more.

As the leader of RBCS, Rex is the most prolific author practicing in the field of software testing today. His popular first book, Managing the Testing Process, now in its third edition, has sold more than 30,000 copies around the world, including Japanese, Chinese, and Indian releases. His five other books on testing, Critical Testing Processes, Foundations of Software Testing, Pragmatic Software Testing, Advanced Software Testing: Volume I, and Advanced Software Testing: Volume II, have also sold tens of thousands of copies, including Hebrew, Indian, Chinese, Japanese, and Russian editions. He has contributed to 10 other books as well. He has written more than 25 articles, presented hundreds of papers, workshops, and seminars, and given about 30 keynote speeches at conferences and events around the world. Rex is a former president of both the International Software Testing Qualifications Board and the American Software Testing Qualifications Board.

When he is not working with clients around the world, developing or presenting a training seminar, or in his office, Rex spends time at home or around the world with his wife and business partner, Laurel Becker; his daughters Emma Grace and Charlotte Catherine; and his faithful canine friends Hank and Cosmo.

Credits

Executive Editor

Robert Elliott

Development Editor

Kelly Talbot

Technical Editor

Judy McKay

Production Editor

Daniel Scribner

Copy Editor

Candace English

Editorial Director

Robyn Siesky

Editorial Manager

Mary Beth Wakefield

Production Manager

Tim Tate

Vice President and Executive Group Publisher

Richard Swadley

Vice President and Executive Publisher

Barry Pruett

Associate Publisher

Jim Minatel

Project Coordinator, Cover

Lynsey Stanford

Proofreader

Nancy C. Hanger / Windhaven

Indexer

Robert Swanson

Cover Image

David Arky / Corbis

Cover Designer

Ryan Sneed

Acknowledgments

This book is a third edition, and that happens only when the first edition and second edition are successful. So, first off, I'd like to thank those of you who bought the first edition or second edition. I hope it's proven a useful reference for you. A special thanks goes to RBCS clients who used these books on their projects, attendees who provided evaluations of RBCS training courses, and readers who wrote reviews or sent me emails about the book. I have addressed your suggestions for improvement in this third edition.

A book gets into people's hands only through a lot of behind-the-scenes hard work by a publisher's team. A special thanks to the fine people at Wiley who helped bring this book to fruition, especially Kelly Talbot and Robert Elliott. RBCS associate Judy McKay provided valuable technical reviewing help. I'd also like to thank Ben Ryan, who shepherded Managing the Testing Process along through the first two editions, starting in 1998. I'd also like to thank my friends at Microsoft Press who helped me with the first edition: Erin O'Connor, John Pierce, Mary Renaud, Ben Ryan (again), and Wendy Zucker.

In the course of learning how to manage test projects, I have worked with many talented professionals as a tester, test manager, and consultant. The list of people who helped me is literally too long to include here, but my gratitude to all of my colleagues and clients is immense.

The material in this book appears in one-day, two-day, and three-day test management courses that RBCS associates and I have presented all around the world. I thank all the attendees of those seminars for their help making this material better in the third edition.

Of course, my appreciation goes out to all my past and current colleagues, subcontractors, employees, clients, and employers. I especially want to thank the clients who graciously agreed to the use of data and documentation from their projects to illustrate many of the tools and techniques I discuss.

Four people I want to name specifically in this regard are Judy McKay, Andrew Brooks, Jodi Mullins, and Steven Gaudreau. Judy is a director of quality assurance at large network equipment company. Andrew Brooks is vice president, CA Network and Voice Management Quality Assurance. Jodi Mullins is senior software engineer, CA Network and Voice Management Test Automation. Steven Gaudreau is software engineer, CA Network and Voice Management Test Automation. Each shared specific case studies, authored by them, about topics central to a chapter of each book. I really appreciate their valuable, practitioner insights.

Please attribute all errors, omissions, mistakes, opinions, and bad jokes in this book solely to me.

In the realm of “without whom,” of course, I thank my parents, Rex, Sr. and Carolynn, for their love and support over the years. My greatest appreciation goes to my wife and business partner, Laurel Becker. Managing the Testing Process has taken me away from a lot of things in my life, three times now, but I especially appreciate my wife's support in terms of her own time given up for me.

I've changed a few of my ideas since I wrote the first and second editions, but the biggest changes in my life have involved the arrival of my daughters. Along with having a burst of wisdom that led me to marry Laurel, I have to say that Emma Grace and Charlotte Catherine are the greatest things to happen in my life. All parents have dreams for their children's success, and I hope that my two beautiful and inspirational daughters have the same luck and success in their careers that I have had. Whatever Emma and Charlotte choose to do, this book is dedicated to them, with the utmost of a father's love.

Introduction

So, you are responsible for managing a computer hardware or software test project? Congratulations! Maybe you've just moved up from test engineering or moved over from another part of the development team, or maybe you've been doing test projects for a while. Whether you are a test manager, a development manager, a technical or project leader, or an individual contributor with some level of responsibility for your organization's test and quality assurance program, you're probably looking for some ideas on how to manage the unique beast that is a test project.

This book can help you. The first edition, published in 1999, and the second edition, published in 2002, have sold over 35,000 copies in the last decade. There are popular Indian, Chinese, and Japanese editions, too. Clients, colleagues, readers, training attendees, and others have read the book, writing reviews and sometimes sending helpful emails, giving me ideas on how to improve and expand it. So, thanks to all of you who read the first and second editions, and especially to those who have given me ideas on how to make this third edition even better.

This book contains what I wish I had known when I moved from programming and system administration to test management. It shows you how to develop some essential tools and apply them to your test project. It offers techniques that can help you get and use the resources you need to succeed. If you master the basic tools, apply the techniques to manage your resources, and give each area just the right amount of attention, you can survive managing a test project. You'll probably even do a good job, which might make you a test project manager for life, like me.

The Focus of This Book

I've written Managing the Testing Process for several reasons. First, many projects suffer from a gap between expectations and reality when it comes to delivery dates, budgets, and quality, especially between the individual contributors creating and testing the software, the senior project managers, and the users and the customers. Similarly, computer hardware development projects often miss key schedule and quality milestones. Effective testing and clear communication of results as an integrated part of a project risk management strategy can help.

Second, when I wrote the first edition, there was a gap in the literature on software and hardware testing. We had books targeting the low-level issues of how to design and implement test cases, as well as books telling sophisticated project managers how to move their products to an advanced level of quality using concepts and tools such as the Capability Maturity Model, software quality metrics, and so forth. However, I believe that test managers like us need a book that addresses the basic tools and techniques, the bricks and mortar, of test project management. While there are now a number of books addressing test management, I believe this book remains unique in terms of its accessibility and immediate applicability to the first-time test manager while also offering guidance in how to incrementally improve a foundational test management approach. It also offers a proven approach that works for projects that include substantial hardware development or integration components.

The tips and tools offered in this book will help you plan, build, and execute a structured test operation. As opposed to the all-too-common ad hoc or purely reactive test project, a structured test operation is planned, repeatable, and documented, but preserves creativity and flexibility in all the right places. What you learn here will allow you to develop models for understanding the meaning of the myriad data points generated by testing so that you can effectively manage what is often a confusing, chaotic, and change-ridden area of a software or hardware development project. This book also shows you how to build an effective and efficient test organization.

To that end, I've chosen to focus on topics unique to test management in the development and maintenance environments. Because they're well covered in other books, I do not address two related topics:

Basic project management tools such as work-breakdown structures, Gantt charts, status reporting, and people management skills. As you move into management, these tools will need to be part of your repertoire, so I encourage you to search out project management books—such as the ones listed in the bibliography in Appendix D—to help you learn them. A number of excellent training courses and certifications currently exist for project management as well.

Computer hardware production testing. If your purview includes this type of testing, I recommend books by W. Edwards Deming, Kaoru Ishikawa, and J. M. Juran as excellent resources on statistical quality control, as well as Patrick O'Connor's book on reliability engineering; see the bibliography in Appendix D for details on books referenced here.

Software production, in the sense of copying unchanging final versions to distribution media, requires no testing. However, both hardware and software production often include minor revisions and maintenance releases. You can use the techniques described in this book to manage the smaller test projects involved in such releases.

The differences between testing software and hardware are well documented, which might make it appear, at first glance, that this book is headed in two directions. I have found, however, that the differences between these two areas of testing are less important from the perspective of test project management than they are from the perspective of test techniques. This makes sense: hardware tests software, and software tests hardware. Thus, you can use similar techniques to manage test efforts for both hardware and software development projects.

Canon or Cookbook?

When I first started working as a test engineer and test project manager, I was a testing ignoramus. While ignorance is resolvable through education, some of that education is in the school of hard knocks. Ignorance can lead to unawareness that the light you see at the end of the tunnel is actually an oncoming train. “How hard could it be?” I thought. “Testing is just a matter of figuring out what could go wrong, and trying it.”

As I soon discovered, however, the flaws in that line of reasoning lie in three key points:

The tasks involved in “figuring out what could go wrong, and trying it”—that is, in designing good test cases—are quite hard indeed. Many authors have written good books on test case engineering, particularly in the last two decades. Unfortunately, my university professors didn't teach about testing, even though Boris Beizer, Bill Hetzel, and Glenford Myers had all published on the topic prior to or during my college career. As software engineering enters its sixth decade, that has begun to change. However, even at prestigious universities the level of exposure to testing that most software-engineers-in-the-making receive remains too low.Testing does not go on in a vacuum. Rather, it is part of an overall project—and thus testing must respond to real project needs, not to the whims of hackers playing around to see what they can break. In short, test projects require test project management.The prevalence of the “how hard can testing be” mindset only serves to amplify the difficulties that testing professionals face. Once we've learned through painful experience exactly how hard testing can be, it sometimes feels as if we are doomed—like a cross between Sisyphus and Dilbert—to explain, over and over, on project after project, why this testing stuff takes so long and costs so much money.

Implicit in these points are several complicating factors. One of the most important is that the capability of an organization's test processes can vary considerably: testing can be part of a repeatable, measured process, or an ad hoc afterthought to a chaotic project. In addition, the motivating factors—the reasons why management bothers to test—can differ in both focus and intensity. Managers motivated by fear of repeating a recent failed project see testing differently than managers who want to produce the best possible product, and both motivations differ from those of people who organize test efforts out of obligation but assign them little importance. Finally, testing is tightly connected to the rest of the project, so the test manager is often subject to a variety of outside influences. These influences are not always benign when scope and schedule changes ripple through the project.

These factors make it difficult to develop a how to guide for planning and executing a test project. As academics might say, test project management does not lend itself to the easy development of a canon. “Understand the following ideas and you can understand this field” is a difficult statement to apply to test management. And the development of a testing canon is certainly not an undertaking I'll tackle in this book.

Do you need a canon to manage test projects properly? I think not. Instead, consider this analogy: I am a competent and versatile cook, an amateur chef. I will never appear in the ranks of world-renowned chefs, but I regularly serve passable dinners to my family. I have successfully prepared a number of multicourse Thanksgiving dinners, some in motel kitchenettes. I mastered producing an edible meal for a reasonable cost as a necessity while working my way through college. In doing so, I learned how to read recipes out of a cookbook, apply them to my immediate needs, juggle a few ingredients here and there, handle the timing issues that separate dinner from a sequence of snacks, and play it by ear.

An edible meal at a reasonable cost is a good analogy for what your management wants from your testing organization. This book, then, can serve as a test project manager's cookbook, describing the basic tools you need and helping you assemble and blend the proper ingredients.

The Tools You Need

Several basic tools underlie my approach to test management:

A solid quality risk analysis. You can't test everything. Therefore, a key challenge to test management is deciding what to test. You need to find the important bugs early in the project. Therefore, a key challenge to test management is sequencing your tests. You sometimes need to drop tests due to schedule pressure. Therefore, a key challenge to test management is test triage in a way that still contains the important risks to system quality. You need to report test results in terms that are meaningful to non-testers. Therefore, a key challenge to test management is tracking and reporting residual levels of risk as test execution continues. Risk based testing, described in this book, will help you do that.

A thorough test plan. A detailed test plan is a crystal ball, allowing you to foresee and prevent potential crises. Such a plan addresses the issues of scope, quality risk management, test strategy, staffing, resources, hardware logistics, configuration management, scheduling, phases, major milestones and phase transitions, and budgeting.

A well-engineered system. Good test systems ferret out, with wicked effectiveness, the bugs that can hurt the product in the market or reduce its acceptance by in-house users. Good test systems mitigate risks to system quality. Good test systems build confidence when the tests finally pass and the bugs get resolved. Good test systems also produce credible, useful, timely information. Good test systems possess internal and external consistency, are easy to learn and use, and build on a set of well-behaved and compatible tools. I use the phrase good test system architecture to characterize such a system. The word architecture fosters a global, structured outlook on test development within the test team. It also conveys to management that creating a good test system involves developing an artifact of elegant construction, with a certain degree of permanence.

A state-based bug tracking database. In the course of testing, you and your intrepid test team will find lots of bugs, a.k.a. issues, defects, errors, problems, faults, and other less-printable descriptions. Trying to keep all these bugs in your head or in a single document courts immediate disaster because you won't be able to communicate effectively within the test team, with programmers, with other development team peers, or with the project management team—and thus won't be able to contribute to increased product quality. You need a way to track each bug through a series of states on its way to closure. I'll show you how to set up and use an effective and simple database that accomplishes this purpose. This database can also summarize the bugs in informative charts that tell management about projected test completion, product stability, system turnaround times, troublesome subsystems, and root causes.

A comprehensive test-tracking spreadsheet. In addition to keeping track of bugs, you need to follow the status of each test case. Does the operating system crash when you use a particular piece of hardware? Does saving a file in a certain format take too long? Which release of the software or hardware failed an important test? A simple set of worksheets in a single spreadsheet can track the results of every single test case, giving you the detail you need to answer these kinds of questions. The detailed worksheets also roll up into summary worksheets that show you the big picture. What percentage of the test cases passed? How many test cases are blocked? How long do the test suites really take to run?

A simple change management database. How many times have you wondered, “How did our schedule get so far out of whack?” Little discrepancies such as slips in hardware or software delivery dates, missing features that block test cases, unavailable test resources, and other seemingly minor changes can hurt. When testing runs late, the whole project slips. You can't prevent test-delaying incidents, but you can keep track of them, which will allow you to bring delays to the attention of your management early and explain the problems effectively. This book presents a simple, efficient database that keeps the crisis of the moment from becoming your next nightmare.

A solid business case for testing. What is the amount of money that testing saves your company? Too few test managers know the answers to this question. However, organizations make tough decisions about the amount of time and effort to invest in any activity based on a cost benefit analysis. I'll show you how to analyze the testing return on investment, based on solid, well established quality management techniques.

This book shows you how to develop and apply these basic tools to your test project, and how to get and use the resources you need to succeed. I've implemented them in the ubiquitous PC-based Microsoft Office suite: Excel, Word, Access, and Project. You can easily use other office-automation applications, as I haven't used any advanced features.

The Resources You Need

In keeping with our culinary analogy, you also need certain ingredients, or resources, to successfully produce a dish. In this testing cookbook, I show you how I assemble the resources I need to execute a testing project. These resources include some or all of the following:

A practical test lab. A good test lab provides people—and computers—with a comfortable and safe place to work. This lab, far from being Quasimodo's hideout, needs many ways to communicate with the development team, the management, and the rest of the world. You must ensure that it's stocked with sufficient software and hardware to keep testers working efficiently, and you'll have to keep that software and hardware updated to the right release levels. Remembering that it is a test lab, you'll need to make it easy for engineers to keep track of key information about system configurations.

Test engineers and technicians. You will need a team of hardworking, qualified people, arranged by projects, by skills, or by a little of both. Finding good test engineers can be harder than finding good development engineers. How do you distinguish the budding test genius from that one special person who will make your life as a manager a living nightmare of conflict, crises, and lost productivity? Sometimes the line between the two is finer than you might expect. And once you have built the team, your work really begins. How do you motivate the team to do a good job? How do you defuse the land mines that can destroy motivation?

Contractors and consultants. As a test manager, you will probably use outsiders, hired guns who work by the hour and then disappear when your project ends. I will help you classify the garden-variety high-tech temporary workers, understand what makes them tick, and resolve the emotional issues that surround them. When do you need a contractor? What do contractors care about? Should you try to keep the good ones? How do you recognize those times when you need a consultant?

External test labs, testing services providers, and vendors. In certain cases, it makes sense to do some of the testing outside the walls of your own test lab—for instance, when you are forced to handle spikes or surprises in test workloads. You might also save time and money by leveraging the skills, infrastructure, and equipment offered by external resources such as testing labs and testing services providers. What can these labs and vendors really do for you? How can you use them to reduce the size of your test project without creating dangerous coverage gaps? How do you map their processes and results onto yours? How does outsourcing fit into your test effort?

Of course, before you can work with any of these resources, you have to assemble them. As you might have learned already, management is never exactly thrilled at the prospect of spending lots of money on equipment to test stuff that—in their view—“ought to work anyway.” With that in mind, I've also included some advice about how to get the green light for the resources you really need.

On Context

I've used these tools and techniques to manage projects large and small. The concepts scale up and down easily, although on larger projects it might pay to implement some of the tools in a more automated fashion. In that case, the tools I've described here can be prototypes or serve as a source of requirements for the automated tools you buy or build.

The concepts also scale across distributed projects. I've used the tools to manage multiple projects simultaneously from a laptop computer in hotel rooms and airport lounges around the world. I've used these tools to test market-driven end-user systems and in-house information technology projects. While context does matter, I've found that adaptations of the concepts in this book apply across a broad range of settings.

Simple and effective, the tools incorporate the best ideas from industry standards such as the IEEE 829 Standard for Software and System Test Documentation and bring you in line with the best test management practices and tools at leading software and hardware vendors. I use these tools to organize my thinking about my projects, to develop effective test plans and test suites, to execute the plans in dynamic high-technology development environments, and to track, analyze, and present the results to project managers. Likewise, my suggestions on test resource management come from successes and failures at various employers and clients.

Because context matters, the final two chapters discuss the importance of fitting the testing process into the overall development or maintenance process. This involves addressing issues such as organizational context, the economic aspects of and justifications for testing, life cycles and methodologies for system development, and increasing test process capability, including test process assessment and maturity models.

Using This Book

Nothing in this book is based on Scientific Truth, double-blind studies, academic research, or even flashes of brilliance. It is merely about what has worked—and continues to work—for me on the dozens of test projects I have managed, what has worked for the clients that my company, RBCS, has the good fortune to serve, and what has worked for the thousands of people who have attended RBCS training courses. You might choose to apply these approaches as is, or you might choose to modify them. You might find all or only some of my approaches useful.

Along similar lines, this is not a book on the state of the art in test techniques, test theory, or the development process. This is a book on test management, both hardware and software, as I have practiced it. In terms of development processes—best practices or your company's practices—the only assumption I make is that you as the test manager became involved in the development project with sufficient lead time to do the necessary test development. Chapter 12 addresses different development processes I have seen and worked within. I cover how the choice of a development life cycle affects testing.

Of course, I can't talk about test management without talking about test techniques to some extent. Because hardware and software test techniques differ, you might find some of the terms I use unclear or contradictory to your usage of them. I have included a glossary to help you decipher the hardware examples if you're a software tester, and vice versa. Finally, the test manager is usually both a technical leader and a manager, so make sure you understand and use best practices, especially in the way of test techniques, for your particular type of testing. Appendix D includes a listing of books that can help you brush up on these topics if needed.

This book is drawn from my experiences, good and bad. The bad experiences—which I use sparingly—are meant to help you avoid some of my mistakes. I keep the discussion light and anecdotal. The theory behind what I've written, where any exists, is available in books listed in the bibliography in Appendix D.

I find that I learn best from examples, so I have included lots of them. Because the tools I describe work for both hardware and software testing, I base many examples on one of these two hypothetical projects:

Most software examples involve the development of a browser-based word-processing package named SpeedyWriter, being written by Software Cafeteria, Inc. SpeedyWriter has all the usual capabilities of a full-featured word processor, plus network file locking, Web integration, and public-key encryption. SpeedyWriter includes various add-ins from other vendors.1Most hardware examples refer to the development of a server named DataRocket, under development by Winged Bytes, LLP. DataRocket is intended to serve a powerful, high-end database, application, and Web server. It runs multiple operating systems. Along with third-party software, Winged Bytes plans to integrate hardware from vendors around the world.

As for the tools discussed in this book, you can find examples of these at www.rbcs-us.com. These include templates and case studies from real projects. In those chapters that describe the use of these tools, I include information to guide you in the use and study of these templates and case studies should you want to do so. That way, you can use these resources to bootstrap your own implementation of the tools. These tools are partially shown in figures in the chapters in which I describe them. However, screen shots can only tell you so much. Therefore, as you read the various chapters, you might want to open and check out the corresponding case studies and templates from the Web site to gain a deeper understanding of how the tools work.

Please note that the tools supplied with the book are usable, but contain only small amounts of dummy data. This data should not be used to derive any rules of thumb about bug counts, defect density, predominant quality risks, or any other metric to be applied to other projects. I developed the tools primarily to illustrate ideas, so some of the sophisticated automation that you would expect in a commercial product won't be there. If you intend to use these tools in your project, allocate sufficient time and effort to adapt and enhance them for your context. For large, complex projects, or for situations where test management is an ongoing activity, you'll want to consider buying commercial tools.

For those wanting to practice with the tools before putting them into use on a real project, I have included exercises at the end of each chapter. For many of these exercises, you can find solutions at www.rbcs-us.com. These exercises make this book suitable as the test management textbook for a course on testing, software engineering, or software project management. Given that testing is increasingly seen by enlightened project managers as a key part of the project's risk management strategy, including material such as this as part of a college or certification curriculum makes good sense.

Finally—in case you haven't discovered this yet—testing is not a fiefdom in which one's cup overfloweth with resources and time. I have found that it's critical to focus on testing what project managers really value. Too often in the past I've ended up wrong-footed by events, spending time handling trivialities or minutiae while important matters escaped my attention. Those experiences taught me to recognize and attend to the significant few and ignore the trivial many. The tools and techniques presented here can help you do the same, especially the risk-based testing elements. A sizeable number of test groups are disbanded in their first couple of years. This book will help keep you out of that unfortunate club.

Although it's clearly more than simply hanging onto a job, success in test management means different things to different people. In my day-to-day work, I measure the benefits of success by the peace of mind, the reduction in stress, and the enhanced professional image that come from actually managing the testing areas in my purview rather than reacting to the endless sequence of crises that ensue in ad hoc environments. I hope that these tools and ideas will contribute to your success as a testing professional.

What's New and Changed in the Third Edition?

For those of you who read the second edition and are wondering whether to buy this third edition, I've included the following synopsis of changes and additions:

I've split the final chapter into two detailed chapters on the importance of fitting the testing process into the overall development or maintenance process. I address organizational context, the economic aspects of and justifications for testing, life cycles and methodologies for system development, test process assessment, and process maturity models.I have addressed the IEEE 829-2008 standard, which came out as I started work on this book. This new version of the IEEE 829 standard includes not only document templates, but also discussion on the testing process. While I'm not endorsing the complete adoption of this standard on your projects, I believe it does provide useful ideas and food for thought.I also added some new metrics. The templates include the tools to generate those metrics. Some of the templates originally published with the book, while usable, contained minor errors. Readers of the first and second editions—being test professionals—caught and pointed out these errors to me. I have corrected those mistakes.In addition to case studies, I have added some exercises. Some of these come from RBCS's live and e-learning course “Managing the Testing Process,” some carried over from the second edition, and some are adapted from Pragmatic Software Testing. You can use these exercises for self-study, as part of a book club, or for classroom education. (Some professors have selected this book as a textbook for a software testing course.) Solutions to many of these exercises are now available at www.rbcs-us.com.Finally, little has changed in terms of the challenges facing those who manage test projects since I wrote the first and second editions. Every time my associates teach RBCS's “Managing the Testing Process” classes, which are drawn directly from this book, at least one attendee tells me, “It's amazing how every issue we've talked about here in class is something that has come up for me on my projects.” However, I have learned some new tricks and broadened my mind. For example, Agile project methodologies are now quite popular, so I've incorporated material to discuss the challenges that Agile techniques pose for testing and how you can manage these challenges.

If you read the second edition, enjoyed it, and found it useful, I think these changes and additions will make this third edition even more useful to you.

1 When I wrote the first edition and used this same example, a browser-based word processor might have struck readers as a bizarre concept. Well, to those at Google, I say, “You're welcome” for the idea!

Chapter 1

Defining What's on Your Plate: The Foundation of a Test Project

Testing requires a tight focus. It's easy to try to do too much. You could run an infinite number of tests against any nontrivial piece of software or hardware. Even if you try to focus on what you think might be “good enough” quality, you can find that such testing is too expensive or that you have trouble figuring out what “good enough” means for your customers and users. Before I start to develop the test system—the testware, the test environment, and the test process—and before I hire the test team, I figure out what I might test, then what I should test, and finally what I can test. Determining the answers to these questions helps me plan and focus my test efforts.

What I might test are all those untested areas that fall within the purview of my test organization. On every project in which I've been involved, some amount of the test effort fell to organizations outside my area of responsibility. Testing an area that another group already covered adds little value, wastes time and money, and can create political problems for you.

What I should test are those untested areas that directly affect the customers' and users' experience of quality. People often use buggy software and computers and remain satisfied nevertheless. Either they never encounter the bugs or the bugs don't significantly hinder their work. Our test efforts should focus on finding the critical defects that will limit people's ability to get work done with our products.

What I can test are those untested, critical areas on which my limited resources are best spent. Can I test everything I should? Not likely, given the schedule and budget I usually have available.1 On most projects, I must make tough choices, using limited information, on a tight schedule. I also need to sell the test project to my managers to get the resources and the time I need.

What You Might Test: The Extended Test Effort

On my favorite software and system projects, testing was pervasive. By this, I mean that a lot of testing went on outside the independent test team. In addition, testing started early. This arrangement not only made sense technically, but also kept my team's workload manageable. This section uses two lenses to examine how groups outside the formal test organization contribute to testing. The first lens is the level of focus—the —of a test. The second is the type of testing performed within various test phases. Perhaps other organizations within your company could be (or are) helping you test.