Pragmatic Software Testing E-Book

Rex Black

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 465

Veröffentlichungsjahr: 2011

Beschreibung

A hands-on guide to testing techniques that deliver reliable software and systems Testing even a simple system can quickly turn into a potentially infinite task. Faced with tight costs and schedules, testers need to have a toolkit of practical techniques combined with hands-on experience and the right strategies in order to complete a successful project. World-renowned testing expert Rex Black provides you with the proven methods and concepts that test professionals must know. He presents you with the fundamental techniques for testing and clearly shows you how to select and apply successful strategies to test a system with budget and time constraints. Black begins by discussing the goals and tactics of effective and efficient testing. Next, he lays the foundation of his technique for risk-based testing, explaining how to analyze, prioritize, and document risks to the quality of the system using both informal and formal techniques. He then clearly describes how to design, develop, and, ultimately, document various kinds of tests. Because this is a hands-on activity, Black includes realistic, life-sized exercises that illustrate all of the major test techniques with detailed solutions.

Leseprobe

Table of Contents

Cover

Title Page

Dedication

Copyright

About the Author

Credits

Preface

Acknowledgments

Introduction

What Kind of Book Is This?

What Topics Will I Cover?

Can I Skip Topics?

Can I Practice with Realistic Exercises?

Does It Matter That I Have (or Haven’t) Read Another Book on Testing?

Part I: Goals, Strategies, and Tactics

Chapter 1: What Does It Mean to Be Pragmatic?

Chapter 2: Triangle Test Exercise

Exercise: The Triangle Test

Your Solution for the Triangle Test

Author’s Triangle Test Solution

Chapter 3: Aligning Testing with the Project

Why Do Organizations Test?

Perspectives on Testing

Testing in Context

Common Test Phases and Objectives

Testing Throughout the Organization

The V Model

Evolutionary and Incremental Models

The Spiral Model

Regrettably Common Model: Code and Fix

Testing Maintenance Releases

System Integration

Hardware/Software Development

The Test Process

Chapter 4: Understanding Test Strategies, Tactics, and Design

Aligning Mission, Strategies, and Tactics

Analytical Test Strategies

Model-Based Test Strategies

Methodical Test Strategies

Process-Oriented Test Strategies

Dynamic Test Strategies

Philosophical Test Strategies

Regression

Three Other Regression Strategies

Tactics: Categories of Testing Techniques

Strategic and Tactical Considerations

The Test System

Classic Principles for Test Design

Phases of Test Development

Synopsis of Test Strategies, Tactics, and Design

Part II: Risk-Based Testing

Chapter 5: Understanding Risks to System Quality

Categories of Quality Risks

Transactions

Regression

Can You Think of Other Quality Risks?

Chapter 6: Aligning Testing with Quality Risks

Prioritize Risks to System Quality

Testing, Customer Usage, and System Configurations

Approaches for Quality Risks Analysis

Informal Quality Risk Analysis

Tips for Risk Analysis

Challenges of Risk Analysis

Chapter 7: Quality Risk Analysis Exercise

My Solution

Comments on My Solution

A Funny Thing Happened to Me on the Way to the Quality Risks…

Bonus Exercise

Template

My Solution

Comments on My Solution

Bonus Example: Grays and Blues Quality Risk Analysis

Part III: Static Testing

Chapter 8: Reviews and Other Static Tests

Testing Requirements and Designs

Reviews Costs and Benefits

Types of Reviews

Reviews for Verification and Validation

Reviews as a Way to Improve the Process, Improve the System, and Reduce Costs

Reviews as a Way to Achieve Consensus and Understanding

The Review Process, Roles, and Responsibilities

Deliverables and Ground Rules from Reviews

Common Requirements and Design Bugs

Reviewing (and Testing) Documentation

Other Static Tests

Answers to the Three Questions about Static Testing

Chapter 9: Review Exercise

Reviews

Your Solution

My Solution

Comments on My Solution

Bonus Exercise: Reviewing Triangle Requirements Title Page

The Requirements Bugs You Found

My Solution: Requirements Bugs

Comments on My Solution

Part IV: Behavioral Testing

Chapter 10: Equivalence Classes and Boundary Values

Equivalence Partitioning

Boundary Value Analysis

Chapter 11: Equivalence Classes and Boundary Values Exercise

Functional Boundaries and Classes

My Solution and Comments

Nonfunctional Boundaries and Classes

Chapter 12: Use Cases, Live Data, and Decision Tables

Use Cases and Scenario Tests

Nouns and Verbs, Adjectives and Adverbs

Live Data and Customer Tests

Decision Tables

Chapter 13: Decision Table Exercise

Decision Table Tests

My Solution and Comments

Decision Tables and Boundary Values

My Solution and Comments

Building a Decision Table for Testing

My Solution and Comments

Chapter 14: State Transition Diagrams

Describing Systems with States Using State Transition Diagrams

State Tables

Printer Server State Transition Diagram

Chapter 15: State Transition Diagram Exercise

Kiosk States

My Solution and Comments

ATM State Models

My Solution and Comments

Grays and Blues and State Transition Diagrams

My Solution and Comments

Chapter 16: Domain Testing

Combinatorial Explosions

A Domain Example Using Frequent-Flyer Programs

Possible Domain Test Values

An Aerospace Example

When Domain Rules Change

Domain Analysis Summary

Complex Domain Testing Example

A General Rule for Complex Domains

Chapter 17: Domain Testing Exercise

My Solution and Comments

Chapter 18: Orthogonal Arrays and All Pairs

Combinatorial Explosions

Orthogonal Arrays and All-Pairs Tables

Two Orthogonal Arrays

Selecting an Orthogonal Array

Applying Orthogonal Arrays to a Complicated Real-World Example

All-Pairs Tables

Other Thoughts on Configuration Testing

Chapter 19: Orthogonal Arrays Exercise

My Solution and Comments

Chapter 20: Reactive Testing

General Facts about Reactive Tests

Error Guessing, Attacks, and Bug Taxonomies

Bug Hunting

Exploratory Tests

Checklists

Other Sources of Inspiration for Reactive Testing

Advantages and Disadvantages

A Case Study of Exploratory Testing

Part V: Structural Testing

Chapter 21: Control-Flow Testing

Code Coverage

McCabe Cyclomatic Complexity

Chapter 22: Control-Flow Testing Exercise

My Solution and Comments

Chapter 23: Data-Flow Testing

Chapter 24: Data-Flow Testing Exercise

My Solutions and Comments

Chapter 25: Integration Testing

Drivers and Stubs

Integration Techniques

Backbone Integration

McCabe Basis Paths for Integration

Enhanced Hex Converter Program

Call Flow

Chapter 26: Integration Basis Test Exercise

My Solution and Comments

Appendixes

Appendix A: Omninet: The Internet EverywhereMarketing Requirements Document

1 Scope

2 Required release date

3 Description of requirements

Appendix B: Omninet:The Internet Everywhere System Requirements Document

Functionality System Requirements

Reliability System Requirements

Usability System Requirements

Efficiency System Requirements

Maintainability System Requirements

Portability System Requirements

Design Models

Appendix C: Bibliography and Other Recommended Readings

RBCS Company Profile

Index

To Laurel, Emma, and Charlotte

Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional

Published byWiley Publishing, Inc.10475 Crosspoint BoulevardIndianapolis, IN 46256www.wiley.com

Copyright © 2007 by Rex BlackPublished by Wiley Publishing, Inc., Indianapolis, IndianaPublished simultaneously in Canada

ISBN: 978-0-470-12790-2

Manufactured in the United States of America

10 9 8 7 6 5 4 3 2 1

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone]. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Website may provide or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services or to obtain technical support, please contact our Customer Care Department within the U.S. at [phone], outside the U.S. at [phone] or fax [phone].

Library of Congress Cataloging-in-Publication Data is available from the publisher.

Trademarks: Wiley, the Wiley logo, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

About the Author

With a quarter century of software and systems engineering experience, Rex Black is president and principal consultant of RBCS, Inc., a leader in software, hardware, and systems testing. For over a dozen years, RBCS has served its worldwide clientele with consulting, outsourcing, assessment, and training services related to testing and quality assurance. RBCS has over 100 clients spanning 20 countries on 6 continents, including Adobe (India), ASB Bank, Bank One, Cisco, Comverse, Dell, the U.S. Department of Defense, Hitachi, NDS, and Schlumberger.

With four books to his credit, Rex is the most prolific author currently practicing in the field of testing and quality assurance today. His popular first book, Managing the Testing Process, now in its second edition, has sold 25,000 copies around the world, including Japanese, Chinese, and Indian releases. His other book on test management, Critical Testing Processes, along with previous editions of this book, marketed as Effective and Efficient Software Testing, have also sold thousands of copies, including Hebrew, Indian, Japanese, and Russian editions.

Rex is the president of both the International Software Testing Qualifications Board (www.istqb.org) and the American Software Testing Qualifica-tions Board (www.istqb). Being a primary coauthor of both the current Foundation syllabus (version 2005) and the upcoming Advanced syllabus (version 2007), he was well qualified to coauthor the definitive text for ISTQB certification candidates, Foundations of Software Testing, with Isabel Evans, Dorothy Graham, and Erik van Veenendaal.

In addition to books, Rex has written over 25 articles; presented hundreds of papers, workshops, and seminars; and given over a dozen keynote speeches at conferences and events around the world.

When Rex is not traveling the world for work or vacation, he lives in Bulverde, Texas, with his wife, Laurel Becker; his daughters, Emma Grace and Charlotte Catherine; and his dogs, Cosmo and Hank.

Credits

Senior Acquisitions Editor

Jim Minatel

Development Editor

Maureen Spears

Production Editor

Martine Dardignac

Copy Editor

Judy Flynn

Editorial Manager

Mary Beth Wakefield

Production Manager

Tim Tate

Vice President and Executive Group Publisher

Richard Swadley

Vice President and Executive Publisher

Joseph B. Wikert

Compositor

Craig Woods, Happenstance Type-O-Rama

Proofreader

Kathryn Duggan

Indexer

Jack Lewis

Anniversary Logo Design

Richard Pacifico

Preface

Testing even a simple system is a potentially infinite task. With tight budgets and schedules, testers need practical techniques, hands-on experience and the right strategies to effectively and efficiently test software.

This book puts those things right in the palm of your hands, literally. Through a sequence of thorough, practical, and, I hope, well-explained chapters, you’ll learn the following skills that are critical to software testing:

How to analyze the risks to system quality and allocate your testing effort appropriately based on the level of risk.Different testing strategies and how to choose the right strategies every time, including effective strategies for handling regression testing.How to design tests based on a system’s expected behavior (black box), including boundary values, equivalence partitioning, decision tables, use cases, state diagrams and tables, all-pairs tables, orthogonal arrays, and domain analysisHow to design tests based on a system’s internal structure (white box), including levels of code coverage, data-flow coverage, and basis-path coverageHow to plan and perform integration testingHow to use your intuition, experience, and knowledge to explore and attack the systemHow to make all your hard work serve the needs of the project

Because testing is a hands-on activity, this book includes 11 complete chapters of realistic, life-sized exercises illustrating all the major test techniques, with detailed solutions.

If you’ve never read a book on test design, if you’ve read other books on test design and found them too hard to follow, or if you’ve read a book on test design and found it stopped just when things got really interesting, this book is for you. By the end of this book, you will know more about the nuts and bolts of testing than most testers learn in an entire career, and you will be ready to put those ideas into action on your next test project.

Acknowledgments

If you’re reading this because you just bought this book, it’s only right for me to start by saying thanks to you. Ultimately, readers are the reason people such as me write books like this one. I hope I repay the favor you are doing me as a reader of my work by teaching you some new and useful testing techniques as you read this book.

This book grew from training materials that date back to the mid-1990s. Therefore, I would like to thank all the students all around the world who took my courses, who number in the thousands, for their help in improving the base material of this book.

The book itself grew by a circuitous route, from an e-learning project that never quite made it to delivery. I wrote the scripts and was ready to record the audio tracks, but then things fizzled. Having done about four or five successful e-learning projects, I wasn’t going to write off the work on this one, so those scripts became the first draft of this book. As the saying goes, “Success has a thousand fathers, while failure is an orphan.” So I won’t name names, but if you’re reading this, thanks for the push that made this book happen.

After this became a book, a number of people reviewed the material. In no particular order, my thanks go out to Judy McKay, Mitsuaki Narita, Barton Layne, Julie Gardiner, Michael Bolton, Mikhail Pavlov, Joe Mata, and Jamie Mitchell for their thoughts.

In another interesting twist, this book happens to be the first book on testing published in Hebrew. I would like to thank David Bassa for pushing the deal forward, Alon Bar-Shamai for the legwork, Tal Pe’er for his insightful comments and questions during the review of the Hebrew translation, and the rest at SELA who helped to make this happen. Toda raba to my friends and colleagues in Israel!

I would also like to thank Charles Arnao, Michael Caruso, and the rest of the team at Bisk Education and Villanova University for selecting this book as a text for our (successful) e-learning project “Essentials of Software Testing.” In addition, thanks go out to Professor Charles Pfohl of the University of Canberra for his use of this text for his course on testing there. Finally, thanks to Noel LeJeune of the Metropolitan State College of Denver for selecting this book as a text for his course too.

This book started its wider life in the United States as a self-published, spiral-bound, soft-copy beast sold on Amazon. I didn’t knock any dents in Dan Brown’s royalty stream with Effective and Efficient Software Testing (this book’s name at that time), but a number of folks were kind enough to buy it. I thank each of you, especially those of you who were even more kind and sent comments and suggestions.

Jim Minatel, my editor at Wiley, has worked with me for years on one of my previous books, Managing the Testing Process. In the years that I polished this book in its various forms, I would go back to Jim from time to time and ask him if Wiley was ready to publish it. A number of “not yets” has finally become “yes,” and I thank Jim for his efforts to bring this book to a wider audience.

Of course, while all these readers, students, and reviewers contributed ideas, thoughts, and opinions, I made the final call about what I would write, positions I would adopt, and jokes I would make. So, please hold me responsible for the content.

Thanks to my wife and business partner, Laurel Becker, for all her help. Self-publishing a book is an interesting experience, and I’m sure it was especially interesting for Laurel. From getting ISBNs to setting up the printing to arranging a listing on Amazon.com, among untold other contributions, thanks for your help, love, and support, on this project as on so many others, and in my life.

Last but not least, thanks to my charming, hilarious, and loving daughters, Emma and Charlotte, and my equally hilarious (though somewhat less charming) dogs, Cosmo and Hank, for providing amusement, friendship, and a wet nose (the dogs, not the girls) when requested — and when not requested. Which makes me realize that every child needs chores. My father made me mow the lawn when I was a kid: I wonder if I can have Emma and Charlotte write the next book?

Introduction

What Kind of Book Is This?

This is a book about software and system testing. This is an ambitious book. In it I cover the strategies, techniques, and concepts that effective and efficient test professionals need to do their job. That covers a lot of ground, and so does this book.

This book is about practical concepts. This book is hands-on. If you work your way through the whole book, you’ll do many realistic exercises to apply these concepts immediately. You’ll be able to compare your solutions with mine.

Appropriately enough, this book is tested. I have used these concepts in my career as a test professional, which began in 1987, four years after I started my software career in a Fortran and C programming job. Since 1997, literally thousands of software and systems professionals around the world have taken the training courses that provide the base material for this book. We have discussed the concepts and worked through the exercises.

What Topics Will I Cover?

In Part I, I discuss the goals, strategies, and tactics of effective and efficient testing. Even experienced testers will find something new here, and I encourage people who are new to testing to work through these chapters completely.

In Part II, I lay the foundation of my technique for risk-based testing. You’ll learn how to analyze, prioritize, and document risks to the quality of the system, using both informal and formal techniques. Unless you are already an experienced risk-analysis professional, I recommend that you work carefully through this section, including the exercise.

In the heart of the book, with the goals of testing defined through quality risk analysis, you’ll start to fill your testing toolkit. In Parts III, IV, and V, you’ll learn to design, develop, and, ultimately, document various kinds of tests. You’ll learn static, black-box, and white-box test techniques, including the following:

Requirements, design, and code reviewsEquivalence classes and boundary value analysisDecision tablesLive data and customer workflow testingState-transition diagramsDomain testingOrthogonal arraysStatement, branch, condition, and loop code coverageMcCabe complexity and unit basis testsData-flow coverageIntegration test techniquesMcCabe integration basis tests

These are fundamental test techniques, the knowledge of which distinguishes the test professional from the part-timer or amateur tester. I suggest that you work through all of these chapters, even if you are an experienced tester, including the exercises. If you’ve already mastered these topics, this material and the exercises should be easy for you, but you might find new nuances in these chapters. I know I did as I was preparing them.

Part VI has Omninet Marketing and Systems Requirements Documents as well as a bibliography and suggestions for further reading.

Can I Skip Topics?

If you feel that one or two major test techniques are inapplicable to you, feel free to skip them. For example, you might work in a group focused entirely on black-box techniques. You can skip the sections on static and white-box testing. The section on black-box testing stands on its own, and each technique can be studied independently, too. Similarly, you could go through static testing and black-box testing and skip white-box testing. It’s up to you.

While all are fundamental test techniques, they are not all of the same degree of applicability. I would group them into three categories:

Generally applicable — equivalence classes, boundary values, reviews, code coverage, and integration test techniquesOften applicable — decision tables, state-transition diagrams, live data and customer workflows, McCabe Cyclomatic Complexity, and orthogonal arraysSometimes applicable — domain analysis, data-flow coverage, and McCabe integration basis tests

You certainly could decide to study only the generally and often applicable techniques if that’s what you feel you need to learn right now. The material is designed to be modular and selectively usable.

However, if your goal is to be a well-rounded test professional, you need to be familiar with all these major test techniques, not just for your current job, but for your future ones as well. Should you want to pursue one of the major test certifications, such as the Foundation or Advanced certificates granted by National Boards of the International Software Testing Qualifications Board, you’ll be tested on most of these concepts.

At one point or other in my 20-plus-year career as a test professional and software engineer, each topic covered in this book has been important and practical. As we go, I’ll point out why I find these topics important, oftentimes with anecdotes from projects I’ve worked on or heard about from credible sources. As the saying goes, “Any fool can learn from his own mistakes [and I hope I have], but the wisest amongst us can learn from the mistakes of others.” So, I’ll share not just success stories, but also some cautionary tales.

Can I Practice with Realistic Exercises?

This book uses a lifelike project, Omninet, to structure many of the exercises. Omninet is a project to deploy a network of public access Internet kiosks in places like malls, theaters, and other public places. Users will be able to buy Web surfing time using cash, debit cards, or credit cards. The realism and complexity of this hypothetical project will give you a chance to try out many of the test concepts we talk about, especially the test design techniques.

The Marketing Requirements Document and the System Requirements Document are included in appendices of this book. I recommend that you review them before you start the first chapter.

Since Omninet wasn’t the perfect way to illustrate every concept, I’ve included a few other examples. I’ll explain those in the exercises when we get there.

In live courses, the time allocated for the exercises in the training materials is tightly constrained, usually between 30 and 120 minutes. This might at first seem artificial, but in reality, most of our work as test professionals is constrained by time (and often money) too. For this reason, I suggest time constraints for each exercise. These time constraints mean that you’ll have fit-and-finish issues in your solutions.

I’ve followed these constraints when preparing my solutions too. The fit-and-finish issues in my solutions indicate the level of fit-and-finish issues that I expect in a typical time-constrained test project. In real life, we don’t always need to — or have a chance to — polish our work, especially when it’s for our own internal use.

In many cases, more than one correct solution exists. So just because your solution differs from mine, that doesn’t make your solution — or mine — wrong. The differences might indicate differences in our assumptions about what’s important. If you get a solution different from mine, ask yourself what differences in perspectives and priorities might lead to those differences.

Does It Matter That I Have (or Haven’t) Read Another Book on Testing?

This book stands on its own, so you needn’t have read any other test books. If you have read my other books, Managing the Testing Process and Critical Testing Processes, there is very little overlap, except in the material on quality risk analysis. Even if you are familiar with my earlier writings on this topic, you’ll probably find some new ideas here.

If you have read other test design books, you will find new ideas on those topics too. I start with the basic ideas for each test design technique, but I go well beyond the basics, especially in my discussion of the exercise solutions.

Part I: Goals, Strategies, and Tactics

In this Part

Chapter 1: What Does It Mean to Be Pragmatic?

Chapter 2: Triangle Test Exercise

Chapter 3: Aligning Testing with the Project

Chapter 4: Understanding Test Strategies, Tactics, and Design

Chapter 1

What Does It Mean to Be Pragmatic?

Let’s start at the beginning by exploring some obvious questions with some not-so-obvious and not-so-universally-accepted answers about pragmatic testing. From a pragmatic, or practical, standpoint, it involves being effective and efficient when testing software. What is effective software testing? What is efficient software testing? What is software testing, anyway? What is quality?

While these might seem like impractical, philosophical questions, in my experience, they are not. Your answers to these questions determine what you expect to do as a tester. Other people’s answers to these questions determine what they expect you to do as a tester. Having common expectations up, down, and across the organizational chart and throughout the project team is essential to success. Without such commonality, no matter what you do, someone’s sure to be disappointed. With common expectations, you can all strive for the same goals, and support others in their endeavors.