Analytic Methods in Systems and Software Testing E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 1063

Veröffentlichungsjahr: 2018

Beschreibung

A comprehensive treatment of systems and software testing using state of the art methods and tools This book provides valuable insights into state of the art software testing methods and explains, with examples, the statistical and analytic methods used in this field. Numerous examples are used to provide understanding in applying these methods to real-world problems. Leading authorities in applied statistics, computer science, and software engineering present state-of-the-art methods addressing challenges faced by practitioners and researchers involved in system and software testing. Methods include: machine learning, Bayesian methods, graphical models, experimental design, generalized regression, and reliability modeling. Analytic Methods in Systems and Software Testing presents its comprehensive collection of methods in four parts: Part I: Testing Concepts and Methods; Part II: Statistical Models; Part III: Testing Infrastructures; and Part IV: Testing Applications. It seeks to maintain a focus on analytic methods, while at the same time offering a contextual landscape of modern engineering, in order to introduce related statistical and probabilistic models used in this domain. This makes the book an incredibly useful tool, offering interesting insights on challenges in the field for researchers and practitioners alike. * Compiles cutting-edge methods and examples of analytical approaches to systems and software testing from leading authorities in applied statistics, computer science, and software engineering * Combines methods and examples focused on the analytic aspects of systems and software testing * Covers logistic regression, machine learning, Bayesian methods, graphical models, experimental design, generalized regression, and reliability models * Written by leading researchers and practitioners in the field, from diverse backgrounds including research, business, government, and consulting * Stimulates research at the theoretical and practical level Analytic Methods in Systems and Software Testing is an excellent advanced reference directed toward industrial and academic readers whose work in systems and software development approaches or surpasses existing frontiers of testing and validation procedures. It will also be valuable to post-graduate students in computer science and mathematics.

Leseprobe

Table of Contents

Cover

Preface

Annotated List of Chapters

Part I: Testing Concepts and Methods

1 Recent Advances in Classifying Risk‐Based Testing Approaches

Synopsis

1.1 Introduction

1.2 Background on Software Testing and Risk Management

1.3 Taxonomy of Risk‐Based Testing

1.4 Classification of Recent Risk‐Based Testing Approaches

1.5 Summary

References

2 Improving Software Testing with Causal Modeling

Synopsis

2.1 Introduction

2.2 From Correlation and Regression to Causal Models

2.3 Bayes' Theorem and Bayesian Networks

2.4 Applying Bayesian Networks to Software Defects Prediction

2.5 Using Software Failure Data to Determine Reliability

2.6 Bayesian Networks for Software Project Risk Assessment and Prediction

2.7 Summary

Appendix 2.1

References

3 Optimal Software Testing across Version Releases

Synopsis

3.1 Introduction

3.2 Mozilla Firefox: An Example of Bug Detection Data

3.3 Modeling Bug Detection across Version Releases

3.4 Bayesian Inference

3.5 The Goel–Okumoto Model

3.6 Application to the Optimal Time Between Releases

3.7 Summary

References

4 Incremental Verification and Coverage Analysis of Strongly Distributed Systems

Synopsis

4.1 Verifying Modern Hardware and Software

4.2 Motivating Example: Cooperation of Three Computational Units

4.3 Extensions of First‐Order Logic

4.4 Weighted Monadic Second‐Order Logic and Weighted Tree Automata

4.5 Modeling Computational Units as Logical Structures

4.6 Parallel Distributed Solution of a Partial Differential Equation

4.7 Disjoint Union and Shuffling of Structures

4.8 Syntactically Defined Translation Schemes

4.9 Strongly Distributed Systems

4.10 Complexity Analysis

4.11 Description of the Method

4.12 Application of WMSOL and WTA to Coverage Analysis

4.13 Summary

References

5 Combinatorial Testing: An Approach to Systems and Software Testing Based on Covering Arrays

Synopsis

5.1 Introduction

5.2 Covering Arrays

5.3 Systems and Software Testing in Practice

5.4 Challenges

References

6 Conceptual Aspects in Development and Teaching of System and Software Test Engineering

Synopsis

6.1 Introduction

6.2 Determining What to Teach

6.3 Systematic Mapping of Software Test Engineering Knowledge

6.4 Teaching for Conceptual Learning

6.5 Formative Assessment: Student Assessment Methods

6.6 Teaching Software Quality with a Massive Open Online Course

6.7 Formative Assessment with MERLO

6.8 Summary

References

Part II: Statistical Models

7 Non‐homogeneous Poisson Process Models for Software Reliability

Synopsis

7.1 Introduction

7.2 The NHPP

7.3 Piecewise Exponential Models

7.4 Hybrid Models

7.5 Statistical Inference

7.6 Two Examples

7.7 Summary

References

8 Bayesian Graphical Models for High‐Complexity Testing: Aspects of Implementation

Synopsis

8.1 The Bayesian Graphical Models Approach to Testing: A Brief Overview

8.2 Modeling for the Test–Retest Process

8.3 Multiple Failure Modes

8.4 Diagnostics for the BGM Approach

8.5 Model Maintenance and Evolution

8.6 End‐to‐End Testing

8.7 Viability of the BGM Approach for Individual Applications

8.8 Summary

Acknowledgment

References

9 Models of Software Reliability

Synopsis

9.1 Introduction

9.2 Time Domain Models

9.3 Data Domain Models

9.4 Summary

References

10 Improved Estimation of System Reliability with Application in Software Development

Synopsis

10.1 Introduction

10.2 A Review of Basic Concepts and Methods

10.3 Statistical Model and Data Structure

10.4 Estimation of a System Reliability Function

10.5 Improved Estimation Through the Shrinkage Idea

10.6 Examples and Numerical Illustration

10.7 Simulated Comparisons of the Estimators

10.8 Summary

Acknowledgments

References

11 Decision Models for Software Testing

Synopsis

11.1 Introduction

11.2 Decision Models in Software Testing

11.3 Games in Software Testing

11.4 Multi‐Objective Optimization in Software Testing

11.5 Summary

References

12 Modeling and Simulations in Control Software Design

Synopsis

12.1 Control System Design

12.2 Modeling and Simulation of Systems

12.3 Testing of Control Systems

12.4 Testing of Control Systems during the Development Process

12.5 Tools for Modeling and Simulations

12.6 Case Studies

12.7 Summary

References

Part III: Testing Infrastructures

13 A Temperature Monitoring Infrastructure and Process for Improving Data Center Energy Efficiency with Results for a High Performance Computing Data Center*

Synopsis

13.1 Introduction

13.2 Related Work

13.3 Background

13.4 Approach to Improving DC Energy Efficiency

13.5 Results

13.6 Application to Root Cause Determination

13.7 Summary

Acknowledgments

References

14 Agile Testing with User Data in Cloud and Edge Computing Environments

Synopsis

14.1 Introduction

14.2 Diagnostic Reports of the Quality of Service

14.3 Service Control

14.4 Implementation

14.5 Summary

References

15 Automated Software Testing

Synopsis

15.1 Introduction

15.2 Modeling Languages for Testing

15.3 Operational Profile Development

15.4 Test Case Generation

15.5 Test Execution

15.6 Summary

References

16 Dynamic Test Case Selection in Continuous Integration

Synopsis

16.1 Introduction

16.2 The Eiffel Framework

16.3 Test Case Selection Strategies

16.4 Automated vs. Manual Tests

16.5 Test Case Selection Based on Eiffel Data

16.6 Test Case Atomicity

16.7 Summary

References

17 An Automated Regression Testing Framework for a Hadoop‐Based Entity Resolution System

Synopsis

17.1 Introduction

17.2 Challenges of Hadoop Regression Testing

17.3 The HiPER Entity Resolution System

17.4 Entity Identity Information Life Cycle and HiPER Run Modes

17.5 The HiPER Regression Testing Framework

17.6 Future Work

17.7 Conclusion

References

Part IV: Testing Applications

18 Testing Defense Systems

Synopsis

18.1 Introduction to Defense System Testing

18.2 Design of Experiments for Defense System Testing

18.3 Statistical Analyses for Complex Systems

18.4 Case Studies of Design and Analysis in Operational Testing

18.5 Summary

References

19 A Search‐Based Approach to Geographical Data Generation for Testing Location‐Based Services

Synopsis

19.1 Introduction

19.2 Search‐Based Testing

19.3 LBS Testing by Improved Simulated Annealing

19.4 Experiments and Results

19.5 Summary

References

20 Analytics in Testing Communication Systems

Synopsis

20.1 Introduction

20.2 Metrics and Benchmarks

20.3 Testing Progress Trends and Prediction

20.4 Intelligent Automation of the Testing Process

20.5 Summary

References

21 Measures in the Systems Integration Verification and Validation Phase and Aerospace Applications Field Experience

Synopsis

21.1 Introduction

21.2 The “Test Cycle”

21.3 Testing by Simulation

21.4 Validation and Verification Testing Methods

21.5 Testing Strategies, Reliability, and Failures of Systems

21.6 Readiness Success Criteria and Risk Mitigation

21.7 The IAI Case Study

21.8 Summary

Acknowledgments

References

Index

End User License Agreement

List of Tables

Chapter 02

Table 2.1 Fatal automobile crashes per month.

Table 2.2 Node probability table for “defects found” node.

Table 2.3 Probability distributions for the nodes of the BN model in Figure 2.15.

Table 2.4 Node probability table for the node “probability of finding a defect.”

Table 2.5 Different scenarios for time period T1.

Table 2.6 Different scenarios for failure‐free periods of demands.

Chapter 03

Table 3.1 The number of bugs logged to Bugzilla for each individual rapid‐release version of Firefox.

Chapter 05

Table 5.1 Hourly software outage cost.

Table 5.2 Input coverage vs. test set size.

Table 5.3 Size of

CA

(

N

; 2,

k

, 2),

k

 ≤ 3003.

Table 5.4

t

‐Coverage (%) and

t

‐Diversity (%) for Figures 5.1–5.4.

Table 5.5

Adjusted t

‐Coverage (%) and

adjusted t

‐Diversity (%) for Figures 5.5 and 5.6.

Table 5.6 Covering array construction tools.

Table 5.7

t

‐Coverage (%) and

t

‐Diversity (%): Multivariate preferences.

Table 5.8 Covering array number: Multivariate.

Table 5.9 Inputs and values for TCAS software module.

Table 5.10

t

‐Coverage (%) and

t

‐Diversity (%): TCAS.

Table 5.11 Covering array number: TCAS.

Table 5.12 Faulty variants detected: TCAS.

Table 5.13

Adjusted t

‐Coverage (%) and

adjusted t

‐Diversity (%): GCC.

Table 5.14 Covering array number: GCC.

Table 5.15 Fault detection effectiveness.

Chapter 06

Table 6.1 Top five soft skills for software testing / quality assurance.

Table 6.2 Teaching material collection template. Topic: Software testing techniques. Subtopic: Test oracle.

Table 6.3 Frequent flyer points table.

Chapter 07

Table 7.1 Failure data for the Naval Tactical Data System.

Table 7.2 Failure times for software developed at Tandem Computers.

Chapter 09

Table 9.1 Simulation results for Bayesian estimator and Bayesian risk.

Chapter 10

Table 10.1 Average losses of the estimators in the series system under different lifetime distributions. These are based on 1000 simulation replications. The systems had ten components, and the sample size for each replication was 10. Also indicated are the average value of the estimated shrinkage coefficient

.

Table 10.2 As Table 10.1, but for the parallel system.

Chapter 13

Table 13.1 Architectures of HPC platforms in the DC under consideration. Wolf was installed after certain changes to the DC cooling configuration had been made.

Table 13.2 Compute hardware operating temperature limits in degrees Celsius.

Chapter 15

Table 15.1 Domain‐specific languages used for testing.

Table 15.2 Classification of literature on testing domains that involve the use of an OP.

Table 15.3 Examples of popular automated testing tools and environments.

Chapter 16

Table 16.1 Comparison of traceability data collection procedures in 2011 and 2015.

Chapter 17

Table 17.1 Inventory of test cases.

Chapter 18

Table 18.1 General classes of test goals.

Table 18.2 Statistical measures of merit.

Table 18.3 Two candidate experiments.

Table 18.4 Overarching open air test – test design summary.

Table 18.5 GPS coordinate attack – test design summary.

Table 18.6 Laser attack – test design summary.

Table 18.7 New attack mode – test design summary.

Table 18.8 Factors and levels used in the OIL testing analysis.

Table 18.9 Results of the model fit to the data. APB‐11 provides a statistically significant improvement.

Chapter 19

Table 19.1 Detected differences between pairs of APIs.

Chapter 20

Table 20.1 Key performance indicators for measuring typical testing success in CSP software projects.

Chapter 21

Table 21.1 Development and testing stages as defined in the Quality Center tool.

Table 21.2 Data of 135 failures recorded from June 2011 to July 2012.

Table 21.3 Snapshot of the data extracted from Quality Center and processed with Minitab.

Table 21.4 Data for 74 failures recorded from August 2012 to July 2013.

List of Illustrations

Chapter 01

Figure 1.1 Core test process steps.

Figure 1.2 Risk‐based testing taxonomy.

Figure 1.3 Combining compliance assessment, security risk assessment, and security testing in RASEN.

Figure 1.4 SmartTesting process (Ramler and Felderer, 2015).

Chapter 02

Figure 2.1 Scatterplot of temperature against road fatalities (each dot represents a month).

Figure 2.2 Causal model for fatal crashes.

Figure 2.3 Scatterplot of LOC against all faults for a major system (each dot represents a module).

Figure 2.4 Scatterplots of cyclomatic complexity against number of pre‐ and post‐release faults for release

n

+ 1 (each dot represents a module).

Figure 2.5 Scatterplot of module fault density against size (each dot represents a module).

Figure 2.6 Scatterplot of pre‐release faults against post‐release faults for a major system (each dot represents a module).

Figure 2.7 Causal view of evidence.

Figure 2.8 Bayesian network for diagnosing disease.

Figure 2.9 Node probability table examples.

Figure 2.10 Reasoning within the Bayesian network.

Figure 2.11 Simple causal model for defects found.

Figure 2.12 Model in its initial (marginal) state.

Figure 2.13 Low number of defects found.

Figure 2.14 Effect of different evidence about testing quality.

Figure 2.15 Bayesian network model for software defects and reliability prediction.

Figure 2.16 Bayesian network model with marginal distributions for variables superimposed on nodes. All of the graphs are probability distributions, but there is a standard convention to represent discrete distributions (such as the node

testing quality

) with horizontal bars (i.e. the probability values are on the

x

‐axis), whereas continuous/numeric distributions (such as the node

defects found in testing

) have vertical bars (i.e. the probability values are on the

y

‐axis).

Figure 2.17 Zero defects in testing and high complexity observed.

Figure 2.18 Very high operational usage.

Figure 2.19 Testing quality is very high.

Figure 2.20 Defects phase risk object.

Figure 2.21 Typical risk object for testing quality.

Figure 2.22 Scenario for “overall testing effectiveness.”

Figure 2.23 Sequence of software testing phases as linked BN risk objects.

Figure 2.24 Single‐phase (demand‐based) testing model.

Figure 2.25 Three‐phase testing with predicted defects found for 200 demands in phase T3.

Figure 2.26 Bayesian network model for software reliability (no faults fixed).

Figure 2.27 Result of observing two failures in 400 demands.

Figure 2.28 Full BN model for software project risk.

Figure 2.29 Schematic for the project‐level model.

Figure 2.30 Two scenarios in Risk Table view.

Figure 2.31 Distributions when functionality delivered is set as “perfect” for the new project (compared with the baseline).

Figure 2.32 When staff quality is “medium,” project duration jumps to a median value of 54 months.

Figure 2.33 Project duration set to 18 months.

Figure 2.34 Project duration = 12 months, people = 10.

Figure 2.35 Quality delivered if process and people quality = medium with resource constraints set.

Figure 2.36 Functionality (function points) delivered if process and people quality = medium with resource constraints set.

Chapter 03

Figure 3.1 Cumulative number of bugs logged to Bugzilla for each individual rapid‐release version of Firefox as a function of time from release 5 (the earliest in the figure) to 25 (the latest).

Figure 3.2 Proportion of bugs belonging to each release of Firefox for each individual rapid‐release version of Firefox as a function of time from release 5 (the earliest in the figure) to 25 (the latest).

Figure 3.3 Box plots of posterior samples of

and

for the Mozilla Firefox data under the independent model.

Figure 3.4 Box plots of posterior samples of

and

for the Mozilla Firefox data under the random sample model.

Figure 3.5 Box plots of posterior samples of

and

for the Mozilla Firefox data under the hierarchical model.

Figure 3.6 The expected utility of testing to time

for the Firefox data with

,

, and

.

Figure 3.7 The optimal time between releases as a function of

for the Firefox data with

and

.

Figure 3.8 The optimal time between releases as a function of

for the Firefox data with

and

.

Figure 3.9 A set of values of

that yield

days, the release cycle length that Firefox uses in the dataset, for

.

Chapter 04

Figure 4.1 A system with three units.

Figure 4.2 Run tree

of the system with three units.

Figure 4.3 Composition of two FSMs.

Figure 4.4 Domain decomposition.

Figure 4.5 Computation Scheme for the domain decomposition method.

Figure 4.6 Translation scheme and its components.

Figure 4.7 Layout of a full ten‐bit adder.

Figure 4.8 Layout of a register.

Figure 4.9 IBM Cell Broadband Engine processor.

Figure 4.10 Intel multicore processor.

Figure 4.11 From System to

.

Figure 4.12 Computation of an FSM coverage metric for functional verification.

Chapter 05

Figure 5.1 An optimal

CA

(6; 2, 5, 2).

Figure 5.2

CA

(12; 3, 5, 2).

Figure 5.3 An optimal

MCA

(9; 2, (3

2

⋅2

3

)).

Figure 5.4

MCA

(21; 3, (3

2

⋅2

3

)).

Figure 5.5 An optimal

CCA

(9; 2, (3

2

⋅2

3

), φ = {{(c

1

,1), (c

3

,1)}}).

Figure 5.6 An optimal

CCA

(9; 2, (3

2

⋅2

3

), φ = {{(c

1

,1), (c

3

,1)}, {(c

1

,1), (c

3

,2)}}).

Figure 5.7 JMP v13 Multivariate platform preferences dialog.

Figure 5.8 Multivariate preferences meta‐control menus.

Figure 5.9 TCAS protection envelope.

Figure 5.10 TCAS II block diagram.

Figure 5.11 GCC constraints as a Boolean expression.

Chapter 06

Figure 6.1 Software Engineering Education Knowledge.

Figure 6.2 Academia/industry feedback cycles.

Figure 6.3 Testing scope explosion.

Figure 6.4 Students' ranking of soft skills.

Figure 6.5 Knowledge collection method.

Figure 6.6 Template for constructing an item family in MERLO.

Figure 6.7 Example of a MERLO item (mathematics/functions).

Figure 6.8 Example of MERLO item: test oracle.

Figure 6.9 An integrated learning environment.

Figure 6.10 Preparation of subtopic CFG.

Figure 6.11 North Pole Camouflage game. Left: Solution for one level. Right: Basic principle.

Figure 6.12 Virtualization of results.

Figure 6.13 MERLO exam on risk‐based testing (a).

Figure 6.14 MERLO exam on risk‐based testing (b).

Figure 6.15 MERLO results.

Chapter 07

Figure 7.1 Complete intensity functions, and complete mean functions for three models: (top) NHPP, (middle) piecewise exponential, and (bottom) a hybrid model. The complete intensity function is the failure intensity conditioned on the past history of the failure process.

Figure 7.2 (Top) Plot of cumulative operating time vs. failure number for the NTDS data. (Bottom) The same data with the Goel–Okumoto and PLP models fit to the mean function.

Figure 7.3 (Top) Plot of estimated complete intensity function (top) and estimated mean function (Bottom) for the JM model applied to the NTDS data set.

Figure 7.4 Profile log‐likelihood function for NTDS data set using the JM model.

Figure 7.5 Tandem Computers software failure data. (Top) Testing time measured in CPU hours. (Middle) Number of failures per week. (Bottom) Failure rates (failures per CPU hour) for each week.

Chapter 08

Figure 8.1 Bayesian graphical model (reduced form) for a software problem with two failure modes.

Chapter 10

Figure 10.1 Schematic block diagrams of series, parallel, series–parallel, and bridge systems.

Figure 10.2 System reliability functions for the series, parallel, series–parallel, and bridge systems when the components have common unit exponential lifetimes and there are five components in the series and parallel systems.

Figure 10.3 Estimated and true system reliability functions over time for a series system with ten components. The estimators are the PLE, ML based on component data, and the improved estimator based on a system sample of size 10.

Figure 10.4 As Figure 10.3, but for a parallel system.

Figure 10.5 Comparing the performance of system‐data‐ and component‐data‐based estimators at different values of

for a ten‐component parallel system. One thousand replications were used, with each replication having a sample size of 10.

Figure 10.6 Comparing the performance of the component‐data‐based estimators at different values of

for a ten‐component parallel system. One thousand replications were used, with each replication having a sample size of 10.

Chapter 11

Figure 11.1 The

‐stage decision tree for the optimal release problem.

Chapter 12

Figure 12.1 Control system design by the V‐model (systems development lifecycle).

Figure 12.2 Model in the loop – simulation process.

Figure 12.3 Software in the loop.

Figure 12.4 Processor in the loop.

Figure 12.5 Hardware in the loop.

Figure 12.6 Rapid control prototyping.

Figure 12.7 Rapid control prototyping and identification in the loop.

Figure 12.8 Component in the loop.

Figure 12.9 Idea of PIL simulation for rotary inverted pendulum.

Figure 12.10 Block scheme of PIL simulation components for rotary inverted pendulum.

Figure 12.11 Typical REX PIL configurations.

Figure 12.12 Three‐dimensional model of a rotary inverted pendulum.

Figure 12.13 Notation for model of rotary inverted pendulum.

Figure 12.14 Control of the real system of the rotary inverted pendulum.

Figure 12.15 Diagram of speed control loop.

Figure 12.16 PIL modeling diagram for the rotary inverted pendulum.

Figure 12.17 Two degrees of freedom control structure.

Figure 12.18 Swing‐up reference state trajectories

x

*

(

t

) and feed‐forward control signal

u

*

(

t

).

Figure 12.19 Block diagram of SIL simulation designed in REX Control System.

Figure 12.20 Comparison of reference and current trajectories

x

1

,

x

2

for SIL simulation.

Figure 12.21 Photo of testing stand for the PIL model of the control of the rotary inverted pendulum.

Figure 12.22 Web visualization screenshot.

Figure 12.23 Heating furnace with four zones.

Figure 12.24 MIL model of neural network for heating zone 4.

Figure 12.25 Diagram of the PIL heating furnace control and testing system.

Figure 12.26 SIL model of the heating furnace for zone 4 in Matlab/Simulink.

Figure 12.27 Fuzzy control function block in Matlab/Simulink, and sample of the PLC source code in ST format.

Figure 12.28 Output data for the zone 4 heating furnace model control loop (PIL) – Matlab/Simulink and Siemens PLC (dotted line: temperature [°C]; normal line: air [% Nm

3

/h]; dashed line: gas [% Nm

3

/h]).

Figure 12.29 Output data for zone 4 of the real heating furnace in the control loop with the Siemens PLC (RCP) (dotted line: temperature [°C]; normal line: air [% Nm

3

/h]; dashed line: gas [% Nm

3

/h]).

Figure 12.30 Comparison of results from the PIL and RCP models (red: temperature [°C]; blue: air [% Nm

3

/h]; orange: gas [% Nm

3

/h]).

Chapter 13

Figure 13.1 Ninety‐five percent upper prediction bounds for the highest temperature that each Mustang node would attain while running HPL1 for 24 hours (Max24Hours) before any changes were made to the DC cooling configuration. This figure from Storlie et al. (2017), published by Taylor and Francis Ltd. in the

Journal of the American Statistical Association

(http://www.tandfonline.com/toc/uasa20/current), reprinted by permission of Taylor and Francis Ltd. and the American Statistical Association.

Figure 13.2 Estimated posterior mean effect on Mustang nodes of the deployment of Wolf and other changes to the DC. This figure from Storlie et al. (2017), published by Taylor and Francis Ltd. in the

Journal of the American Statistical Association

(http://www.tandfonline.com/toc/uasa20/current), reprinted by permission of Taylor and Francis Ltd. and the American Statistical Association.

Figure 13.3 Estimated posterior mean effect on Mustang nodes of running HPL2 instead of HPL1. This figure from Storlie et al. (2017), published by Taylor and Francis Ltd. in the

Journal of the American Statistical Association

(http://www.tandfonline.com/toc/uasa20/current), reprinted by permission of Taylor and Francis Ltd. and the American Statistical Association.

Chapter 14

Figure 14.1 Chart of user activities.

Figure 14.2 Usability control.

Figure 14.3 Alerting.

Chapter 15

Figure 15.1 Flowchart for the MBT process.

Figure 15.2 Example UML diagrams of an ATM.

Figure 15.3 A TTCN‐3 example of ATM test.

Figure 15.4 Automated testing process.

Chapter 16

Figure 16.1 A simple example of an Eiffel event graph. Event names have been abbreviated: the full names of all Eiffel events are of the form

Eiffel…Event

, e.g.

EiffelArtifactCreatedEvent

.

Figure 16.2 Plot of organizational size (number of mainline committers times hierarchical depth) versus average commit size, excluding commits by external consultants, in six industry cases.

Figure 16.3 Eiffel events required for selecting tests that tend to fail when certain parts of the source code are modified.

Chapter 17

Figure 17.1 HiPER functional components.

Figure 17.2 Identity Capture Only configuration of HiPER.

Figure 17.3 Identity Capture Update configuration of HiPER.

Figure 17.4 Identity Resolution configuration of HiPER.

Figure 17.5 Assertion configuration of HiPER.

Figure 17.6 Merge–Purge configuration of HiPER.

Figure 17.7 Test Framework architecture.

Figure 17.8 HiPER Testing Framework detailed diagram.

Figure 17.9 Framework example report outputs.

Figure 17.10 Logic flow for <TestCaseScripts> using <GenerateAssertions>.

Chapter 18

Figure 18.1 Engagement profiles and factor plots.

Figure 18.2 Scaled prediction variance for two candidate experimental designs.

Figure 18.3 FDS graph for candidate experimental designs.

Figure 18.4 Prediction plot for time to find initial target (minutes).

Figure 18.5 Illustration of lognormal transformation on data.

Figure 18.6 Overarching design power at 80% confidence level.

Figure 18.7 Coordinate attack design power at 80% confidence level.

Figure 18.8 Laser attack design power at 80% confidence level.

Figure 18.9 Normal attack design power at the 80% confidence level.

Figure 18.10 Color map of correlations for all factors and two‐way interactions for normal attack design.

Figure 18.11 Operator‐in‐the‐loop test design matrix.

Figure 18.12 Raw results from the operator‐in‐the‐loop testing.

Figure 18.13 Empirical cumulative distribution of the OIL data.

Figure 18.14 Model predictions (see figure legend), along with the median detection time observed in each bin.

Figure 18.15 Soldiers emplacing the Q‐53 radar during operational testing.

Figure 18.16 Example fire mission including relevant geometric factors impacting Q‐53 system performance.

Figure 18.17 Detection probabilities for 323 fire missions conducted during the Q‐53 IOT&E.

Figure 18.18 The probability of detection for the Q‐53 counterfire radar using the 360° operating mode against single‐fired artillery.

Figure 18.19 Quantile–quantile (QQ) plots are used to visually assess normality.

Figure 18.20 Q‐53 target location error for estimated weapon locations.

Chapter 19

Figure 19.1 Mobile cloud architecture overview.

Figure 19.2 Approach overview.

Figure 19.3 An example of the search process.

Figure 19.4 The number of detected differences of each type for each API pair.

Figure 19.5 Comparison between SA and random algorithms.

Chapter 20

Figure 20.1 Amdocs BEAT Mobile Reports, a solution for CSP managers to view ad hoc testing progress in real time from their mobile device.

Figure 20.2 Rayleigh distribution graph that can be used to model defect detection in waterfall acceptance testing.

Figure 20.3 Sum of Rayleigh functions, and the number of real defects detected per week on an Amdocs project with several drops.

Figure 20.4 Model of effort effect on quality, in an example of a total of 100 defects, and testing in ten weeks with four drops, one on each given odd week.

Figure 20.5 Example of logistic trend of the percent of defects unfixed in a (one‐year) CSP software testing project with agile delivery.

Figure 20.6 Multiplying the two models of detection rate and closure rate creates a new model of the number of open defects.

Figure 20.7 Identified exponential decrease in defect density from release to release.

Figure 20.8 Amdocs BEAT Design Console uses defect history to alert on risk.

Chapter 21

Figure 21.1 V‐model of software and system development.

Figure 21.2 Engineering hours to fix problems by detection phase.

Figure 21.3 IAI space programs – characteristic internal program structure.

Figure 21.4 STAM profile analysis using data from June 2011 to July 2012.

Figure 21.5 STAM profiles using data from August 2012 to July 2013.

Guide

Cover

Table of Contents

Begin Reading

Pages

iii

iv

v

ix

x

xi

xii

xiii

xv

xvi

xvii

xviii

xix

1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

195

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

405

406

407

408

409

410

411

412

413

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

439

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

Analytic Methods in Systems and Software Testing

Edited by

Ron S. Kenett

KPA, Israel and Samuel Neaman Institute, Technion, Israel

Fabrizio Ruggeri

CNR-IMATI, Italy

FrederickW. Faltin

The Faltin Group and Virginia Tech, USA

This edition first published 2018© 2018 John Wiley & Sons Ltd

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Professor Ron S. Kenett, Dr Fabrizio Ruggeri and Frederick W. Faltin to be identified as the authors of the editorial material in this work has been asserted in accordance with law.

Registered Office(s)John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USAJohn Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

Editorial Office9600 Garsington Road, Oxford, OX4 2DQ, UK

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of WarrantyWhile the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging-in-Publication Data

Names: Kenett, Ron, editor. | Ruggeri, Fabrizio, editor. | Faltin, Frederick W., editor.Title: Analytic methods in systems and software testing / edited by, Ron S. Kenett, KPA, Raanana, Israel and Neaman Institute, Technion, Haifa, Israel, Fabrizio Ruggeri, CNR-IMATI, IT, Frederick W. Faltin, The Faltin Group, USA.Description: 1 edition. | Hoboken, NJ, USA : Wiley, [2018] | Includes bibliographical references and index. |Identifiers: LCCN 2018008695 (print) | LCCN 2018017885 (ebook) | ISBN 9781119487364 (pdf) | ISBN 9781119487401 (epub) | ISBN 9781119271505 (cloth)Subjects: LCSH: Computer software–Testing.Classification: LCC QA76.76.T48 (ebook) | LCC QA76.76.T48 A52 2018 (print) | DDC 005.1/4–dc23LC record available at https://lccn.loc.gov/2018008695

Cover design by WileyCover image: © SergeyNivens/iStockphoto

To Jonathan, Alma, Tomer, Yadin, Aviv, Gili, Matan and Eden

–Ron S. Kenett

To Anna, Giacomo and Lorenzo

–Fabrizio Ruggeri

To Donna, Erin, Travis and Maddie

–FrederickW. Faltin

List of Contributors

Dani AlmogBen Gurion University of the Negev; and Sami Shamoon College of EngineeringBeer Sheva, Israel

Sarit AssarafIsrael Aerospace Industries Missile and Space GroupIsrael

Matthew AveryInstitute for Defense Analyses (IDA)4850 Mark Center DriveAlexandria, VA 22311USA

Xiaoying BaiDepartment of Computer Science and TechnologyTsinghua UniversityBeijing, 100084China

Amanda M. BonnieHPC Environments GroupLos Alamos National LaboratoryPO Box 1663 MS T084Los Alamos, NM 87545USA

Jan BoschChalmers University of TechnologyLindholmspiren 541756 GöteborgSweden

Hadas ChasidimSami Shamoon College ofEngineering Beer ShevaIsrael

Justace ClutterInstitute for Defense Analyses (IDA)4850 Mark Center DriveAlexandria, VA 22311USA

Frank CoolenDepartment of Mathematical SciencesDurham UniversityDurham, DH1 3LEUnited Kingdom

Laura A. DaveyHPC Environments GroupLos Alamos National LaboratoryPO Box 1663 MS T080Los Alamos, NM 87545,USA

Xiaoxu DiaoOhio State University201 W. 19th AvenueScott Laboratory W382Columbus, OH 43210USA

Tomas DocekalVSB‐Technical University of Ostrava, FEECSDepartment of Cybernetics and Biomedical Engineering17. Listopadu 15708 33 Ostrava‐PorubaCzech Republic

Michael FeldererDepartment of Computer ScienceUniversity of Innsbruck6020 InnsbruckAustria

Norman FentonQueen Mary University of London London; and Agena Ltd, CambridgeUnited Kingdom

Laura J. FreemanInstitute for Defense Analyses (IDA)4850 Mark Center DriveAlexandria, VA 22311USA

Michael GoldsteinDepartment of Mathematical SciencesDurham UniversityDurham, DH1 3LEUnited Kingdom

Jürgen GroßmannFraunhofer Institute for Open Communication Systems FOKUSKaiserin‐Augusta‐Allee 3110589 BerlinGermany

Avi HarelErgolightHaifaIsrael

Kejia HouDepartment of Computer Science and TechnologyTsinghua UniversityBeijing, 100084China

Jun HuangDepartment of Computer Science and TechnologyTsinghua UniversityBeijing, 100084China

Joshua R. JohnsonBlack Oak Analytics3600 Cantrell Road, Suite 205Little Rock, Arkansas, 72202USA

Thomas JohnsonInstitute for Defense Analyses (IDA)4850 Mark Center DriveAlexandria, VA 22311USA

Ron S. KenettKPA, Raanana; and Samuel Neaman InstituteTechnion, HaifaIsrael

Jiri KoziorekVSB‐Technical University of Ostrava, FEECSDepartment of Cybernetics and Biomedical Engineering17. Listopadu 15708 33 Ostrava‐PorubaCzech Republic

Boyuan LiOhio State University201 W. 19th AvenueScott Laboratory W382Columbus, OH 43210USA

V. Bram LillardInstitute for Defense Analyses (IDA)4850 Mark Center DriveAlexandria, VA 22311USA

Sarah E. MichalakStatistical Sciences GroupLos Alamos National LaboratoryPO Box 1663 MS F600Los Alamos, NM 87545USA

Andrew J. MontoyaHPC Environments GroupLos Alamos National LaboratoryPO Box 1663 MS T084Los Alamos, NM 87545USA

Joseph MorganSAS Institute IncorporatedCary NC 27513USA

Thomas E. Moxley IIIHPC Systems GroupLos Alamos National LaboratoryPO Box 1663 MS T084Los Alamos, NM 87545USA

Martin NeilQueen Mary University of London United Kingdom; and Agena Ltd, CambridgeUnited Kingdom

Seán Ó RíordáinSchool of Computer Science and StatisticsTrinity CollegeDublin 2Ireland

Stepan OzanaVSB‐Technical University of Ostrava, FEECSDepartment of Cybernetics and Biomedical Engineering17. Listopadu 15708 33 Ostrava‐PorubaCzech Republic

Edsel A. PeñaDepartment of StatisticsUniversity of South CarolinaColumbia, SC 29208USA

Daniel PullenBlack Oak Analytics3600 Cantrell Road, Suite 205Little Rock, Arkansas, 72202USA

Beidi QiangDepartment of Mathematics and StatisticsVadalabene Center 1028Southern Illinois University EdwardsvilleEdwardsville, IL 62026‐1653USA

Elena V. RavveOrt Braude College of EngineeringCarmielIsrael

Brian J. ReichDepartment of StatisticsNorth Carolina State UniversityCampus Box 82035234 SAS HallRaleigh, NC, 27695USA

Steven E. RigdonSaint Louis UniversitySalus Center3545 Lafayette Ave., Room 481St. Louis, MO 63104USA

Manuel RodriguezOhio State University201 W. 19th AvenueScott LaboratoryColumbus, OH 43210USA

Fabrizio RuggeriCNR‐IMATIVia Bassini 1520131 MilanoItaly

William N. RustStatistical Sciences GroupLos Alamos National LaboratoryPO Box 1663 MS F600Los Alamos, NM 87545USA

Ina SchieferdeckerFraunhofer Institute for Open Communication Systems FOKUSKaiserin‐Augusta‐Allee 3110589 BerlinGermany

Uri ShafrirKMDI, iSchoolUniversity of TorontoTorontoCanada

Gilli ShamaAmdocs8 HaPnina StreetRa'anana 4321545Israel

Carol SmidtsOhio State University201 W. 19th AvenueScott Laboratory E429Columbus, OH 43210USA

Refik SoyerDepartment of Decision SciencesThe George Washington UniversityFunger Hall2201 G Street NWWashington, DC 20052USA

Vilem SrovnalVSB‐Technical University of Ostrava, FEECSDepartment of Cybernetics and Biomedical Engineering17. Listopadu 15708 33 Ostrava‐PorubaCzech Republic

Daniel StåhlEricsson ABDatalinjen 358330 LinköpingSweden

Curtis B. StorlieDepartment of Biomedical Statistics and InformaticsMayo Clinic College of MedicineRochester, MN, 55905USA

John R. TalburtInformation Science DepartmentUniversity of Arkansas Little Rock2801 South University AvenueLittle Rock, Arkansas, 72204USA

Lawrence O. TicknorStatistical Sciences GroupLos Alamos National LaboratoryPO Box 1663 MS F600Los Alamos, NM 87545USA

Zeev VolkovichOrt Braude College of EngineeringCarmielIsrael

Pei WangBiomedical InformaticsUniversity of Arkansas for Medical Sciences4301 W Markham StreetLittle Rock, Arkansas, 72205USA

Simon P. WilsonSchool of Computer Science and StatisticsTrinity CollegeDublin 2Ireland

David WooffDepartment of Mathematical SciencesDurham UniversityDurham, DH1 3LEUnited Kingdom

Mingli YuDepartment of Computer Science and TechnologyTsinghua UniversityBeijing, 100084China

Shelemyahu ZacksBinghamton UniversityBinghamton, NYUSA

Preface

The objective of this edited volume is to compile leading edge methods and examples of analytical approaches to systems and software testing from leading authorities in applied statistics, computer science, and software engineering. The book provides a collection of methods for practitioners and researchers interested in a general perspective on this topic that affects our daily lives, and will become even more critical in the future. Our original objective was to focus on analytic methods but, as the many co-authors show, a contextual landscape of modern engineering is required to appreciate and present the related statistical and probabilistic models used in this domain. We have therefore expanded the original scope and offered, in one comprehensive collection, a state of the art view of the topic.

Inevitably, testing and validation of advanced systems and software is comprised in part of general theory and methodology, and in part of application-specific techniques. The former are transportable to new domains. While the latter generally are not, we trust the reader will share our conviction that case study examples of successful applications provide a heuristic foundation for adaptation and extension to new problem contexts. This is yet another example of where statistical methods need to be integrated with expert knowledge to provide added value.

The structure of the book consists of four parts:

Part I: Testing Concepts and Methods (Chapters 1–6)

Part II: Statistical Models (Chapters 7–12)

Part III: Testing Infrastructures (Chapters 13–17)

Part IV: Testing Applications (Chapters 18–21)

It constitutes an advanced reference directed toward industrial and academic readers whose work in systems and software development approaches or surpasses existing frontiers of testing and validation procedures. Readers will typically hold degrees in statistics, applied mathematics, computer science, or software engineering.

The 21 chapters vary in length and scope. Some are more descriptive, some present Advanced mathematical formulations, some are more oriented towards system and software engineering. To inform the reader about the nature of the chapters, we provide an annotated list below with a brief description of each.

The book provides background, examples, and methods suitable for courses on system and software testing with both an engineering and an analytic focus. Practitioners will find the examples instructive and will be able to derive benchmarks and suggestions to build upon. Consultants will be able to derive a context for their work with clients and colleagues. Our additional goal with this book is to stimulate research at the theoretical and practical level. The testing of systems and software is an area requiring further developments. We hope that this work will contribute to such efforts.

The authors of the various chapters clearly deserve most of the credit. They were generous in sharing their experience and taking the time to write the chapters. We wish to thank them for their collaboration and patience in the various stages of writing this project. We also acknowledge the professional help of the Wiley team who provided support and guidance throughout the long journey that lead to this book.

To help the reader we provide next an annotated list of chapters giving a peak preview as to their content. The chapters are grouped in three parts but there is no specific sequence to them so that one can meander from topic to topic without following the numbered order.

Annotated List of Chapters

1

Recent Advances in Classifying Risk-Based Testing Approaches

A taxonomy of risk-based testing aligned with risk considerations in all phases of a test process. It provides a framework to understand, categorize, assess, and compare risk-based testing approaches.

2

Improving Software Testing with Causal Modeling

An introduction to causal modeling using Bayesian networks and how it can be used to improve software testing strategies and predict software reliability.

3

Optimal Software Testing across Version Releases

Looks at models for bug detection in software with a regular schedule of version releases. Model inference is developed from a Bayesian perspective and applied to data on bug reports of Mozilla Firefox.

4

Incremental Verification and Coverage Analysis of Strongly Distributed Systems

A model of both software and hardware hierarchical systems as logical structures using a finite state machine. Properties to be tested are expressed with extensions of first-order logic.

5

Combinatorial Testing: An Approach to Systems and Software Testing Based on Covering Arrays

Combinatorial testing is an approach for sampling from the input test space. It exploits evidence that system failures are typically due to the interaction of just a few inputs.

6

Conceptual Aspects in Development and Teaching of System and Software Test Engineering

A novel approach for improving the learning of system and software testing in academic and lifelong learning programs. It builds on Meaning Equivalence Reusable Learning Objects (MERLO) used in formative assessment of conceptual understanding.

7

Non-homogeneous Poisson Process Models for Software Reliability

The non-homogeneous Poisson process is used as a model for the reliability of repairable hardware systems. The chapter discusses its applicability to software reliability.

8

Bayesian Graphical Models for High-Complexity Testing: Aspects of Implementation

The Bayesian graphical models approach to software testing is presented. It is a method for the logical structuring of software testing where the focus is on high reliability final stage integration testing.

9

Models of Software Reliability

Software reliability time domain and data domain models are discussed. Topics such as stopping time for fault detection, the Chernoff–Ray procedure, and capture–recapture methods are covered.

10

Improved Estimation of System Reliability with Application in Software Development

Shrinkage ideas are implemented to obtain improvements in the estimation of software reliability functions. This helps decide if software can be deployed in the field or further debugging is required.

11

Decision Models for Software Testing

Game- and decision-theoretic frameworks are used to develop optimal software testing strategies. Specifically, optimal choice of release time for software under different scenarios is provided.

12

Modeling and Simulations in Control Software Design

Control systems are studied through modeling and simulations. These designs have a major impact on the quality of control applications and the reliability, sustainability, and further extension of the system.

13

A Temperature Monitoring Infrastructure and Process for Improving Data Center Energy Efficiency with Results for a High Performance Computing Data Center

An approach to energy efficiency improvement is presented. It includes a temperature monitoring infrastructure and a process that enables changes to DC cooling systems, infrastructure, and equipment.

14

Agile Testing with User Data in Cloud and Edge Computing Environments

A framework for mitigating the risks of user errors is presented. It incorporates usability considerations in the design, testing, deployment, and operation of dynamic collaborative systems.

15

Automated Software Testing

This chapter presents model-based automatic software testing challenges and solutions. Building such models involves modeling language selection and operational profile development.

16

Dynamic Test Case Selection in Continuous Integration: Test Result Analysis Using the Eiffel Framework

Dynamic test case selection determines which tests to execute at a given time, rather than from predefined static lists. The Eiffel framework for continuous integration and delivery is presented.

17

An Automated Regression Testing Framework for a Hadoop-Based Entity Resolution System

A framework built to support a large-scale entity resolution application is presented. It addresses distributed processing environments such as Hadoop Map/Reduce and Spark with parallel processing.

18

Testing Defense Systems

Defense systems consistently push the limits of scientific understanding. The chapter highlights, with examples, the core statistical methodologies that have proven useful in testing defense systems.

19

A Search-Based Approach to Geographical Data Generation for Testing Location-Based Services

A framework to support automatic location-based services testing is presented from two aspects, geographical test data generation and query results. How to design and validate tests is discussed.

20

Analytics in Testing Communication Systems

This chapter is about communication service providers’ testing metrics and data analytics to support decision making processes. An example of using analytics to predict testing progress trends is presented.

21

Measures in the Systems Integration Verification and Validation Phase and Aerospace Applications Field Experience

A presentation of a methodology called Software Trouble Assessment Matrix (STAM) that generates metrics of development process performance. An example from a large satellite program is provided.

The editors of Analytic Methods in Systems and Software Testing:Ron S. Kenett, KPA Ltd. and Samuel Neaman Institute, Technion, IsraelFabrizio Ruggeri, CNR-IMATI, ItalyFrederick W. Faltin, The Faltin Group, and Virginia Tech, USA

Part ITesting Concepts and Methods

1Recent Advances in Classifying Risk‐Based Testing Approaches

Michael Felderer Jürgen Großmann, and Ina Schieferdecker

Synopsis

In order to optimize the usage of testing efforts and to assess risks of software‐based systems, risk‐based testing uses risk (re‐)assessments to steer all phases in a test process. Several risk‐based testing approaches have been proposed in academia and/or applied in industry, so that the determination of principal concepts and methods in risk‐based testing is needed to enable a comparison of the weaknesses and strengths of different risk‐based testing approaches. In this chapter we provide an (updated) taxonomy of risk‐based testing aligned with risk considerations in all phases of a test process. It consists of three top‐level classes: contextual setup, risk assessment, and risk‐based test strategy. This taxonomy provides a framework to understand, categorize, assess, and compare risk‐based testing approaches to support their selection and tailoring for specific purposes. Furthermore, we position four recent risk‐based testing approaches into the taxonomy in order to demonstrate its application and alignment with available risk‐based testing approaches.

1.1 Introduction

Testing of safety‐critical, security‐critical or mission‐critical software faces the problem of determining those tests that assure the essential properties of the software and have the ability to unveil those software failures that harm the critical functions of the software. However, for “normal,” less critical software a comparable problem exists: Usually testing has to be done under severe pressure due to limited resources and tight time constraints with the consequence that testing efforts have to be focused and be driven by business risks.

Both decision problems can be adequately addressed by risk‐based testing, which considers the risks of the software product as the guiding factor to steer all the phases of a test process, i.e., test planning, design, implementation, execution, and evaluation (Gerrard and Thompson, 2002; Felderer and Ramler, 2014a; Felderer and Schieferdecker, 2014). Risk‐based testing is a pragmatic approach widely used in companies of all sizes (Felderer and Ramler, 2014b, 2016) which uses the straightforward idea of focusing test activities on those scenarios that trigger the most critical situations of a software system (Wendland et al., 2012).

The recent international standard ISO/IEC/IEEE 29119 Software Testing (ISO, 2013) on testing techniques, processes, and documentation even explicitly specifies risk considerations to be an integral part of the test planning process. Because of the growing number of available risk‐based testing approaches and its increasing dissemination in industrial test processes (Felderer et al., 2014), methodological support to categorize, assess, compare, and select risk‐based testing approaches is required.

In this paper, we present an (updated) taxonomy of risk‐based testing that provides a framework for understanding, categorizing, assessing, and comparing risk‐based testing approaches and that supports the selection and tailoring of risk‐based testing approaches for specific purposes. To demonstrate the application of the taxonomy and its alignment with available risk‐based testing approaches, we position four recent risk‐based testing approaches, the RASEN approach (Großmann et al., 2015), the SmartTesting approach (Ramler and Felderer, 2015), risk‐based test case prioritization based on the notion of risk exposure (Yoon and Choi, 2011), and risk‐based testing of open source software (Yahav et al., 2014a), in the taxonomy.

A taxonomy defines a hierarchy of classes (also referred to as categories, dimensions, criteria, or characteristics) to categorize things and concepts. It describes a tree structure whose leaves define concrete values to characterize instances in the taxonomy. The proposed taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top‐level classes context (with subclasses risk driver, quality property, and risk item), risk assessment (with subclasses factor, estimation technique, scale, and degree of automation), and risk‐based test strategy (with subclasses risk‐based test planning, risk‐based test design and implementation, and risk‐based test execution and evaluation). The taxonomy presented in this chapter extends and refines our previous taxonomy of risk‐based testing (Felderer and Schieferdecker, 2014).

The remainder of this chapter is structured as follows. Section 1.2 presents background on software testing and risk management. Section 1.3 introduces the taxonomy of risk‐based testing. Section 1.4 presents the four selected recent risk‐based testing approaches and discusses them in the context of the taxonomy. Finally, Section 1.5 summarizes this chapter.

1.2 Background on Software Testing and Risk Management

1.2.1 Software Testing

Software testing (ISTQB, 2012) is the process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation, and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose, and to detect defects. According to this definition it comprises static activities like reviews but also dynamic activities like classic black‐ or white‐box testing. The tested software‐based system is called the system under test (SUT). As highlighted before, risk‐based testing (RBT) is a testing approach which considers the risks of the software product as the guiding factor to support decisions in all phases of the test process (Gerrard and Thompson, 2002; Felderer and Ramler, 2014a; Felderer and Schieferdecker, 2014). A risk is a factor that could result in future negative consequences and is usually expressed by its likelihood and impact (ISTQB, 2012). In software testing, the likelihood is typically determined by the probability that a failure assigned to a risk occurs, and the impact is determined by the cost or severity of a failure if it occurs in operation. The resulting risk value or risk exposure is assigned to a risk item. In the context of testing, a risk item is anything of value (i.e., an asset) under test, for instance, a requirement, a component, or a fault.

Risk‐based testing is a testing‐based approach to risk management that can only deliver its full potential if a test process is in place and if risk assessment is integrated appropriately into it. A test process consists of the core activities test planning, test design, test implementation, test execution, and test evaluation (ISTQB, 2012) – see Figure 1.1. In the following, we explain the particular activities and associated concepts in more detail.

Figure 1.1 Core test process steps.

According to ISO (2013) and ISTQB (2012), test planning is the activity of establishing or updating a test plan. A test plan is a document describing the scope, approach, resources, and schedule of intended test activities. It identifies, amongst others, objectives, the features to be tested, the test design techniques, and exit criteria to be used and the rationale of their choice. Test objectives are the reason or purpose for designing and executing a test. The reason is either to check the functional behavior of the system or its non‐functional properties. Functional testing is concerned with assessing the functional behavior of an SUT, whereas non‐functional testing aims at assessing non‐functional requirements such as security, safety, reliability, or performance. The scope of the features to be tested can be components, integration, or system. At the scope of component testing (also referred to as unit testing), the smallest testable component, e.g., a class, is tested in isolation. Integration testing combines components with each other and tests those as a subsystem, that is, not yet a complete system. In system testing, the complete system, including all subsystems, is tested. Regression testing is the selective retesting of a system or its components to verify that modifications have not caused unintended effects and that the system or the components still comply with the specified requirements (Radatz et al., 1990). Exit criteria are conditions for permitting a process to be officially completed. They are used to report against and to plan when to stop testing. Coverage criteria aligned with the tested feature types and the applied test design techniques are typical exit criteria. Once the test plan has been established, test control begins. It is an ongoing activity in which the actual progress is compared against the plan, which often results in concrete measures.

During the test design phase the general testing objectives defined in the test plan are transformed into tangible test conditions and abstract test cases. Test implementation comprises tasks to make the abstract test cases executable. This includes tasks like preparing test harnesses and test data, providing logging support, or writing test scripts, which are necessary to enable the automated execution of test cases. In the test execution phase, the test cases are then executed and all relevant details of the execution are logged and monitored. Finally, in the test evaluation phase the exit criteria are evaluated and the logged test results are summarized in a test report.

1.2.2 Risk Management

Risk management comprises the core activities risk identification, risk analysis, risk treatment, and risk monitoring (Standards Australia/New Zealand, 2004; ISO, 2009