99,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Serie: Quantitative Software Engineering Series
- Sprache: Englisch
Explores and identifies the main issues, concepts, principles and evolution of software testing, including software quality engineering and testing concepts, test data generation, test deployment analysis, and software test management
This book examines the principles, concepts, and processes that are fundamental to the software testing function. This book is divided into five broad parts. Part I introduces software testing in the broader context of software engineering and explores the qualities that testing aims to achieve or ascertain, as well as the lifecycle of software testing. Part II covers mathematical foundations of software testing, which include software specification, program correctness and verification, concepts of software dependability, and a software testing taxonomy. Part III discusses test data generation, specifically, functional criteria and structural criteria. Test oracle design, test driver design, and test outcome analysis is covered in Part IV. Finally, Part V surveys managerial aspects of software testing, including software metrics, software testing tools, and software product line testing.
- Presents software testing, not as an isolated technique, but as part of an integrated discipline of software verification and validation
- Proposes program testing and program correctness verification within the same mathematical model, making it possible to deploy the two techniques in concert, by virtue of the law of diminishing returns
- Defines the concept of a software fault, and the related concept of relative correctness, and shows how relative correctness can be used to characterize monotonic fault removal
- Presents the activity of software testing as a goal oriented activity, and explores how the conduct of the test depends on the selected goal
- Covers all phases of the software testing lifecycle, including test data generation, test oracle design, test driver design, and test outcome analysis
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 593
Veröffentlichungsjahr: 2015
Ähnliche
CONTENTS
Cover
Title page
Preface
ACKNOWLEDGMENT
Part I: Introduction to Software Testing
1 Software Engineering: A Discipline Like No Other
1.1 A YOUNG, RESTLESS DISCIPLINE
1.2 AN INDUSTRY UNDER STRESS
1.3 LARGE, COMPLEX PRODUCTS
1.4 EXPENSIVE PRODUCTS
1.5 ABSENCE OF REUSE PRACTICE
1.6 FAULT-PRONE DESIGNS
1.7 PARADOXICAL ECONOMICS
1.8 CHAPTER SUMMARY
1.9 BIBLIOGRAPHIC NOTES
2 Software Quality Attributes
2.1 FUNCTIONAL ATTRIBUTES
2.2 OPERATIONAL ATTRIBUTES
2.3 USABILITY ATTRIBUTES
2.4 BUSINESS ATTRIBUTES
2.5 STRUCTURAL ATTRIBUTES
2.6 CHAPTER SUMMARY
2.7 EXERCISES
2.8 BIBLIOGRAPHIC NOTES
3 A Software Testing Lifecycle
3.1 A SOFTWARE ENGINEERING LIFECYCLE
3.2 A SOFTWARE TESTING LIFECYCLE
3.3 THE V-MODEL OF SOFTWARE TESTING
3.4 CHAPTER SUMMARY
3.5 BIBLIOGRAPHIC NOTES
Part II: Foundations of Software Testing
4 Software Specifications
4.1 PRINCIPLES OF SOUND SPECIFICATION
4.2 RELATIONAL MATHEMATICS
4.3 SIMPLE INPUT OUTPUT PROGRAMS
4.4 RELIABILITY VERSUS SAFETY
4.5 STATE-BASED SYSTEMS
4.6 CHAPTER SUMMARY
4.7 EXERCISES
4.8 PROBLEMS
4.9 BIBLIOGRAPHIC NOTES
5 Program Correctness and Verification
5.1 CORRECTNESS: A DEFINITION
5.2 CORRECTNESS: PROPOSITIONS
5.3 VERIFICATION
5.4 CHAPTER SUMMARY
5.5 EXERCISES
5.6 PROBLEMS
5.7 BIBLIOGRAPHIC NOTES
6 Failures, Errors, and Faults
6.1 FAILURE, ERROR, AND FAULT
6.2 FAULTS AND RELATIVE CORRECTNESS
6.3 CONTINGENT FAULTS AND DEFINITE FAULTS
6.4 FAULT MANAGEMENT
6.5 CHAPTER SUMMARY
6.6 EXERCISES
6.7 PROBLEMS
6.8 BIBLIOGRAPHIC NOTES
7 A Software Testing Taxonomy
7.1 THE TROUBLE WITH HYPHENATED TESTING
7.2 A CLASSIFICATION SCHEME
7.3 TESTING TAXONOMY
7.4 EXERCISES
7.5 BIBLIOGRAPHIC NOTES
Part III: Test Data Generation
8 Test Generation Concepts
8.1 TEST GENERATION AND TARGET ATTRIBUTES
8.2 TEST OUTCOMES
8.3 TEST GENERATION REQUIREMENTS
8.4 TEST GENERATION CRITERIA
8.5 EMPIRICAL ADEQUACY ASSESSMENT
8.6 CHAPTER SUMMARY
8.7 EXERCISES
8.8 BIBLIOGRAPHIC NOTES
8.9 APPENDIX: MUTATION PROGRAM
9 Functional Criteria
9.1 DOMAIN PARTITIONING
9.2 TEST DATA GENERATION FROM TABULAR EXPRESSIONS
9.3 TEST GENERATION FOR STATE BASED SYSTEMS
9.4 RANDOM TEST DATA GENERATION
9.5 TOURISM AS A METAPHOR FOR TEST DATA SELECTION
9.6 CHAPTER SUMMARY
9.7 EXERCISES
9.8 BIBLIOGRAPHIC NOTES
10 Structural Criteria
10.1 PATHS AND PATH CONDITIONS
10.2 CONTROL FLOW COVERAGE
10.3 DATA FLOW COVERAGE
10.4 FAULT-BASED TEST GENERATION
10.5 CHAPTER SUMMARY
10.6 EXERCISES
10.7 BIBLIOGRAPHIC NOTES
Part IV: Test Deployment and Analysis
11 Test Oracle Design
11.1 DILEMMAS OF ORACLE DESIGN
11.2 FROM SPECIFICATIONS TO ORACLES
11.3 ORACLES FOR STATE-BASED PRODUCTS
11.4 CHAPTER SUMMARY
11.5 EXERCISES
12 Test Driver Design
12.1 SELECTING A SPECIFICATION
12.2 SELECTING A PROCESS
12.3 SELECTING A SPECIFICATION MODEL
12.4 TESTING BY SYMBOLIC EXECUTION
12.5 CHAPTER SUMMARY
12.6 EXERCISES
12.7 BIBLIOGRAPHIC NOTES
13 Test Outcome Analysis
13.1 LOGICAL CLAIMS
13.2 STOCHASTIC CLAIMS: FAULT DENSITY
13.3 STOCHASTIC CLAIMS: FAILURE PROBABILITY
13.4 CHAPTER SUMMARY
13.5 EXERCISES
13.6 PROBLEMS
13.7 BIBLIOGRAPHIC NOTES
Part V: Management of Software Testing
14 Metrics for Software Testing
14.1 FAULT PRONENESS
14.2 FAULT DETECTABILITY
14.3 ERROR DETECTABILITY
14.4 ERROR MASKABILITY
14.5 FAILURE AVOIDANCE
14.6 FAILURE TOLERANCE
14.7 AN ILLUSTRATIVE EXAMPLE
14.8 CHAPTER SUMMARY
14.9 EXERCISES
14.10 BIBLIOGRAPHIC NOTES
15 Software Testing Tools
15.1 A CLASSIFICATION SCHEME
15.2 SCRIPTING TOOLS
15.3 RECORD-AND-REPLAY TOOLS
15.4 PERFORMANCE-TESTING TOOLS
15.5 ORACLE DESIGN TOOLS
15.6 EXCEPTION DISCOVERY
15.7 COLLABORATIVE TOOLS
15.8 CHAPTER SUMMARY
16 Testing Product Lines
16.1 PLE: A STREAMLINED REUSE MODEL
16.2 TESTING ISSUES
16.3 TESTING APPROACHES
16.4 ILLUSTRATION
16.5 CHAPTER SUMMARY
16.6 EXERCISES
16.7 PROBLEMS
16.8 BIBLIOGRAPHIC REFERENCES
Bibliography
Index
End User License Agreement
List of Tables
Chapter 01
Table 1.1 Lifecycle cost distribution: design versus manufacturing
Table 1.2 Lifecycle cost distribution: development versus testing
Table 1.3 Maintenance cost distribution: corrective versus adaptive
Table 1.4 Corrective maintenance cost distribution: design versus wear and tear
List of Illustrations
Chapter 01
Figure 1.1 Diseconomies of scale in software engineering.
Chapter 03
Figure 3.1 A reference software lifecycle.
Figure 3.2 Verification and validation.
Figure 3.3 A generic testing lifecycle.
Figure 3.4 The V-model of software testing.
Chapter 04
Figure 4.1 Special relations.
Figure 4.2 Complement and inverse.
Figure 4.3 Relational representation of sets.
Figure 4.4 Relational product.
Figure 4.5 Multiplying with universal relation.
Figure 4.6 Pre and post restriction.
Figure 4.7 Properties of relations.
Figure 4.8 A lattice of refinement.
Figure 4.9 R′ and R″ refine R.
Figure 4.10 Least upper bound of relations R
1
and R
2
.
Figure 4.11 The Join of compatible relations.
Figure 4.12 Incompatible relations.
Figure 4.13 Range of valid specifications.
Figure 4.14 Safety vs. reliability.
Chapter 05
Figure 5.1 Interpretation of dom(R ∩ P).
Figure 5.2 Flowchart of if-statement.
Figure 5.3 Flowchart of if-else statement.
Figure 5.4 Flowchart of while statement.
Figure 5.5 Structure of an Inference.
Chapter 06
Figure 6.1 Relative correctness and relative reliability.
Figure 6.2 To be more-correct without duplicating correct behavior.
Figure 6.3 Ordering candidate programs by relative correctness.
Figure 6.4 A framework for monotonic fault removal.
Chapter 08
Figure 8.1 A hierarchy of attributes.
Figure 8.2 Inadequate test data.
Figure 8.3 Possibly adequate test data.
Figure 8.4 Certainly adequate test data.
Figure 8.5 Partitioning the domain of the specification.
Figure 8.6 Mimicking a probability distribution.
Chapter 09
Figure 9.1 Partitioning the set of triangles.
Figure 9.2 Visiting each state.
Figure 9.3 Visiting each state transition.
Chapter 10
Figure 10.1 Flowchart of a GCD program.
Figure 10.2 Flowchart of a Triangle program: nested version.
Figure 10.3 Flowchart of a Triangle program: sequential version.
Figure 10.4 Hierarchy of test generation criteria.
Chapter 13
Figure 13.1 Fault distribution (native vs. seeded).
Figure 13.2 Estimating native faults.
Figure 13.3 Impact of usage patterns.
Figure 13.4 Targeted test coverage.
Figure 13.5 Regression log(MTTF) by N.
Chapter 14
Figure 14.1 Counting nodes and edges.
Figure 14.2 Measuring non-injectivity.
Figure 14.3 Enlarging the output space, preserving the range.
Chapter 16
Figure 16.1 Domain engineering and application engineering lifecycles.
Figure 16.2 Reference architecture of the queue simulation product line.
Guide
Cover
Table of Contents
Begin Reading
Pages
ii
iii
iv
v
xiv
xv
xvi
xvii
1
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
Quantitative Software Engineering Series
The Quantitative Engineering Series focuses on the convergence of systems engineering with emphasis on quantitative engineering trade-off analysis. Each title brings the principles and theory of programming in-the-large and industrial strength software into focus.
This practical series helps software developers, software engineers, systems engineers, and graduate students understand and benefit from this convergence through the unique weaving of software engineering case histories, quantitative analysis, and technology into the project effort. You will find each publication reinforces the series goal of assisting the reader with producing useful, well-engineered software systems.
Series Editor: Lawrence Bernstein
Late Professor Bernstein was Industrial Research Professor at the Stevens Institute of Technology. He previously pursued a distinguished executive career at Bell Laboratories. He was a fellow of the IEEE and ACM.
Trustworthy Systems for Quantitative Software Engineering / Larry Bernstein and C.M. Yuhas
Software Measurement and Estimation: A Practical Approach / Linda M. Laird and M. Carol Brennan
World Wide Web Application Engineering and Implementation / Steven A. Gabarro
Software Performance and Scalability / Henry H. Liu
Managing the Development of Software-Intensive Systems / James McDonald
Trustworthy Compilers / Vladimir O. Safonov
Oracle Database Performance and Scalability: A Quantitative Approach / Henry H. Liu
Enterprise Software Architecture and Design: Entities, Services and Resources / Dominic Duggan
Software Testing: Concepts and Operations / Ali Mili and Fairouz Tchier
Software Testing
Concepts and Operations
Ali Mili
NJIT, USA
Fairouz Tchier
KSU, KSA
Copyright © 2015 by John Wiley & Sons, Inc. All rights reserved
Published by John Wiley & Sons, Inc., Hoboken, New JerseyPublished simultaneously in Canada
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data
Mili, Ali. Software testing : concepts and operations / Ali Mili. pages cm Includes bibliographical references and index.
ISBN 978-1-118-66287-8 (cloth)1. Computer software–Testing. I. Title. QA76.76.T48M56 2015 005.1′4–dc23
2015001931
Dedicated to my parentsin honor of their 68 years of mutual devotionand to Amel, Noor, Farah Aisha, and Serena Aida.May they realize their hopes and dreams.A.M.
Dedicated to my loving parents,my husband Jamel, and my children Sarah, Bellal, and Amine. May their lives be filled with happiness and success. F.T.
Preface
Software engineering is the only engineering discipline where product testing is a major technical and organizational concern, as well as an important cost factor. Several factors contribute to this state of affairs:
The first factor that makes software testing such a big concern is, of course, the size and complexity of software products, which make the design of software products a high-risk, error-prone proposition.
The second factor is the lack of a standardized development process for software products, which means that product quality cannot be ensured by process controls, and has to be ensured by product controls instead.
The third factor is the scarcity of practical, scalable methods that can ensure product quality through static product analysis, shifting the burden to dynamic methods.
Other factors include the absence of a general reuse discipline, the lack of scalability of correctness-preserving development methods, and the pervasiveness of specification changes through the development, maintenance, and evolution process, etc.
The subject of this book is the study of software testing; amongst the many books that are currently available on the same subject, this book can be characterized by the following premises:
Software testing as an integral part of software quality assurance
. We view software testing as part of a comprehensive strategy for software quality assurance, alongside many other techniques. The law of diminishing returns advocates the use of a variety of diverse techniques, which complement each other, in such a way that each is used wherever it delivers the greatest return on investment. Hence software testing is better studied in a broader context that also encompasses other methods rather than to be studied as an isolated set of techniques.
Software testing as a complementary technique to static analysis.
Since the early days of software engineering, we have witnessed a colorful debate on the respective merits of software testing versus static program analysis in terms of effectiveness, scalability, ease of use, etc. We take the position that each of these techniques is effective against some type of specifications and less effective against other types; also, very often, when we find that one technique or another is difficult to use, it is not the result of any intrinsic shortcoming of the technique, rather it is because the technique is used against the wrong kind of specification. Of course, we do not always get to choose the specification against which we must ensure product correctness; but we can, in fact, decompose a complex specification into components and map each component to the technique that is most adapted to it. This is illustrated in
Chapter 6
.
Software testing as a systematic stepwise process
. Early on, software testing earned the reputation of being a means to prove the presence of faults in programs, but never their absence; this is an undeserved reputation, in fact, because testing can be used for all sorts of goals, as we discuss in
Chapter 7
. Nevertheless, whether deserved or not, this reputation has had two consequences: first, the assumption that the only possible goal of testing is fault exposure, diagnosis, and removal. Second, the (consequent) belief that testing amounts merely to test data generation, specifically the generation of test data that has the greatest potential to expose faults. By contrast, we argue that testing follows a multiphase process that includes goal identification and analysis, test data generation, oracle design, test driver design, test deployment, and test outcome analysis. We devote different chapters to each one of these phases.
Software testing as a formal/formalizable process
. Because it requires relatively little analysis of the software product under test or its specification, testing is often perceived as an activity that can be carried out casually, and informally. By contrast, we argue that testing ought to be carried out with the same level of rigor as static program verification, and that to perform testing effectively, one must be knowledgeable in software specifications, in program correctness, in relative correctness, in the meaning of a fault, in fault removal, etc.. This is discussed in more detail in
Chapter 6
.
Software testing as a goal-oriented activity
. We argue that far from being solely dedicated to finding and removing faults, testing may have a wide range of goals, including such goals as estimating fault density, estimating reliability, certifying reliability, etc. This is discussed in detail in
Chapter 7
.
This book stems from lecture notes of a course on software testing and quality assurance and hence is primarily intended for classroom use; though it may also be of interest to practicing software engineers, as well as to researchers in software engineering. The book is divided into five broad parts, including 3 or 4 chapters per part, to a total of 16 chapters.
Part I
introduces software testing in the broader context of software engineering and explores the qualities that testing aims to achieve or ascertain, as well as the lifecycle of software testing.
Part II
introduces mathematical foundations of software testing, which include software specification, program correctness and verification, concepts of software dependability, and a software testing taxonomy. It is uncommon for a software testing book to discuss specifications, verification, and dependability to the extent that we do in this book. We do it in this book for many reasons:
First, we believe that it is not possible to study software testing without a sound understanding of software specifications, since these capture the functional attributes that are testing candidate programs against and are the basis for oracle design.
Second, when we test a program in the context of product certification or in the context of acceptance testing, what is at stake is whether the candidate program is correct; surely, we need to understand what correctness means, for this purpose.
Third, if dynamic program testing and static program analysis are to be used in concert, to reach a more complete conclusion than any one method alone, they need to be cast in the same mathematical model.
Fourth, the act of removing a fault from a program, which is so central to testing, can only be modeled by defining the property of
relative correctness
, which provides that the program is
more-correct
once the fault is removed than it was prior to fault removal; relative correctness, in turn, can only be defined and understood if we understand the property of (absolute) correctness.
The taxonomy of software testing techniques classify these techniques according to a number of criteria, including in particular the criterion of goals: It is important to recognize the different goals that one may pursue in conducting software testing, and how each goal affects all the phases of the testing lifecycle, from test data generation to oracle design to test deployment to test outcome analysis.
Part III
explores a phase of software testing that has so dominated the attention of researchers and practitioners that it is often viewed as the only worthwhile issue in software testing: test data generation. In this part, we briefly discuss some general concepts of test data generation and then we explore the two broad criteria of test data generation, namely: functional criteria (
Chapter 9
) and structural criteria (
Chapter 10
). We discuss test data generation for simple programs that map initial states onto final states, as well as for state-bearing programs, whose output depends on their input history.
Part IV
discusses the remaining phases of the software testing lifecycle that arise after test data generation and include test oracle design, test driver design, and test outcome analysis. Test oracles (
Chapter 11
) are derived from target specifications according to the definition of correctness and depend on whether we are talking about simple state-free programs or about programs that have an internal state. Test driver design (
Chapter 12
) depends on whether test data has been generated offline and is merely deployed from an existing medium, or whether it is generated at random according to some probability law. As for the analysis of test outcomes (
Chapter 13
), it depends of course on the goal of the test and ranges from reliability estimation to reliability certification to fault density estimation to product acceptance, etc.
Part V
concludes the book by surveying some managerial aspects of software testing, including software metrics (
Chapter 14
), software testing tools (
Chapter 15
), and software product line testing (
Chapter 16
).
In compiling the material of this book, we focused our attention on analyzing and modeling important aspects of software testing, rather than on surveying and synthesizing the latest research on the topic; several premises determined this decision:
This book is primarily intended to be an educational tool rather than a research monograph.
In an area of active research such as software testing, students are better served by focusing on fundamental concepts that will serve them in the long run regardless of what problem they may encounter rather than to focus on the
latest techniques
, which by definition will not remain
latest
for too long.
In the perennial academic debate of whether we serve our students best by making them operational in the short term or by presenting them with fundamentals and enabling them to adapt in the long run, we have decided to err on the side of the latter option.
ACKNOWLEDGMENT
Special thanks are due to the late Professor Lawrence Bernstein for inviting us to write this book for inclusion in his distinguished series.
We thank our successive cohorts of students, who tolerated our caprices as we fine-tuned and refined the contents of our lecture notes term after term. We also thank Slim Frikha, a summer intern from ParisTech, France, who reviewed and evaluated software testing tools to help us with Chapter 15. This publication was made possible in part by a grant from the Qatar National Research Fund NPRP 04-1109-1-174. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the QNRF.
F. TCHIER
A. MILI
Part IIntroduction to Software Testing
In this part we introduce software testing by discussing what makes software engineering so special that testing should occupy such an important part of its lifecycle. Then we survey software qualities that testing techniques may be used to assess. Finally we review the various lifecycle models of software testing that may be followed depending on the context and goal of testing.
1Software Engineering: A Discipline Like No Other
On the face of it, software engineering sounds like an engineering discipline among others, such as chemical engineering, mechanical engineering, civil engineering, and electrical engineering. We will explore, in this chapter, in what way and to what extent software engineering differs from other engineering disciplines.
1.1 A YOUNG, RESTLESS DISCIPLINE
Civil engineering and mechanical engineering date back to antiquity or before, as one can see from various sites (buildings, road networks, utility infrastructures, etc.) around the Mediterranean basin. Chemical engineering (Lavoisier and others) and electrical engineering (Franklin and others) can be traced back to the eighteenth century. Nuclear engineering (Pierre and Marie Curie) emerged at the turn of the twentieth century and industrial engineering emerged around the time of the Second World War, with issues of logistics. By contrast, software engineering is a comparatively young discipline, emerging as it did in the second half of the twentieth century. The brief history of this discipline can be divided into five broad eras, lasting approximately one decade each, which are as follows:
The Sixties: The Era of Pioneers.
This era marks the first time that practitioners and researchers came face to face with the complexities, paradoxes, and anomalies of software engineering. Software projects of this era were ventures into unchartered territory, characterized by high levels of risk, unpredictable outcomes, and massive cost and schedule overruns. The programming languages that were dominant in this era are assembler, Fortran, Cobol, and (in academia) Algol.
The Seventies: Structured Software Engineering.
This era is characterized by the general belief that software engineering problems are of a technical nature and that if we evolved techniques for software specification, design, and verification to control complexity, all software engineering problems would be resolved. Given that structure is our main intellectual tool for dealing with complexity, this era has seen the emergence of a wide range of structured techniques, including structured programming, structured design, structured analysis, structured specifications, etc. The programming languages that were dominant in this era are C and (in academia) Pascal.
The Eighties: Knowledge-Based Software Engineering.
This era is characterized by the realization that software engineering problems are of a managerial and organizational nature more than a technical nature. This realization was concurrent with the emergence of the Fifth Generation Computing initiative, which started in Japan and spread across the globe (the United States, Europe, Canada), and was focused on thinking machines designed with extensive use of artificial intelligence techniques. This general approach permeated the discipline of software engineering with the emergence of knowledge-based software engineering techniques. The programming languages that were dominant in this era are Prolog, Scheme/Lisp, and Ada.
The Nineties: Reuse-Based Software Engineering.
As it became increasingly clear that fifth-generation computing was not delivering on its promise, and worldwide fifth-generation initiatives were fading, software researchers and practitioners turned their attention to reuse as a possible savior of the discipline. Software engineering is, after all, the only discipline where reuse is not an integral part of the routine engineering process. It was felt that if only software engineers had large databases of reusable software components readily available, the industry would achieve great gains in productivity, quality, time to market, and reduced process risk. This evolution was concurrent with the emergence of object-oriented programming, which supports a bottom–up design discipline that facilitates product reuse. The programming languages that were dominant in this era are C, C++, Eiffel, and Smalltalk.
The First Decade of the Millennium: Lightweight Software Engineering.
While software reuse is not practical as a general paradigm in software engineering, it is feasible in limited application domains, giving rise to
product line engineering
. Other attributes of this era include Java programming, with its focus on web applications; agile programming, with its focus on rapid and flexible response to change; and component-based software engineering, with its focus on software architecture and software composition. The programming languages that were dominant in this era are Java, C++, and (in academia) Python.
Perhaps as result of this young and eventful history, the discipline of software engineering is characterized by a number of paradoxes and counter-intuitive properties, which we explore in this chapter.
1.2 AN INDUSTRY UNDER STRESS
Nowadays, software runs all aspects of modern life and accounts for a large and increasing share of the world economy. This trend started slowly with the advent of computing in the middle of the twentieth century and was further precipitated by the emergence of the World Wide Web at the end of the twentieth and the beginning of the twenty-first century. This phenomenon has spawned a great demand for software products and services and generated a market pressure that the software industry takes great pains to cater to.
Many fields of science and engineering (such as bioinformatics, medical informatics, weather forecasting, and modeling and simulation) are so dependent on software that they can almost be considered as mere applications of software engineering. Also, it is possible to observe that many computer science curricula are slowly inching toward more software engineering contents at the expense of traditional theoretical material, which may be perceived as less and less relevant to today’s job market. Some engineering colleges are preempting the trend by starting software engineering degrees in computer science departments or by starting complete software engineering departments alongside traditional computer science departments.
Concurrent with a widening demand for software to serve ever-broader needs, we are also witnessing higher and higher expectations in terms of product quality. As software takes on ever more vital functions in life-critical and mission-critical applications and in applications that carry massive financial stakes, it becomes increasingly important to ensure that software products fulfill their function with a high degree of dependability. This requires that we deploy a wide range of techniques, including the following:
Process controls
, ensuring that software products are developed and evolved according to certified, mature processes.
Product controls
, ensuring that software products meet quality standards commensurate with their application domain requirements; this is achieved by a combination of techniques, including static analysis, dynamic testing, reliability estimation, fault tolerance, etc.
In summary, it is fair to argue that the software industry is under massive stress to deliver both quantity and quality; as we discuss in subsequent sections, this is both difficult and expensive.
1.3 LARGE, COMPLEX PRODUCTS
The demand for complex hardware/software systems has increased more rapidly than the ability to design, implement, test and maintain them.
Michael Lyu, Handbook of Software Reliability Engineering, 1996
Not only is it critical for us to build software products that are of high quality, it is also very difficult, due to their size and complexity. When it was built in the mid-60s, the IBM operating system OS360 (©IBM Corporation), with a million lines of code and a price tag of 500 million dollars, was considered as the most complex human artifact ever produced up to then. This size was subsequently dwarfed by Microsoft’s Windows operating systems (©Microsoft): The 1993 version (Windows NT 3.1) is estimated to be 5 millions lines of code, whereas the 2003 version (Windows Server 2003) is estimated to be 50 million lines of code. Completing projects of this kind of size is not only a major engineering undertaking but also a major organizational challenge; it is estimated that the production of the Windows Server 2003 involved 2000 software personnel (programmers, analysts, engineers) for development and 2400 software personnel for software testing.
Another example of software size growth is given by NASA’s flight software. A study published in 2009 by NASA’s Jet Propulsion laboratory under the title NASA Study on Flight Software Complexity (Jet Propulsion Laboratory, 2009) plots the evolution of flight software size of the various human and robotic space programs from 1968 to 2005. Both series (flight software for human missions and flight software for robotic mission) show a near-perfect linear evolution through the years, except that they are plotted on a logarithmic scale for size, meaning in effect that flight software size grows exponentially from year to year. Hence for human missions, flight software grows from 8.5 kilo lines of code (KLOC) for the Apollo program in 1968 to 470 KLOC for the space shuttle program in 1980 to 1.5 million lines of code (MLOC) for the international space station in 1989. For robotic missions, software size grows from 30 line of code (LOC) for the Mariner-6 mission in 1968 to 3 KLOC for Voyager in 1977 to 8 KLOC for Galileo in 1989 to 349 KLOC for DS1 (Deep Space 1) in 1999 to 545 KLOC for MRO (Mars Reconnaissance Orbiter) in 2005. The same Jet Propulsion Laboratory (JPL) report describes the recent evolution of military avionics software in the following terms: between 1960 and 2000, the percentage of flight control functionality that is delegated to software jumped from 8 to 80%, leading to an increase in size from one generation of aircrafts to another; hence it went from 1000 lines of code for the F-4A to 1.7 million lines of code for the F-22 to 5.7 million lines of code for the F-35 Joint Strike Fighter. The authors of the report argue that the increase in the size and complexity of flight software stems from software serving as a ‘complexity sponge,’ whereby complexity migrates from other parts of the system to software, on account of its flexibility and its adaptability.
A panel convened by the Software Engineering Institute (www.sei.cmu.edu) in 2005–2006 to analyze software systems of the future and draw a research agenda to manage such systems estimates that future software systems are expected to have sizes up to a billion lines of code. Along with this dry measure of size, such systems will be large in terms of other dimensions, such as (www.sei.cmu.edu/uls/) the amount of data stored, accessed, manipulated, and refined; the number of connections and interdependencies; the number of hardware elements; the number of computational elements; the number of system purposes and user perception of these purposes; the number of routine processes, interactions, and emergent behaviors; the number of overlapping policy domains and enforceable mechanisms; and the number of parties involved in the operation of the system (developers, maintainers, end users, stakeholders, etc.).
Size changes everything: such systems (referred to as ultra-large–Scale (ULS) systems) challenge all our knowledge and assumptions about software and are estimated to have a number of distinguishing features, such as the following:
Decentralization in fundamental dimensions, such as decentralized development, decentralized evolution, and decentralized operation.
Conflicting, unknown, and diverse requirements: Whereas the traditional view in software engineering is that requirements must be analyzed, compiled, and specified prior to software design and development, the view taken by the ULS approach is that at no time can we claim that all relevant requirements have been collected and specified.
Continuous evolution and deployment: Whereas the traditional view of software engineering is that a software product proceeds sequentially through successive phases of development, then maintenance, then phase out, ULS systems are developed, evolved, and deployed concurrently (made up of parts that are at different stages in their evolutionary process).
Heterogeneous, inconsistent, changing elements: Whereas a traditional software product is developed as a cohesive monolithic system by a development team, ULS systems emerge as the aggregate of many components, which may have evolved independently, using different paradigms and different technologies, by different teams, and from different stakeholder classes. Also, different components of the system are expected to evolve relatively independently.
Deep erosion of the people-system boundary: Whereas traditional systems are defined in terms of a distinct boundary that separates them from the outside world, ULS systems are envisioned to include human users as an integral part so that when a user interacts with a ULS system, she/he may be engaging human actors along with system behavior.
Failure is normal and frequent: Whereas in traditional software systems we think of failures as exceptional events and consider that failure avoidance is contingent upon fault removal, in ULS systems, we take a broader view of successful (failure-free) operation, which does not exclude the presence of faults but makes provisions for system redundancy and requirements nondeterminacy to make up for the presence of faults.
1.4 EXPENSIVE PRODUCTS
Not only are software products very large and complex, they are also very expensive to produce. Of course, if a product is large, one expects it to be costly, but what is surprising is that the unitary cost of software, that is, the cost per LOC, does, itself, increase with size. Whereas any programmer one asks may say that they can produce a hundred lines of code in a day or more, a more realistic figure, across all areas of software development, is closer to about 10 lines of code per day, or about 200 lines of code per person-month. This figure includes all costs that are spent producing software, including the cost of all phases of the software lifecycle, from requirements analysis and specification to software testing. If we assume the cost of a person-month to be 20,000 dollars (in salary, fringe benefits, and related expenses), this amounts to about $100 per LOC. If, for the sake of argument, we apply Boehm’s COnstructive COst MOdel (COCOMO) cost estimation model to a bespoke (custom-tailored) software project of size 500,000 source lines of code developed in embedded mode (the hardest/most costly development mode), we find 80 source lines of code per person-month.
In most other engineering disciplines, one way to mitigate costs is to use economies of scale, that is, to produce in such a large volume as to lower the unitary cost. Economies of scale are possible because in most engineering disciplines, the production process requires an initial up-front cost that is all the better amortized as the volume of production increases. The same process applies in software engineering: If we invest resources to acquire software tools, to train software professionals, or to set up a programming environment, then the more software we produce the better our investment is amortized. But in software we are also dealing with a phenomenon of diseconomy of scale: the more software we produce within a single product, the more interdependencies we create between the components of the product so that the unitary cost (per LOC) of large software products is larger than that of smaller products. This phenomenon of diseconomy of scale overrides the traditional economy of scale (that comes from amortizing up-front investments); the net result is a diseconomy of scale, which is all the more acute that the software product is larger or more complex; see Figure 1.1.
Figure 1.1Diseconomies of scale in software engineering.
Many of these costs are mitigated nowadays by the use of a variety of coarse-grained software development methods, which proceed to build software by composing existing components, rather than by painstakingly writing code from scratch line by line. Another trend that is emerging recently to address software cost and quality is the use of so-called Agile methodologies. These methodologies control the costs and risks of traditional lifecycles by following an iterative, incremental, flexible lifecycle, where the user participates actively in the specification and development of successive versions of the targeted software product.
1.5 ABSENCE OF REUSE PRACTICE
In the absence of economies of scale, one would hope to control costs by a routine discipline of reuse; in the case of software, it turns out that reuse is also very difficult to achieve on a routine basis. In any engineering discipline, reuse is made possible by the existence of a standard product architecture that is shared between the producer and the consumer of reusable assets: for example, automobiles have had a basic architecture that has not changed for over a century; all cars have a chassis, four wheels, an engine, a battery, a transmission, a cab, a steering column, a braking system, a horn, an exhaust system, shock absorbers, etc. Thanks to this architecture, the design of a new car is relatively straightforward and is driven primarily by design and marketing considerations; the designer of a new model does not have to reinvent a car from scratch and can depend on a broad market of companies that provide standard components, such as batteries, tires, and spare parts. The standard architecture of a car dictates market structure and creates great efficiencies in the production and maintenance of a car.
Unfortunately, no standard architecture exists in software products; this explains, to a large extent, why the expectations that software engineering researchers and practitioners pinned on a discipline of software reuse never fully materialized. Several software reuse initiatives were launched in the last decade of the last century, making available a wide range of software products and sophisticated search and assessment algorithms; but they were unsuccessful because software reuse requires not only functional matching between the available components and the requirements of the user but also architectural matching, which was often lacking. The absence of a standard architecture of software products also explains why software product lines have achieved some degree of success: product line engineering is a form of software reuse that is practiced in the context of a narrow application domain, in which it is possible to define and enforce a reference architecture. As an example, if we define a product line of e-commerce systems, we may want to define the reference architecture as being composed of the following components: a web front-end; a shopping cart component; an order-processing component; a banking component; a marketing and recommendations component; a network interface; and a database interface.
1.6 FAULT-PRONE DESIGNS
In other engineering disciplines, the presence of a standard product architecture, the availability of usable product components, the availability of compiled engineering knowledge, and the application of mandated safety requirements all contribute to reducing the design space of a product so as to make it manageable. The design of an engineering product (e.g., a bridge, a road, or a car) within this limited design space is a fairly straightforward operation that proceeds from requirements to finished product in a systematic, predictable manner.
In software engineering, the situation is significantly more complex, for several reasons, which are as follows:
There is no standard software architecture, except perhaps for some vague architectures of broad families of software products, such as data-processing applications, transaction-processing applications, event-processing applications, and language-processing applications.
There is little or no availability of software reusable assets, in the traditional sense of engineering assets that can be used to compose software products; the only assets that may be used widely across the industry are small assets (such as abstract data types (ADTs)) that deliver limited gains in terms of reduced lifecycle costs or reduced process risk.
There is little software engineering knowledge that may be used across applications in the same way that engineering knowledge is reused in complied form across products in other engineering disciplines.
Software specifications are very complex artifacts that typically involve vast amounts of detailed functional information; the breadth of the specification space precludes the ability to organize the design space in a systematic manner.
Because the design space of software products is so vast, software design is significantly more error prone than design in other engineering disciplines.
1.7 PARADOXICAL ECONOMICS
While technology can change quickly, getting your people to change takes a great deal longer. That is why the people-intensive job of developing software has had essentially the same problems for over 40 years.
Watts Humphrey, Winning with Software: An Executive Strategy, 2001
1.7.1 A Labor-Intensive Industry
If we consider the cost of an automobile, for example, and ponder the question of what percentage of this cost is due to the design process and what percentage is due to manufacturing, we find that most of the cost (more than 99%, perhaps) is due to manufacturing. Typically, by the time one buys a car, the effort that went into designing the new model has long since been amortized by the number of cars sold; what one is paying for is all the raw materials and the processing that went into manufacturing the car. By contrast, when one is buying a software product, one is paying essentially for the design effort, as there are no manufacturing costs to speak of (loading compact disks or downloading program files). Table 1.1 shows, summarily, how the cost of a software product differs from the cost of another engineering product in terms of distribution between design and manufacturing.
Table 1.1Lifecycle cost distribution: design versus manufacturing
Software engineering, %
Other engineering, %
Design
>99
<1
Manufacturing
<1
>99
1.7.2 Absence of Automation
The labor-intensive nature of software engineering has an immediate impact on the potential to automate software engineering processes. In all engineering processes, one can achieve savings in manufacturing by automating the manufacturing process or at least streamlining it, as in assembly lines. This is possible because manufacturing follows a simple, systematic process that requires little or no creativity. By contrast, design cannot be automated because it requires creativity, artistic appreciation, aesthetic sense, and so on. Automating the manufacturing process has an impact in traditional engineering disciplines because it helps reduce a cost factor that accounts for more than 99% of production costs; but it has no impact in software engineering because it affects less than 1% of production costs. Hence the automated development of software products is virtually impossible in general.
The only exception to this general rule is the development of applications within a limited application domain, where many of the design decisions may be taken a priori when the automated tool is developed and hardwired into the operation of the tool. One of the most successful areas of automated software development is compiler construction, where it is possible (thanks to several decades of intensive research) to produce compilers automatically, from a syntactic definition of the source language and relevant semantic definitions of its statements. Not surprisingly, this is a very narrow application domain.
1.7.3 Limited Quality Control
The lack of automation and hence the absence of process control make it difficult to control product quality. Whereas in traditional engineering disciplines, the production process is a systematic, repeatable process, one can control quality analytically by certifying the process or empirically by statistical observation. Because the production of software proceeds through a creative process, neither approach is feasible, since the process is neither systematic nor repeatable. This shifts the control of product quality to product controls, such as static analysis, or dynamic program testing.
1.7.4 Unbalanced Lifecycle Costs
In most other engineering disciplines, products are produced in large volume and are generally assumed to behave as expected; in software engineering, due to the foregoing discussion, such an assumption is unfounded, and the only way to ensure the quality of a software product is to subject that product to extensive analysis. This turns out to be an expensive proposition, in practice, and the source of another massive paradox in software engineering economics. Whereas testing (and more generally, verification and quality assurance) takes up a small percentage of the production cost of any engineering artifact, it accounts for a large percentage of the lifecycle cost of a software product. As a practical example, consider that the development of Windows Server 2003 (©Microsoft Corp.) was carried out by a team of 4400 software engineers, of whom 2000 formed the development team and a staggering 2400 formed the test team. More generally, testing accounts for around 50% of lifecycle costs, which is much higher than traditional manufacturing industries (where the likelihood of a defective product is so low as to make any significant amount of testing wasteful) (Table 1.2).
Table 1.2Lifecycle cost distribution: development versus testing
Software engineering, %
Other engineering, %
Development
≈50
>99
Testing
≈50
<1
Good software engineering practice dictates that more effort ought to be spent on up-front specification and design activities and that such up-front investment enhances product quality and lessens the need for massive investment in a posteriori testing. While these practices appear to be promising, they have not been used sufficiently widely to make a tangible impact; so that software testing remains a major cost factor in software lifecycles.
1.7.5 Unbalanced Maintenance Costs
It is common to distinguish in software maintenance between several types of maintenance activity; the two most important types (in terms of cost) are as follows:
Corrective maintenance, which aims to remove software faults
Adaptive maintenance, which aims to adapt the software product to evolving requirements
Empirical studies show that adaptive maintenance accounts for the vast majority of maintenance costs. This contrasts with other engineering disciplines, where there is virtually no adaptive maintenance to speak of: it is not possible for a car buyer to return to the dealership to make her/his car more powerful, add seats to it, or make it more fuel-efficient. Hence, it is possible to distinguish between software products and other engineering products by the distribution of maintenance, as shown in Table 1.3.
Table 1.3Maintenance cost distribution: corrective versus adaptive
Software engineering, %
Other engineering, %
Corrective
≈20
>99
Adaptive
≈80
<1
While it is not realistic to expect a car dealership to change a car to meet different specifications, it is certainly their responsibility to repair if it no longer meets its original specifications. Another distinguishing feature arises when one considers corrective maintenance: Whereas in software products corrective maintenance consists in changing the design or implementation of the product, in other engineering disciplines products need (corrective) maintenance due to wear and tear (Table 1.4).
Table 1.4Corrective maintenance cost distribution: design versus wear and tear
Software engineering, %
Other engineering, %
Design
≈100
1
Wear and tear
≈0
99
The only cases where a maintenance action on a brick-and-mortar product (e.g., a car) is of type design are cases where a manufacturer makes a product recall; these are sufficiently rare that they are usually newsworthy and are broadly advertised in public forums.
1.8 CHAPTER SUMMARY
This chapter introduces the discipline of software engineering with all its specific characteristics and paradoxes, contrasts it with more traditional engineering disciplines, and elucidates the role that software testing plays within this discipline.
1.9 BIBLIOGRAPHIC NOTES
For more information on the COCOMO cost model, consult (Boehm, 1981 ) or (Boehm et al., 2000); for more information on the JPL report on the evolution of avionics and space flight software, consult (Jet Propulsion Laboratory, 2009); for more information on the classification of software products into broad families of applications, consult (Somerville, 2004).
