109,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Fachliteratur
- Serie: Wiley-ASME Press Series
- Sprache: Englisch
Since mathematical models express our understanding of how nature behaves, we use them to validate our understanding of the fundamentals about systems (which could be processes, equipment, procedures, devices, or products). Also, when validated, the model is useful for engineering applications related to diagnosis, design, and optimization.
First, we postulate a mechanism, then derive a model grounded in that mechanistic understanding. If the model does not fit the data, our understanding of the mechanism was wrong or incomplete. Patterns in the residuals can guide model improvement. Alternately, when the model fits the data, our understanding is sufficient and confidently functional for engineering applications.
This book details methods of nonlinear regression, computational algorithms,model validation, interpretation of residuals, and useful experimental design. The focus is on practical applications, with relevant methods supported by fundamental analysis.
This book will assist either the academic or industrial practitioner to properly classify the system, choose between the various available modeling options and regression objectives, design experiments to obtain data capturing critical system behaviors, fit the model parameters based on that data, and statistically characterize the resulting model. The author has used the material in the undergraduate unit operations lab course and in advanced control applications.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 749
Veröffentlichungsjahr: 2016
Ähnliche
Table of Contents
Wiley-ASME Press Series List
Title Page
Copyright
Series Preface
Preface
Utility
Access to Computer Code
Preview of the Recommendations
Philosophy
Acknowledgments
Nomenclature
Symbols
Subscripts, Superscripts, and Marks
Part I: Introduction
Chapter 1: Introductory Concepts
1.1 Illustrative Example – Traditional Linear Least-Squares Regression
1.2 How Models Are Used
1.3 Nonlinear Regression
1.4 Variable Types
1.5 Simulation
1.6 Issues
1.7 Takeaway
Exercises
Chapter 2: Model Types
2.1 Model Terminology
2.2 A Classification of Mathematical Model Types
2.3 Steady-State and Dynamic Models
2.4 Pseudo-First Principles – Appropriated First Principles
2.5 Pseudo-First Principles – Pseudo-Components
2.6 Empirical Models with Theoretical Grounding
2.7 Empirical Models with No Theoretical Grounding
2.8 Partitioned Models
2.9 Empirical or Phenomenological?
2.10 Ensemble Models
2.11 Simulators
2.12 Stochastic and Probabilistic Models
2.13 Linearity
2.14 Discrete or Continuous
2.15 Constraints
2.16 Model Design (Architecture, Functionality, Structure)
2.17 Takeaway
Exercises
Part II: Preparation for Underlying Skills
Chapter 3: Propagation of Uncertainty
3.1 Introduction
3.2 Sources of Error and Uncertainty
3.3 Significant Digits
3.4 Rounding Off
3.5 Estimating Uncertainty on Values
3.6 Propagation of Uncertainty – Overview – Two Types, Two Ways Each
3.7 Which to Report? Maximum or Probable Uncertainty
3.8 Bootstrapping
3.9 Bias and Precision
3.10 Takeaway
Exercises
Chapter 4: Essential Probability and Statistics
4.1 Variation and Its Role in Topics
4.2 Histogram and Its PDF and CDF Views
4.3 Constructing a Data-Based View of PDF and CDF
4.4 Parameters that Characterize the Distribution
4.5 Some Representative Distributions
4.6 Confidence Interval
4.7 Central Limit Theorem
4.8 Hypothesis and Testing
4.9 Type I and Type II Errors, Alpha and Beta
4.10 Essential Statistics for This Text
4.11 Takeaway
Exercises
Chapter 5: Simulation
5.1 Introduction
5.2 Three Sources of Deviation: Measurement, Inputs, Coefficients
5.3 Two Types of Perturbations: Noise (Independent) and Drifts (Persistence)
5.4 Two Types of Influence: Additive and Scaled with Level
5.5 Using the Inverse CDF to Generate
n
and
u
from UID(0, 1)
5.6 Takeaway
Exercises
Chapter 6: Steady and Transient State Detection
6.1 Introduction
6.2 Method
6.3 Applications
6.4 Takeaway
Exercises
Part III: Regression, Validation, Design
Chapter 7: Regression Target – Objective Function
7.1 Introduction
7.2 Experimental and Measurement Uncertainty – Static and Continuous Valued
7.3 Likelihood
7.4 Maximum Likelihood
7.5 Estimating
σ
x
and
σ
y
Values
7.6 Vertical SSD – A Limiting Consideration of Variability Only in the Response Measurement
7.7 r-Square as a Measure of Fit
7.8 Normal, Total, or Perpendicular SSD
7.9 Akaho's Method
7.10 Using a Model Inverse for Regression
7.11 Choosing the Dependent Variable
7.12 Model Prediction with Dynamic Models
7.13 Model Prediction with Classification Models
7.14 Model Prediction with Rank Models
7.15 Probabilistic Models
7.16 Stochastic Models
7.17 Takeaway
Exercises
Chapter 8: Constraints
8.1 Introduction
8.2 Constraint Types
8.3 Expressing Hard Constraints in the Optimization Statement
8.4 Expressing Soft Constraints in the Optimization Statement
8.5 Equality Constraints
8.6 Takeaway
Exercises
Chapter 9: The Distortion of Linearizing Transforms
9.1 Linearizing Coefficient Expression in Nonlinear Functions
9.2 The Associated Distortion
9.3 Sequential Coefficient Evaluation
9.4 Takeaway
Exercises
Chapter 10: Optimization Algorithms
10.1 Introduction
10.2 Optimization Concepts
10.3 Gradient-Based Optimization
10.4 Direct Search Optimizers
10.5 Takeaway
Chapter 11: Multiple Optima
11.1 Introduction
11.2 Quantifying the Probability of Finding the Global Best
11.3 Approaches to Find the Global Optimum
11.4 Best-of-
N
Rule for Regression Starts
11.5 Interpreting the CDF
11.6 Takeaway
Chapter 12: Regression Convergence Criteria
12.1 Introduction
12.2 Convergence versus Stopping
12.3 Traditional Criteria for Claiming Convergence
12.4 Combining DV Influence on OF
12.5 Use Relative Impact as Convergence Criterion
12.6 Steady-State Convergence Criterion
12.7 Neural Network Validation
12.8 Takeaway
Exercises
Chapter 13: Model Design – Desired and Undesired Model Characteristics and Effects
13.1 Introduction
13.2 Redundant Coefficients
13.3 Coefficient Correlation
13.4 Asymptotic and Uncertainty Effects When Model is Inverted
13.5 Irrelevant Coefficients
13.6 Poles and Sign Flips w.r.t. the DV
13.7 Too Many Adjustable Coefficients or Too Many Regressors
13.8 Irrelevant Model Coefficients
13.9 Scale-Up or Scale-Down Transition to New Phenomena
13.10 Takeaway
Exercises
Chapter 14: Data Pre- and Post-processing
14.1 Introduction
14.2 Pre-processing Techniques
14.3 Post-processing
14.4 Takeaway
Exercises
Chapter 15: Incremental Model Adjustment
15.1 Introduction
15.2 Choosing the Adjustable Coefficient in Phenomenological Models
15.3 Simple Approach
15.4 An Alternate Approach
15.5 Other Approaches
15.6 Takeaway
Exercises
Chapter 16: Model and Experimental Validation
16.1 Introduction
16.2 Logic-Based Validation Criteria
16.3 Data-Based Validation Criteria and Statistical Tests
16.4 Model Discrimination
16.5 Procedure Summary
16.6 Alternate Validation Approaches
16.7 Takeaway
Exercises
Chapter 17: Model Prediction Uncertainty
17.1 Introduction
17.2 Bootstrapping
17.3 Takeaway
Chapter 18: Design of Experiments for Model Development and Validation
18.1 Concept – Plan and Data
18.2 Sufficiently Small Experimental Uncertainty – Methodology
18.3 Screening Designs – A Good Plan for an Alternate Purpose
18.4 Experimental Design – A Plan for Validation and Discrimination
18.5 EHS&LP
18.6 Visual Examples of Undesired Designs
18.7 Example for an Experimental Plan
18.8 Takeaway
Exercises
Chapter 19: Utility versus Perfection
19.1 Competing and Conflicting Measures of Excellence
19.2 Attributes for Model Utility Evaluation
19.3 Takeaway
Exercises
Chapter 20: Troubleshooting
20.1 Introduction
20.2 Bimodal and Multimodal Residuals
20.3 Trends in the Residuals
20.4 Parameter Correlation
20.5 Convergence Criterion – Too Tight, Too Loose
20.6 Overfitting (Memorization)
20.7 Solution Procedure Encounters Execution Errors
20.8 Not a Sharp CDF (OF)
20.9 Outliers
20.10 Average Residual Not Zero
20.11 Irrelevant Model Coefficients
20.12 Data Work-Up after the Trials
20.13 Too Many
r
s!
20.14 Propagation of Uncertainty Does Not Match Residuals
20.15 Multiple Optima
20.16 Very Slow Progress
20.17 All Residuals are Zero
20.18 Takeaway
Exercises
Part IV: Case Studies and Data
Chapter 21: Case Studies
21.1 Valve Characterization
21.2 CO
2
Orifice Calibration
21.3 Enrollment Trend
21.4 Algae Response to Sunlight Intensity
21.5 Batch Reaction Kinetics
Appendix A: VBA Primer: Brief on VBA Programming – Excel in Office 2013
A.1 To Start
A.2 General
A.3 I/O to Excel Cells
A.4 Variable Types and Declarations
A.5 Operations
A.6 Loops
A.7 Conditionals
A.8 Debugging
A.9 Run Buttons (Commands)
A.10 Objects and Properties
A.11 Keystroke Macros
A.12 External File I/O
A.13 Solver Add-In
A.14 Calling Solver from VBA
A.15 Access to Computer Code
Appendix B: Leapfrogging Optimizer Code for Steady-State Models
Appendix C: Bootstrapping with Static Model
References and Further Reading
Index
End User License Agreement
Pages
xiii
xv
xvi
xvii
xviii
xix
xx
xxi
xxiii
xxv
xxvi
xxvii
xxviii
xxix
xxx
xxxi
xxxii
xxxiii
xxxiv
xxxv
xxxvi
xxxvii
xxxviii
1
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
41
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
117
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
297
298
299
300
301
302
303
304
305
307
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
341
342
343
344
345
346
347
348
349
350
351
352
353
355
356
357
358
359
360
361
Guide
Table of Contents
Preface
Part I
Begin Reading
List of Illustrations
Chapter 1: Introductory Concepts
Figure 1.1 Illustration of regression concepts
Figure 1.2 Illustration of regression coefficient optimization
Figure 1.3 Simulation concept
Chapter 3: Propagation of Uncertainty
Figure 3.1 Illustration of variability in measurements and relation of standard deviation and error
Figure 3.2 Illustration of the relation between uncertainty on
y
=
f
(
x
) w.r.t. uncertainty in the value of
x
Chapter 4: Essential Probability and Statistics
Figure 4.1 Histogram represented by line segments that connect the bin tops at bin center
Figure 4.2 Cumulative distribution of the data in Figure 4.1
Figure 4.3 Probability distribution function for normally distributed data
Figure 4.4 Cumulative distribution function for normally distributed data
Figure 4.5 Experimental CDF plot
Figure 4.6 Experimental PDF plot illustrating the impact on the user's choice of
n
Figure 4.7 Probability distribution function for a log-normal distribution
Figure 4.8 Analysis of a log-normal distribution
Figure 4.9 Logistic model
Figure 4.10 Probability distribution function for an exponential distribution
Figure 4.11 Illustration of a confidence interval
Figure 4.12 PDF of a statistic when H
0
is true
Figure 4.13 PDF of a statistic when H
0
is true or not true
Figure 4.14 Increasing
N
in Figure 4.13
Chapter 5: Simulation
Figure 5.1 Simulation of experimental data
Figure 5.2 A realization of an ARMA(1, 1) drift
Figure 5.3 CDF used to generate random variable values
Chapter 6: Steady and Transient State Detection
Figure 6.1 Illustration of a transient trend to the steady state
Figure 6.2 Illustration actual process (dashed line), noisy measurements (diamond markers), filtered data (solid line), and deviations
Figure 6.3 Data showing autocorrelation
Figure 6.4 Data showing no autocorrelation when the interval is five samplings
Figure 6.5 Example
Figure 6.6 SSID application to nonlinear regression
Chapter 7: Regression Target – Objective Function
Figure 7.1 Illustration of two types of experimental uncertainty
Figure 7.2 Illustration of the Gaussian distribution
Figure 7.3 Illustration of maximum likelihood in regression
Figure 7.4 Equivalence of likelihood interpretation
Figure 7.5 Illustration of a wrong model functionality that seems right
Figure 7.6 Maximum likelihood illustration in scaled variable space
Figure 7.7 Illustration of Akaho's technique
Figure 7.8 Comparison of probable error on normal and vertical deviations in nonlinear regression
Figure 7.9 Illustration of a dynamic response
Chapter 8: Constraints
Figure 8.1 Illustration of equal concern deviations
Figure 8.2 Illustration that a soft constraint permits a small violation
Chapter 9: The Distortion of Linearizing Transforms
Figure 9.1 Nonlinearity of the log transformation
Figure 9.2 Illustration of experimental data
Figure 9.3 Log-transformed model and data
Figure 9.4 Sequential coefficient evaluations: (a) the findings and (b) comparison to simultaneous regression
Chapter 10: Optimization Algorithms
Figure 10.1 Illustration of a 2-DV optimization, 3-D surface net, and 2-D contour map
Figure 10.2 Illustration of numerical approximations to the slope
Figure 10.3 Illustration of possible misdirection by numerical slope estimates
Figure 10.4 Illustration of steepest descent in a 2-DV contour map
Figure 10.5 An example of incremental steepest descent
Figure 10.6 Leapfrogging in a two-dimensional function
Figure 10.7 Leapfrogging procedure flow chart
Chapter 11: Multiple Optima
Figure 11.1 Illustrations of multiple optima for a single DV model
Figure 11.2 Illustrations of multiple optima for a two DV model – contour plot
Figure 11.3 Quantifying multiple optima
Figure 11.4 Histogram of OF results
Figure 11.5 Histogram of OF results – greater detail
Figure 11.6 Probability density function of OF results
Figure 11.7 Varieties of PDF(OF) results and associated CDF(OF) graphs
Figure 11.8 Illustration for interpreting developing CDF graphs
Chapter 12: Regression Convergence Criteria
Figure 12.1 Illustration of the problem with threshold on OF as convergence criterion
Figure 12.2 Illustration of the problem with threshold on ΔOF as convergence criterion
Figure 12.3 Illustration of the problem with threshold on ΔDV as convergence criterion
Figure 12.4 Characteristic OF approach to optimum w.r.t. iteration
Figure 12.5 Characteristic SSD or rms from random subset approach to optimum
Figure 12.6 Characteristic approach to optimum
Figure 12.7 Distributions of the ratio-statistic value as process changes state
Figure 12.8 Illustration of critical values around steady state
Figure 12.9 Random subset RMS deviation versus optimization iteration
Figure 12.10 Random subset RMS deviation versus optimization iteration – bimodal trace
Chapter 13: Model Design – Desired and Undesired Model Characteristics and Effects
Figure 13.1 A view of parameter correlation
Figure 13.2 An indication of parameter correlation
Figure 13.3 Illustration of noninvertibility and polytonic problems
Figure 13.4 Illustration of uncertainty
Figure 13.5 Illustration of sensitivity, uncertainty, and polytonic problems
Figure 13.6 Chemical engineering enrollment cycling
Figure 13.7 Illustration of wrong choice of input variable
Figure 13.8 Illustration of the right regressor but too few adjustable coefficients
Figure 13.9 Illustration of number of adjustable coefficients
Figure 13.10 Illustration of too many adjustable coefficients
Chapter 14: Data Pre- and Post-processing
Figure 14.1 Concept for a noisy steady state
Figure 14.2 The delay of
N
samples with a moving average
Figure 14.3 Illustration of the lag of a first-order filter
Figure 14.4 An illustration of normal residuals
Figure 14.5 An illustration of bimodal residuals
Chapter 16: Model and Experimental Validation
Figure 16.1 Linear model: good
r
-square does not mean a good model
Figure 16.2 Quadratic model: good fit to data, but does not extrapolate
Figure 16.3 Quadratic model inadequacy is exposed by data with less variability
Figure 16.4 Comparison of stochastic model to data
Figure 16.5 Comparison of stochastic model to data – too few bins
Figure 16.6 Comparison of stochastic model to data – too many bins
Figure 16.7 Comparison of stochastic model to data – CDF approach
Figure 16.8 Bias illustration
Figure 16.10 Curvature illustration
Figure 16.9 Skew illustration
Figure 16.11 Illustration of bias but no curvature or skew
Figure 16.12 Illustration of curvature but not bias and the impact of data order
Figure 16.13 Nonuniform variance
Figure 16.14 Data with skew
Chapter 17: Model Prediction Uncertainty
Figure 17.1 Bootstrapping estimate of model uncertainty due to data.
Chapter 18: Design of Experiments for Model Development and Validation
Figure 18.4 Levels in experimentation prevent evaluation of in between behavior.
Figure 18.1 Desirable data patterns.
Figure 18.2 Regions of missing data.
Figure 18.3 Range is not large relative to variability.
Figure 18.5 Array.
Figure 18.6 Array run order.
Figure 18.7 A better plan.
Figure 18.8 Experimental plan evolution: (a) the initially desired plan; (b) parity plot reveals voids; (c) revised plan adds data (triangles) to fill the gaps; (d) parity plot with added data shows full coverage.
List of Tables
Preface
Table 1 Bloom's taxonomy
Table 2 Desired engineering attributes
Chapter 1: Introductory Concepts
Table 1.1 Illustration of data for Figure 1.1
Table 1.2 Nomenclature for variable types
Chapter 2: Model Types
Table 2.1 Steady-state data and model
Table 2.2 Steady-state data and model with data rearranged
Table 2.3 Dynamic data and model
Chapter 3: Propagation of Uncertainty
Table 3.1 Rounding examples
Table 3.2 Numerical propagation of maximum uncertainty
Chapter 4: Essential Probability and Statistics
Table 4.1 Example of calculating the Wilcoxon signed rank statistic
Chapter 7: Regression Target – Objective Function
Table 7.1 Illustration of a classification model
Table 7.2 Illustration of rank assessments
Chapter 11: Multiple Optima
Table 11.1 The impact of repeating optimization trials from random locations (if the probability of finding the global in any one trial is 0.20)
Chapter 14: Data Pre- and Post-processing
Table 14.1 First consideration of a chi-square contingency analysis
Table 14.2 A chi-square contingency analysis
Chapter 16: Model and Experimental Validation
Table 16.1 A contingency Table of outcomes
Wiley-ASME Press Series List
Introduction to Dynamics and Control of MechanicalEngineering Systems
To
March 2016
Fundamentals of Mechanical Vibrations
Cai
May 2016
Nonlinear Regression Modeling for EngineeringApplications
Rhinehart
August 2016
Stress in ASME Pressure Vessels
Jawad
November 2016
Bioprocessing Piping and Equipment Design
Huitt
November 2016
Combined Cooling, Heating, and Power Systems
Shi
January 2017
Nonlinear Regression Modeling for Engineering Applications
Modeling, Model Validation, and Enabling Design of Experiments
R. Russell Rhinehart
This edition first published 2016
© 2016, John Wiley & Sons, Ltd
First Edition published in 2016
Registered office
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom
For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.
The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.
Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. It is sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a competent professional should be sought.
Library of Congress Cataloging-in-Publication Data:
Names: Rhinehart, R. Russell, 1946- author.
Title: Nonlinear regression modeling for engineering applications : modeling,
model validation, and enabling design of experiments / R. Russell
Rhinehart.
Description: Chichester, UK ; Hoboken, NJ : John Wiley & Sons, 2016. |
Includes bibliographical references and index.
Identifiers: LCCN 2016012932 (print) | LCCN 2016020558 (ebook) | ISBN
9781118597965 (cloth) | ISBN 9781118597934 (pdf) | ISBN 9781118597958
(epub)
Subjects: LCSH: Regression analysis–Mathematical models. |
Engineering–Mathematical models.
Classification: LCC TA342 .R495 2016 (print) | LCC TA342 (ebook) | DDC
620.001/519536–dc23
LC record available at https://lccn.loc.gov/2016012932
A catalogue record for this book is available from the British Library.
Series Preface
The Wiley-ASME Press Series in Mechanical Engineering brings together two established leaders in mechanical engineering publishing to deliver high-quality, peer-reviewed books covering topics of current interest to engineers and researchers worldwide.
The series publishes across the breadth of mechanical engineering, comprising research, design and development, and manufacturing. It includes monographs, references and course texts.
Prospective topics include emerging and advanced technologies in Engineering Design; Computer-Aided Design; Energy Conversion & Resources; Heat Transfer; Manufacturing & Processing; Systems & Devices; Renewable Energy; Robotics; and Biotechnology.
Preface
Utility
Mathematical models are important.
Engineers use mathematical models to describe the natural world and then rearrange the model equations to answer the question, “How do I create an environment that makes Nature behave the way I want it to?” The answer to the mathematical rearrangement of the model equations reveals how to design processes, products, and procedures. It also reveals how to operate, use, monitor, and control them. Modeling is a critical underpinning for engineering analysis, design, control, and system optimization.
Further, since mathematical models express our understanding of how Nature behaves, we use them to validate our understanding of the fundamentals about processes and products. We postulate a mechanism and then derive a model grounded in that mechanistic understanding. If the model does not fit the data, our understanding of the mechanism was wrong or incomplete. Alternately, if the model fits the data we can claim our understanding may be correct. Models help us develop knowledge.
These models usually have coefficients representing some property of Nature, which has an unknown value (e.g., the diffusivity of a new molecule in a new medium, drag coefficient on a new shape, curing time of a new concrete mix, a catalyst effective surface area per unit mass, a heat transfer fouling factor). Model coefficient values must be adjusted to make the model match the experimentally obtained data, and obtaining the value of the coefficient adds to knowledge.
The procedure for finding the model coefficient values that makes a model best fit the data is called regression.
Although regression is ages old, there seem to be many opportunities for improvements related to finding a global optimum; finding a universal, effective, simple, and single stopping criterion for nonlinear regression; validating the model; balancing model simplicity and sufficiency with perfection and complexity; discriminating between competing models; and distinguishing functional sufficiency from prediction accuracy.
I developed and used process and product models throughout my 13-year industrial career. However, my college preparation for the engineering career did not teach me what I needed to know about how to create and evaluate models. I recognized that my fellow engineers, regardless of their alma mater, were also underprepared. We had to self-learn as to what was needed. Recognizing the centrality of modeling to engineering analysis, I have continued to explore model development and use during my subsequent academic career.
This textbook addresses nonlinear regression from a perspective that balances engineering utility with scientific perfection, a view that is often missing in the classroom, wherein the focus is often on the mathematical analysis, which pretends that there are simple, first-attempt solutions. Mathematical analysis is intellectually stimulating and satisfying, and sometimes useful for the practitioner. Where I think it adds value, I included analysis in this book. However, development of a model, choosing appropriate regression features, and designing experiments to generate useful data are iterative procedures that are guided by insight from progressive experience. It would be a rare event to jump to the right answers on the first try. Accordingly, balancing theoretical analysis, this book provides guides for procedure improvement.
This work is a collection of what I consider to be best practices in nonlinear regression modeling, which necessarily includes guides to design experiments to generate the data and guides to interpret the models. Undoubtedly, my view of best has been shaped with my particular uses for the models within the context of process and product modeling. Accordingly, this textbook has a focus on models with continuous-valued variables (either deterministic, discretized, or probabilities) as opposed to rank or classification, nonlinear as opposed to linear, constrained as opposed to not, and of a modest number of variables as opposed to Big Data.
This textbook includes the material I wish I had known when starting my engineering career and now what I would like my students to know. I hope it is useful for you.
The examples and discussion presume basic understanding of engineering models, regression, statistics, optimization, and calculus. This textbook provides enough details, explicit equation derivations, and examples to be useful as an introductory learning device for an upper-level undergraduate or graduate. I have used much of this material in the undergraduate unit operations lab course, in my explorations of model-based control on pilot-scale units, and in modeling of diverse processes (including the financial aspects of my retirement and the use of academic performance in the first two college years to project upper-level success). A person with an engineering degree and some experience with regression should be able to follow the concepts, analysis, and discussion.
My objective is to help you answer these questions:
How to choose model inputs (variables, delays)?
How to choose model form (linear, quadratic, or higher order, or equivalent model structures or architectures such as dimension or number of neurons)?
How to design experiments to obtain adequate data (in number, precision, and placement) for determining model coefficient values?
What to use for the regression objective (vertical least squares, total least squares, or maximum likelihood)?
How to define goodness of model (
r
-square, fitness for use, utility, simplicity, data-based validation, confidence interval for prediction)?
How to choose the right model between two different models?
What optimization algorithm should be used for the regression to be able to handle the confounding issues of hard or soft constraints, discontinuities, discrete and continuous variables, multiple optima, and so on?
What convergence criteria should be used to stop the optimizer (to recognize when it is close enough to optimum)?
Should you linearize and use linear regression or use nonlinear regression?
How to recognize outliers?
How can you claim that a model properly captures some natural phenomena?
The underlying techniques needed for the answers include propagation of uncertainty, probability and statistics, optimization, and experience and heuristics. The initial chapters review/develop the basics. Subsequent chapters provide the application techniques, description of the algorithms, and guides for application.
Access to Computer Code
Those interested can visit the author's web site, www.r3eda.com, for open access to Excel VBA macros to many of the procedures in this book.
Years back our college decided to standardize with Visual Basic for Applications (VBA) for the undergraduate computer programming course. As a result, routines supporting this text are written in VBA, which is convenient to me, and also a widely accessible platform. However, VBA is not the fastest, and some readers may not be familiar with that language. Therefore, this text also provides a VBA primer and access to the code so that a reader may convert the VBA code to some other personally preferred platform. If you understand any structured text procedures, you can understand the VBA code here.
Preview of the Recommendations
Some of the recommendations in this book are counter to traditional practice in regression and design of experiments (DoE), which seem to be substantially grounded in linear regression. As a preview, opinions offered in this textbook are:
If the equation is nonlinear in the coefficients, use nonlinear regression. Even if the equation can be log-transformed into a linear form, do not do it. Linearizing transformations distort the relative importance of data points within the data set. Unless data variance is relatively low and/or there are many data points, linearizing can cause significant error in the model coefficient values.
Use data pre-processing and post-processing to eliminate outliers.
Use direct search optimizers for nonlinear regression rather than gradient-based optimizers. Although gradient-based algorithms converge rapidly in the vicinity of the optimum, direct search optimizers are more robust to surface aberrations, can cope with hard constraints, and are faster for difficult problems. Leapfrogging is offered as a good optimizer choice.
Nonlinear regression may have multiple minima. No optimizer can guarantee finding the global minimum on a first trial. Therefore, run the optimizer for
N
trials, starting from random locations, and take the best of the
N
trials.
N
can be calculated to meet the user desire for the probability of finding an optimum within a user-defined best fraction. The equation is shown.
Pay as much attention to how constraints are defined and included in the optimization application as you do to deriving the model and objective function (OF) statement. Constraints can have a substantial influence on the regression solution.
The choice of stopping criteria is also influential to the solution. Conventional stopping criteria are based on thresholds on the adjustable model coefficient values (decision variables, DVs), and/or the regression target (usually the sum of squared deviations) that we are seeking to optimize (OF). Since the right choice for the thresholds requires
a priori
knowledge, is scale-dependent, and requires threshold values on each regression coefficient (DV) and/or optimization target (OF), determining right threshold values requires substantial user experience with the specific application. This work recommends using steady-state identification to declare convergence. It is a single criterion (only looking at one index – statistical improvement in OF relative to data variability from the model), which is not scale-dependent.
Design the experimental plan (sequence, range, input variables) to generate data that are useful for testing the validity of the nonlinear model. Do not follow conventional statistical DoE methods, which were devised for alternate outcomes – to minimize uncertainty on the coefficients in nonmechanistic models, in linear regression, within idealized conditions.
Design the experimental methods of gathering data (measurement protocol, number and location of data sets) so that uncertainty on the experimental measurements has a minimal impact on model coefficient values.
Use of the conventional least-squares measure of model quality, ∑(
y
data
−
y
model
)
2
, is acceptable for most purposes. It can be defended by idealizing maximum likelihood conditions. Maximum likelihood is more compatible with reality and can provide better model coefficient values, but it presumes knowledge of the variance on both experimental inputs and output, and requires a nested optimization. Maximum likelihood can be justified where scientific precision is paramount, but adds complexity to the optimization.
Akaho's method is a computationally simple improvement for the total east-squares approximation to maximum likelihood.
Establish nonlinear model validity with statistical tests for bias and either autocorrelation or runs. Do not use
r
-square or ANOVA techniques, which were devised for linear regression under idealized conditions.
Eliminate redundant coefficients, inconsequential model terms, and inconsequential input variables.
Perform both logic-based
and
data-based tests to establish model validity.
Model utility (fitness for use) and model validity (representation of the truth about Nature) are different. Useful models often do not need to be true. Balance perfection with sufficiency, complexity with simplicity, rigor with utility.
Philosophy
I am writing to you, the reader, in a first-person personal voice, a contrast to most technical works. There are several aspects that led me to do so, but all are grounded in the view that humans will be implementing the material.
I am a believer in the Scientific Method. The outcomes claimed by a person should be verifiable by any investigator. The methodology and analysis that led to the outcomes should be grounded in the widely accepted best practices. In addition, the claims should be tempered and accepted by the body of experts. However, the Scientific Method wants decisions to be purely rational, logical, and fact based. There should be no personal opinion, human emotion, or human bias infecting decisions and acceptances about the truth of Nature. To preserve the image of no human involvement, most technical writing is in the third person. However, an author's choice of idealizations, acceptances, permissions, assumptions, givens, basis, considerations, suppositions, and such, are necessary to permit mathematical exactness, proofs, and the consequential absolute statements. However, the truth offered is implicitly infected by the human choices. If a human is thinking it, or if a human accepts it, it cannot be devoid of that human's perspective and values. I am not pretending that this book is separate from my experiences and interpretations so I am writing in the first person.
Additionally, consider the individuals applying techniques. They are not investigating a mathematical analysis underlying the technique, but need to use the technique to get an answer for some alternate purpose. Accordingly, utility with the techniques is probably as important as understanding the procedure basis. Further, the application situation is not an idealized simplification. Nature confounds simplicity with complexity. Therefore, as well as proficiency in use, a user must understand and interpret the situation and choose the right techniques. The human applies it and the human must choose the appropriate technique. Accordingly, to make a user functional, it is important for a textbook to understand the limits and appropriateness of techniques. The individual is the agent and primary target, the tool is just the tool. The technique is not the truth, so I am writing to the user.
It is also essential that a user truly understands the basis of a tool, to use it properly. Accordingly, in addition to discussing the application situations, this text develops the equations behind the methods, includes mathematical analysis, and reveals nuances through examples. The book also includes exercises so the user can develop skills and understanding.
In the 1950s Benjamin Bloom chaired a committee of educators that subsequently published a taxonomy of Learning Objectives, which has come to be known as Bloom's Taxonomy. One of the domains is termed the Cognitive, related to thinking/knowing. There are six levels in the Taxonomy. Here is my interpretation for engineering (Table 1).
Table 1 Bloom's taxonomy
Level
Name
Function – person does
Examples
6
Evaluation (E)
Judge goodness, sufficiency, and completeness of something, choose the best among options, know when to stop improving. Must consider all aspects
Decide that a design, report, research project, or event planning is finished when considering all issues (technical completeness, needs of all stakeholders, ethical standards, safety, economics, impact, etc.)
5
Synthesis (S)
Create something new: purposefully integrate parts or concepts to design something new that meets a function
Design a device to meet all stakeholders' approvals within constraints. Create a new homework problem integrating all relevant technology, design a procedure to meet multiple objectives, create a model, create a written report, design experiments to generate useful data
4
Analysis (An)
Two aspects related to context
One
. Separate into parts or stages, define and classify the mechanistic relationships of something within the whole
One
. Describe and model the sequence of cause-and-effect mechanisms: tray-to-tray model that relates vapor boil-up to distillate purity, impact of transformer start-up on the entire grid, impact of an infection on the entire body and person health
Two
. Critique, assess goodness, determine functionality of something within the whole
Two
. Define and compute metrics that quantify measures of utility or goodness
3
Application (Ap)
Independently apply skills to fulfill a purpose within a structured set of “givens”
Properly follow procedures to calculate bubble point, size equipment, use the Excel features to properly present data, solve classroom “word problems”
2
Understanding/comprehension (U/C)
Understand the relation of facts and connection of abstract to concrete
Find the diameter of a 1-inch diameter pipe, convert units, qualitatively describe staged equilibrium separation phenomena, explain the equations that describe an RC circuit, understand what Excel cell equations do
1
Knowledge (K)
Memorize facts and categorization
Spell words, recite equations, name parts of a valve, read resistance from color code, recite the six Bloom levels
Notably most of classroom instruction has the student working in the lower three levels, where there are no user-choices. There is only one way to spell “cat,” only one right answer to the calculation of the required orifice diameter using the ideal orifice equation and givens in the word problem, and so on. In school, the instructor analyzes the situation, synthesizes the exercise, and judges the correctness of the answer. By contrast, competency and success in professional and personal life requires the individual to mentally work in the upper levels where the situation must be interpreted, where the approach must be synthesized, and where the propriety of the approach and answer must be evaluated. When instruction prevents the student from working in the upper cognitive levels, it misrepresents the post-graduation environment, which does a disservice to the student and employers who have to redirect the graduate's perspective. Accordingly, my aim is to facilitate the reader's mental activity in the upper levels where human choices have to be made. I am therefore writing to the human, not just about the technology.
A final perspective, on the philosophy behind the style and contents of this book is grounded in a list of desired engineering attributes. The members of the Industrial Advisory Committee for our School helped the faculty develop a list of desired engineering attributes, which we use to shape what we teach and shape the student's perspectives. Engineering is an activity, not a body of knowledge. Engineering is performed by humans within a human environment; it is not the intellectual exercise about isolated mathematical analysis. There are opposing ideals in judging engineering and the list of Desired Engineering Attributes reveals them. The opposing ideals are highlighted in bold (Table 2).
Table 2 Desired engineering attributes
Engineering is an activity that delivers solutions that work for all stakeholders. Desirably engineering:
Seeks
simplicity
in analysis and solutions, while being
comprehensive
in scope.
Is
careful
, correct, self-critical, and defensible; yet is performed with a
sense of urgency
.
Analyzes
individual mechanisms
and integrates stages to
understand the whole
.
Uses state-of-the-art
science
and
heuristics
.
Balances
sufficiency
with
perfection.
Develops
sustainable solutions
– profitable and accepted
today
, without burdening
future stakeholders
.
Tempers
personal gain
with
benefit to others
.
Is
creative
, yet
follows codes
, regulations, and standard practices.
Balances probable
loss
with probable
gain
but not at the expense of EHS&LP –
manages risk
.
Is a collaborative,
partnership activity
, energized by
individuals
.
Is an
intellectual analysis
that leads to
implementation and fruition
.
Is
scientifically valid
, yet
effectively communicated
for all stakeholders.
Generates
concrete
recommendations that honestly reveal
uncertainty
.
Is grounded in
technical fundamentals
and the
human context
(societal, economic, and political).
Is grounded in
allegiance to the bottom line of the company
and to
ethical standards of technical and personal conduct
.
Supports
enterprise harmony
while seeking to
cause beneficent change
.
Engineering is not just about technical competence. State-of-the-art commercial software beats novice humans in speed and completeness with technical calculations. Engineering is a decision-making process about technology within human enterprises, value systems, and aspirations, and I believe this list addresses a fundamental aspect of the essence of engineering. As a complement to fundamental knowledge and skill of the core science and technical topics, instructors need to understand the opposing ideals, the practice of application, so that they can integrate the issues into the student's experience and so that student exercises have students practice right perspectives as they train for technical competency.
A straight line is very long. Maybe the line goes between pure science on one end and pure follow-the-recipe and accept-the-computer-output on the other end. No matter where one stands, the line disappears into the horizons to the left and to the right. No matter where one stands, it feels like the middle, the point of right balance between the extremes. However, the person way to the left also thinks they are in the middle. If Higher Education is to prepare graduates for industrial careers, instructors need to understand the issues surrounding Desired Engineering Attributes from an industrial perspective, not their academic/science perspective. Therefore, I am writing to the human about how to balance those opposing ideals when using nonlinear regression techniques for applications.
Acknowledgments
My initial interest in modeling processes and products arose from my engineering experience within industry, and most of the material presented here benefited from the investigations of my graduate students as they explored the applicability of these tools, guidance from industrial advisors as they provided input on graduate projects and undergraduate education outcomes, and a few key mentors who helped me see these connections. Thank you all for revealing issues, providing guidance, and participating in my investigations.
Some of the techniques in this text are direct outcomes of the research performed by Gaurav Aurora, R. Paul Bray, Phoebe Brown, Songling Cao, Chitan Chandak, Sandeep Chandran, Solomon Gebreyohannes, Anand Govindrajanan, Mahesh S. Iyer, Suresh Jayaraman, Junyi Li, Upasana Manimegalai-Sridhar, Siva Natarajan, Jing Ou, Venkat Padmanabhan, Anirudh (Andy) Patrachari, Neha Shrowti, Anthony Skach, Ming Su, John Szella, Kedar Vilankar, and Judson Wooters.
A special thanks goes to Robert M. Bethea (Bob) who invited me to coauthor the text Applied Engineering Statistics, which was a big step toward my understanding of the interaction between regression, modeling, experimental design, and data analysis. Another special thank you to Richard M. Felder, always a mentor in understanding and disseminating engineering science and technology.
As a professor, funding is essential to enable research, investigation, discovery, and the pursuit of creativity. I am grateful to both the Edward E. and Helen Turner Bartlett Foundation and the Amoco Foundation (now BP) for funding endowments for academic chairs. I have been fortunate to be the chair holder for one or the other, which means that I was permitted to use some proceeds from the endowment to attract and support graduate students who could pursue ideas that did not have traditional research support. This book presents many of the techniques explored, developed, or tested by the graduate students.
Similarly, I am grateful for a number of industrial sponsors of my graduate program who recognized the importance of applied research and its role in workforce development.
Most of all, career accomplishments of any one person are the result of the many people who nurtured and developed the person. I am of course grateful to my parents, teachers, and friends, but mostly to Donna, who for the past 26 years has been everything I need.
Nomenclature
Accept
Not reject. There is not statistically sufficient evidence to confidently claim that the null hypothesis is not true. There is not a big enough difference. This is equivalent to the not guilty verdict, when the accused might have done it, but the evidence is not beyond reasonable doubt. Not guilty does not mean innocent. Accept means cannot confidently reject and does not mean correct.
Accuracy
Closeness to the true value, bias, average deviation. In contrast to precision.
AIC
Akiake Information Criterion, a method for assessing the balance of model complexity to fit to data.
A priori
Latin origin for “without prior knowledge.”
Architecture
The functional form of the mathematical model.
ARL
Average run length, the average number of samples to report a confident result.
Autocorrelation
One value of a variable that changes in time is related to prior values of that variable.
Autoregressive
A mathematical description that one value of a variable that changes in time is related to prior values of that variable; the cause would be some fluctuating input that has a persisting influence.
Batch regression
The process of regression operates on all of the data in one operation.
Best-of-N
Start the optimizer
N
times with independent initializations and take the best of the
N
trials as the answer.
Bias
A systematic error, a consistent shift in level, an average deviation from true.
Bimodal
A pattern in the residuals that indicates there are two separate distributions, suggesting two separate treatments affected the data.
Bootstrapping
A numerical, Monte Carlo, technique for estimating the uncertainty in a model-predicted value from the inherent variability in the data used to regress model coefficient values.
Cardinal
Integers, counting numbers, a quantification of the number of items.
Cauchy's technique
An optimization approach of successive searches along the line of local steepest descent.
CDF
The cumulative distribution function, the probability of obtaining an equal or smaller value.
Chauvenet's criterion
A method for selecting data that could be rejected as an outlier.
Class
The variable that contains the name of a classification – nominal, name, category.
Coefficient correlation
When the optimizer does not find a unique solution, perhaps many identical or nearly identical OF values for different DV values, a plot of one DV value w.r.t. another reveals that one coefficient is correlated to the other. Often termed parameter correlation.
Coefficient or model coefficient
A symbol in a model that has a fixed value from the model use perspective. Model constants or parameters. Some values are fundamental such as Pi or the 2 in square root. Other values for the coefficients are determined by fitting model to data. Such coefficient values will change as new data is added.
Confidence
The probability that a statement is true.
Constraints
Boundaries that cannot be violated, often rational limits for regression coefficients.
Convergence
The optimizer trial solution has found the proximity of the optimum within desired precision.
Convergence criterion
The metric used to test for convergence – could be based on the change in DVs, change in OF, and so on.
Correlation
Two variables are related to each other. If one rises, the other rises. The relation might be confounded by noise and variation, and represent a general, not exact relation. The relation does not have to be linear.
Cross correlation
Two separate variables are related to each other. Contrast to autocorrelation in which values of one variable are related to prior values.
Cumulative sum
CUSUM, cumulative sum of deviations scaled by the standard deviation in the data.
CUSUM
Cumulative sum of deviations scaled by the standard deviation in the data.
Cyclic heuristic
CH, an optimizer technique that makes incremental changes in one DV at a time, taking each in turn. If the OF is improved, that new DV value is retained and the next increment for that DV will be larger. Otherwise, the old DV value is retained and the next increment for that DV will be both smaller and in the opposite direction.
Data
As a singular data point (set of conditions) or as the plural set of all data points.
Data-based validation
The comparison of model to data to judge if the model properly captures the underlying phenomena.
Data model
The calculation procedure used to take experimental measurements to generate data for the regression modeling, the method to calculate
y
and
x
experimental from sensor measurements.
Data reconciliation
A method for correcting a set of measurements in light of a model that should make the measurements redundant.
Decision variables
DVs are what you adjust to minimize the objective function (OF). In regression, the DVs are the model coefficients that are adjusted to make the model best fit the data.
Dependent variable
The output variable, output from model, result, impact, prediction, outcome, modeled value.
Design
Devising a procedure to achieve desired results.
Design of experiments
DoE, the procedure/protocol/sequence/methodology of executing experiments to generate data.
Deterministic
The model returns one value representing an average, or parameter value, or probability.
Deviation
A variable that indicates deviation from a reference point (as opposed to absolute value).
Direct search
An optimization procedure that uses heuristic rules based on function evaluations, not derivatives. Examples include Hooke–Jeeves, leapfrogging, and particle swarm.
Discrete
A variable that has discrete (as opposed to continuum) values – integers, the last decimal value.
Discrimination
Using validation to select one model over another.
Distribution
The description of the diversity of values that might result from natural processes (particle size), simulations (stochastic process, Monte Carlo simulation), or an event probability.
DoE
Design of experiments.
DV
Decision variable.
Dynamic
The process states are changing in time in response to an input, often termed transient.
EC
Equal concern – a scaling factor to balance the impact of several measures of undesirability in a single objective function. Essentially, the reciprocal of the Lagrange multiplier.
Empirical
The model has a generic mathematical functional relation (power series, neural network, wavelets, orthogonal polynomials, etc.) with coefficients chosen to best shape the functionalities to match the experimentally obtained data.
Ensemble
A model that uses several independent equations or procedures to arrive at predictions, then some sort of selection to choose the average or representative value.
Equal concern factor
The degree of violation of one desire that raises the same level of concern as a specified violation of another desire, weighting factors in a penalty that are applied as divisors as opposed to Lagrange multipliers.
Equality constraints
A constraint that relates variables in an equality relation, useful in reducing the number of DVs.
EWMA
Exponentially weighted moving average, a first-order filtered value of a variable.
EWMV
Exponentially weighted moving variance, a first-order filtered value of a variance.
Experiment
A procedure for obtaining data or results. The experiment might be physical or simulated.
Exponentially weighted moving average
EWMA, a first-order filtered value of a variable.
Exponentially weightedmoving variance
EWMV, a first-order filtered value of a variance.
Final prediction error
FPE, Ljung's take on Akaike's approach to balancing model complexity with reduction in SSD. Concepts are similar in Mallows' Cp and Akaike's information criterion.
First principles
An approach that uses a fundamental mechanistic approach to develop an elementary model. A phenomenological model, but not representing an attempt to be rigorous or complete.
First-order filter
FOF – an equation for tempering noise by averaging, an exponentially weighted moving average, the solution to a first-order differential equation, the result of an RC circuit for tempering noise on a voltage measurement.
FL
Fuzzy logic – models that use human linguistic descriptions, such as: “Its cold outside so wear a jacket.” This is not as mathematically precise as, “The temperature is 38 °F, so use a cover with an insulation
R
-value of 12,” but fully adequate to take action.
FOF
First-order filter.
FPE
Final prediction error, which is Ljung's take on Akaike's approach to balancing model complexity with reduction in SSD. Concepts are similar in Mallows' Cp and Akaike's information criterion.
Fuzzy logic
FL – models that use human linguistic descriptions, such as: “Its cold outside so wear a jacket.” This is not as mathematically precise as, “The temperature is 38 °F, so use a cover with an insulation
R
-Vvalue of 12,” but fully adequate to take action.
Gaussian distribution
The bell-shaped or normal distribution.
Generalized reduced gradient
GRG, a gradient-based optimization approach that reduces the number of DVs when a constraint is encountered by replacing the inequality with an equality constraint as long as the constraint is active.
Global optimum
The extreme lowest minima or highest maxima of a function.
Gradient
The vector of first derivatives of the OF w.r.t. each DV, the direction of steepest descent. Gradient-based optimizers include Cauchy's sequential line search, Newton–Raphson, Levenberg–Marquardt, and GRG.
Gradient-based optimization
Optimization approaches that use the gradient, the direction of steepest descent. Gradient-based optimizers include Cauchy's sequential line search, Newton–Raphson, Levenberg–Marquardt, and GRG.
GRG
Generalized reduced gradient, a gradient-based optimization approach that reduces the number of DVs when a constraint is encountered by replacing the inequality with an equality constraint as long as the constraint is active.
Hard constraint
May not be violated, because it leads to an operation that is impossible to execute (square root of a negative, divide by zero) or violates some physical law (the sum of all compositions must be less than or equal to 100%).
Histogram
A bar graph representing the likelihood (probability, frequency) of obtaining values within numerical intervals.
HJ
Hooke–Jeeves, an optimization procedure that searches a minimal pattern of local OF values to determine where to incrementally move the pattern, moves the pattern center, and repeats.
Homoscedasity
Constant variance throughout a range.
Hooke–Jeeves
HJ, an optimization procedure that searches a minimal pattern of local OF values to determine where to incrementally move the pattern, moves the pattern center, and repeats.
Imputation
The act of creating missing data values from correlations to available data.
Incremental regression
The model coefficients are incrementally adjusted at each sampling, so that the model evolves with the changing process that generates the data.
Incremental steepest descent
ISD, an optimization technique that makes incremental steps in the steepest descent direction, re-evaluating the direction of steepest descent after each incremental TS move.
Independent variable
Input variable, input to the model, cause, influence.
Inequality constraints
A constraint that related variables in an inequality relation, a less than or greater than relation. Could be treated as either a hard or soft constraint.
Input variable
An influence, cause, source, input, forcing function, independent value to the model.
Inverse
The model is used “backward” to answer the question, “What inputs are required to provide a desired output?”
ISD
Incremental steepest descent, an optimization technique that makes incremental steps in the steepest descent direction, re-evaluating the direction of steepest descent after each incremental TS move.
Lag
In statistics it refers to the time interval between time-discretized data. A lag of 5 means a delay of five samples. In dynamic modeling it refers to a first-order, asymptotic dynamic response to a final value. Both definitions are used in this book.
Lagrange multipliers
Weighting factors in a penalty that are applied as multipliers as opposed to equal concern factors.
Leapfrogging
LF, an optimization technique that scatters players throughout DV space and then leaps the worst over the best, to converge on the optimum.
Levenberg–Marquardt
LM, an optimization technique that blends incremental steepest descent and Newton–Raphson.
LHS
Left-hand side, the terms on the left-hand side of an equation (either equality or inequality).
Likelihood
A measure of the probability that a model could have generated the experimental data.
Linear
The relation between two variables is a straight line.
Linearizing transforms
Mathematical operations that linearize an OF, providing the convenience of using linear regression solution methods. Be cautious about the weighting distortion that results.
LF
Leapfrogging, an optimization technique that scatters players throughout DV space, then leaps the worst over the best, to converge on the optimum.
LM
Levenberg–Marquardt, an optimization technique that blends incremental steepest descent and Newton–Raphson.
Local optimum
One of several minima or maxima of a function, but not the extreme.
Logic-based validation
Comparison of model functionality to rational, logical expectations.
MA
Moving average, the average of the chronologically most recent
N
data values in a time series.
Maximum
Highest or largest value.
Maximum error
An estimate of the uncertainty in a calculated value, based on all sources of uncertainty providing their maximum perturbation and influencing the outcome in the same direction.
Mean
The expected average in a list of data.
Median
The middle value in a list of data. To find it, repeatedly exclude the high and low values. If an odd-numbered list, the one that remains is the median. If an even-numbered list, average the two that remain.
Minimum
Lowest or smallest value.
Model
A mathematical representation of the human's understanding of Nature's response to the influence.
Model architecture
The mathematical structure, the functional relations within a model.
Moving average
MA, the average of the chronologically most recent
N
data values in a time series.
Nature
A respectful anthropomorphic representation of the mystery of the processes that generate data and which tortures us with complexity and variation.
Nelder–Mead
NM, a direct search optimization technique that uses the Simplex geometry and moves the worst local trial solution through the centroid of the others.
Neural network
NN – a modeling approach that was intended to mimic how brain neurons “calculate.”
Newton–Raphson
NR, an optimization method that uses the local OF derivatives and a second-order series approximation of the OF to predict the optimum, jumps there and repeats.
NID(μ, σ)
Normally (Gaussian) and independently distributed with a mean of
μ
and a standard deviation of
σ
.
NM
Nelder–Mead, a direct search optimization technique that uses the Simplex geometry and moves the worst local trial solution through the centroid of the others.
NN
Neural network – a modeling approach that was intended to mimic how brain neurons “calculate.”
Noise
Random, independent perturbations to a conceptually deterministic value.
Nominal
Latin origin for name, a class/category/string variable.
Nonlinear
Means not linear. This could be any not linear relation.
Nonparametric test
The category of statistical tests that do not presume a normal model of the residuals.
Normal equations
The set of linear equations that arise in linear regression when the analytical derivatives of the OF w.r.t. each coefficient are set to zero.
Normal SSD
The sum of squared differences between the model and data in both the
x
and
y
axes, normal to the model, perpendicular to the model, often called total SSD.
NR
