Statistics for Censored Environmental Data Using Minitab and R E-Book

Dennis R. Helsel

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 560

Veröffentlichungsjahr: 2011

Beschreibung

Praise for the First Edition " . . . an excellent addition to an upper-level undergraduate course on environmental statistics, and . . . a 'must-have' desk reference for environmental practitioners dealing with censored datasets." --Vadose Zone Journal Statistics for Censored Environmental Data Using Minitab¯® and R, Second Edition introduces and explains methods for analyzing and interpreting censored data in the environmental sciences. Adapting survival analysis techniques from other fields, the book translates well-established methods from other disciplines into new solutions for environmental studies. This new edition applies methods of survival analysis, including methods for interval-censored data to the interpretation of low-level contaminants in environmental sciences and occupational health. Now incorporating the freely available R software as well as Minitab¯® into the discussed analyses, the book features newly developed and updated material including: * A new chapter on multivariate methods for censored data * Use of interval-censored methods for treating true nondetects as lower than and separate from values between the detection and quantitation limits ("remarked data") * A section on summing data with nondetects * A newly written introduction that discusses invasive data, showing why substitution methods fail * Expanded coverage of graphical methods for censored data The author writes in a style that focuses on applications rather than derivations, with chapters organized by key objectives such as computing intervals, comparing groups, and correlation. Examples accompany each procedure, utilizing real-world data that can be analyzed using the Minitab¯® and R software macros available on the book's related website, and extensive references direct readers to authoritative literature from the environmental sciences. Statistics for Censored Environmental Data Using Minitab¯® and R, Second Edition is an excellent book for courses on environmental statistics at the upper-undergraduate and graduate levels. The book also serves as a valuable reference for??environmental professionals, biologists, and ecologists who focus on the water sciences, air quality, and soil science.

Leseprobe

Contents

Cover

Wiley Series in Statistics in Practice

Title Page

Copyright

Preface

Acknowledgments

Introduction to the First Edition: An Accident Waiting To Happen

Introduction to the Second Edition: Invasive Data

Chapter 1: Things People Do with Censored Data that Are Just Wrong

1.1 Why Not Substitute—Missing the Signals that Are Present in the Data

1.2 Why Not Substitute?—Finding Signals That Are Not There

1.3 So Why Not Substitute?

1.4 Other Common Misuses of Censored Data

Chapter 2: Three Approaches for Censored Data

2.1 Approach 1: Nonparametric Methods After Censoring at the Highest Reporting Limit

2.2 Approach 2: Maximum Likelihood Estimation

2.3 Approach 3: Nonparametric Survival Analysis Methods

2.4 Application of Survival Analysis Methods to Environmental Data

2.5 Parallels To Uncensored Methods

Chapter 3: Reporting Limits

3.1 Limits When The Standard Deviation Is Considered Constant

3.2 Insider Censoring–Biasing Interpretations

3.3 Reporting the Machine Readings of All Measurements

3.4 Limits When the Standard Deviation Changes with Concentration

3.5 For Further Study

Chapter 4: Reporting, Storing, and Using Censored Data

4.1 Reporting and Storing Censored Data

4.2 Using Interval-Censored Data

Exercises

Chapter 5: Plotting Censored Data

5.1 Boxplots

5.2 Histograms

5.3 Empirical Distribution Function

5.4 Survival Function Plots

5.5 Probability Plot

5.6 X–Y Scatterplots

Exercises

Chapter 6: Computing Summary Statistics and Totals

6.1 Nonparametric Methods After Censoring at the Highest Reporting Limit

6.2 Maximum Likelihood Estimation

6.3 The Nonparametric Kaplan–Meier and Turnbull Methods

6.4 Ros: A “Robust” Imputation Method

6.5 Methods in Excel

6.6 Handling Data With High Reporting Limits

6.7 A Review of Comparison Studies

6.8 Summing Data With Censored Observations

Exercises

Chapter 7: Computing Interval Estimates

7.1 Parametric Intervals

7.2 Nonparametric Intervals

7.3 Intervals for Censored Data by Substitution

7.4 Intervals for Censored Data by Maximum Likelihood

7.5 Intervals for the Lognormal Distribution

7.6 Intervals Using “Robust” Parametric Methods

7.7 Nonparametric Intervals for Censored Data

7.8 Bootstrapped Intervals

7.9 For Further Study

Exercises

Chapter 8: What Can be Done When All Data Are Below the Reporting Limit?

8.1 Point Estimates

8.2 Probability of Exceeding the Reporting Limit

8.3 Exceedance Probability for a Standard Higher Than the Reporting Limit

8.4 Hypothesis Tests Between Groups

8.5 Summary

Exercises

Chapter 9: Comparing Two Groups

9.1 Why not Use Substitution?

9.2 Simple Nonparametric Methods After Censoring at the Highest Reporting Limit

9.3 Maximum Likelihood Estimation

9.4 Nonparametric Methods

9.5 Value of the Information in Censored Observations

9.6 Interval-Censored Score Tests: Testing Data That Include (DL to RL) Values

9.7 Paired Observations

9.8 Summary of Two-Sample Tests for Censored Data

Exercises

Chapter 10: Comparing Three or More Groups

10.1 Substitution Does Not Work—Invasive Data

10.2 Nonparametric Methods After Censoring at the Highest Reporting Limit

10.3 Maximum Likelihood Estimation

10.4 Nonparametric Method—The Generalized Wilcoxon Test

10.5 Summary

Exercises

Chapter 11: Correlation

11.1 Types of Correlation Coefficients

11.2 Nonparametric Methods After Censoring at the Highest Reporting Limit

11.3 Maximum Likelihood Correlation Coefficient

11.4 Nonparametric Correlation Coefficient—Kendall's Tau

11.5 Interval-Censored Score Tests: Testing Correlation with (DL to RL) Values

11.6 Summary: A Comparison Among Methods

11.7 For Further Study

Exercises

Chapter 12: Regression and Trends

12.1 Why not Substitute?

12.2 Nonparametric Methods After Censoring at the Highest Reporting Limit

12.3 Maximum Likelihood Estimation

12.4 Akritas–Theil–Sen Nonparametric Regression

12.5 Additional Methods for Censored Regression

Exercises

Chapter 13: Multivariate Methods for Censored Data

13.1 A Brief Overview of Multivariate Procedures

13.2 Nonparametric Methods After Censoring at the Highest Reporting Limit

13.3 Multivariate Methods for Data With Multiple Reporting Limits

13.4 Summary of Multivariate Methods for Censored Data

Chapter 14: The NADA for R Software

14.1 A Brief Overview of R and the NADA Software

14.2 Summary of The Commands Available In NADA

Appendix: Datasets

References

Index

Statistics in Practice

Wiley Series in Statistics in Practice

Advisory Editor, Marian Scott, University of Glasgow, Scotland, UK

Founding Editor, Vic Barnett, Nottingham Trent University, UK

Statistics in Practice is an important international series of texts which provide detailed coverage of statistical concepts, methods, and worked case studies in specific fields of investigation and study.

With sound motivation and many worked practical examples, the books show in down-to-earth terms how to select and use an appropriate range of statistical techniques in a particular practical field within each title's special topic area.

The books provide statistical support for professionals and research workers across a range of employment fields and research environments. Subject areas covered include medicine and pharmaceutics; industry, finance, and commerce; public services; the earth and environmental sciences, and so on.

The books also provide support to students studying statistical courses applied to the above areas. The demand for graduates to be equipped for the work environment has led to such courses becoming increasingly prevalent at universities and colleges.

It is our aim to present judiciously chosen and well-written workbooks to meet everyday practical needs. Feedback of views from readers will be most valuable to monitor the success of this aim.

A complete list of titles in this series appears at the end of the volume.

First edition published under the title Nondetects And Data Analysis

Copyright © 2012 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Helsel, Dennis R.

Statistics for censored environmental data using Minitab® and R / Dennis R. Helsel. – 2nd ed.

p. cm. – (Wiley series in statistics in practice)

Rev. ed. of: Nondetects and data analysis / Dennis R. Helsel. 2005.

Includes bibliographical references and index.

ISBN 978-0-470-47988-9 (cloth)

1. Environmental sciences–Statistical methods. 2. Pollution–Measurement–Statistical methods. 3. Minitab. 4. R (Computer program language) I. Helsel, Dennis R. Nondetects and data analysis.

II. Title.

GE45.S73H45 2012

363.730285′53–dc23

2011028945

Preface

This book introduces methods for censored data, some simple and some more complex, to potential users who until now were not aware of their existence, or perhaps not aware of their utility. These methods are directly applicable to air quality, water quality, soils, and contaminants in biota, among other types of data. Most of the methods come from the field of survival analysis, where the primary variable being investigated is length of time. Here they are instead applied to environmental measures such as concentration. The first edition (under the name Nondetects And Data Analysis) has influenced the methods used by scientists in several disciplines, as reflected in guidance documents and usage in journals. It is my hope that the second edition of this book will continue this progress, broadening the readership to statisticians who are just becoming familiar with environmental applications for these methods.

Within each chapter, examples have been provided in sufficient detail so that readers may apply these methods to their own work. Readily available software was used so that methods would be easily accessible. Examples throughout the book were computed using Minitab® (version 16), one of several software packages providing routines for survival analysis, and using the freely available R statistical software system.

The web site linked with this book: http://practicalstats.com/nada contains material for the reader that augments this textbook. Located on the web site are

1. answers to exercises computed using Minitab and R,

2.Minitab macros and R scripts,

3. a link to the NADA for R package,

4. data sets used in this book, and

5. as necessary, an errata sheet listing corrections to the text.

Comments and feedback on both the web site and the book may be emailed to me at [email protected]

I sincerely hope that you find this book helpful in your work.

Dennis HelselApril 2011

Acknowledgments

My sincere appreciation goes to Dr. Ed Gilroy and to a host of students in our Nondetects And Data Analysis short courses who have reviewed portions of notes and overheads, making many suggestions and improvements.

To A.T. Miesch, who led the way decades ago.

To my wife Cindy, for her patience and support during what seems to her a never-ending process.

Yesterday upon the stair I saw a man who wasn't there He wasn't there again today Oh how I wish he'd go away.

Hughes Mearns (1875–1965)

Introduction to the First Edition: An Accident Waiting To Happen

On January 28, 1986 the space shuttle Challenger exploded 73 seconds after liftoff from Kennedy Space Center, killing all seven astronauts on board and severely wounding the US space program. In addition to career astronauts, on board was America's Teacher In Space, Christa McAuliffe, who was to tape and broadcast lessons designed to interest the next generation of children in America's space program. Her participation ensured that much of the country, including its school children, was watching.

What caused the accident? Would it happen again on a subsequent launch? Four months later the Presidential Commission investigating the accident issued its final report (Rogers Commission, 1986). It pinpointed the cause as a failure of O-rings to flex and seal in the 30°F temperatures at launch time. Rocket fuel exploded after escaping through an opening left by a failed O-ring. An on-camera experiment during the hearings by physicist Richard Feynman illustrated how a section of O-ring, when placed in a glass of ice water, failed to recover from being squeezed by pliers. The experiment's refreshing clarity contrasted sharply with days of inconclusive testimony by officials who debated what might have taken place.

The most disturbing part of the Commission's report was that the O-ring failure had been foreseen by engineers of the booster rockets' manufacturer, who were unable to convince managers to delay the launch. Rocket tests had previously shown evidence of thermal stress in O-rings when temperatures were 65°F and colder. No data were available for the extremely low temperatures predicted for launch time. Faxes sent to NASA on January 27th, the night before launch, presented a graph of damage incidents to one or more rocket O-rings as a function of temperature (Figure i1). This evidence given in the figure seemed inconclusive to managers—there were few data and no apparent pattern.

Figure i1 Plot of flights with incidents of O-ring thermal distress—“censored observations” deleted. (Figure 6 from Rogers Commission, 1986, p. 146.)

The Rogers Commission noted in its report that the above graph had one major flaw—flights where damage had not been detected were deleted. The Commission produced a modified graph, their assessment of what should have been (but was not) sent to NASA managers. Their graph added back in the censored values (Figure, i2). By including all recorded data, the Commission proved that the pattern was a bit more striking.

Figure i2 Plot of flights with and without incidents of O-ring thermal distress— “censored observations” included. (Figure 7 from Rogers Commission, 1986, p. 146.)

What type of graph could the engineers have used to best illustrate the risk they believed was present? The vast store of information in censored observations is contained in the proportions at which they occur. A simple bar chart could have focused on the proportion of O-rings exhibiting damage. For a possible total of three damage incidents in each rocket, a graph of the proportion of failure incidents by ranges of 5° in temperature is shown in Figure i3. The increase in the proportion of damaged O-rings with lower temperatures is clear.

Figure i3 O-ring thermal distress data, re-expressed as proportions.

In Figure i1, the information content of data below a (damage) detection threshold was discounted, and the data ignored. Not recognizing and recovering this information was a serious error by engineers. Today the same types of errors are being made by numerous environmental scientists. Deleting censored observations, concentrations below a measurement threshold, obscures the information in graphs and numerical summaries. Statements such as the one below from the ASTM committee on intralaboratory quality control are all too common:

Results reported as “less than” or “below the criterion of detection” are virtually useless for either estimating outfall and tributary loadings or concentrations for example.

(ASTM D4210, 1983)

A second, equally serious error occurred prior to the Challenger launch when managers assumed that they possessed more information on launch safety than was contained in their data. They decided to launch without knowing the consequences of very low temperatures. According to Richard Feynman, their attitude had become “a kind of Russian roulette . . . . We can lower our standards a little bit because we got away with it the last time” (Rogers Commission, 1986, p. 148). A similar error is now frequently made by environmental programs that fabricate numbers, such as one-half the detection limit, to replace censored observations. Substituting a constant value is even mandated by some Federal agencies—it seemed to work the last time they used it. Its primary error lies in assuming that the scientist/regulator knows more information than what is actually contained in their data. This can easily result in the wrong conclusion, such as declaring that an area is “clean” when it really is not. For the Challenger accident, the consequences were a tragic one-time loss of life. For environmental sciences, the consequences are likely to be more chronic and continuous. The health effects of many environmental contaminants occur in the same ranges as current detection limits. Assuming that measurements are at one value when they could be at another is not a safe practice, and as we shall see, totally unnecessary. Fabricating numbers for concentrations could also lead to unnecessary expenditures for cleanup, declaring an area is worse than it actually is. With the large (but limited) amounts of funding now spent on environmental measurements and evaluations, it is incumbent on scientists to use the best available methodologies. In regards to deleting censored observations, or fabricating numbers for them, there are better ways.

When interpreting data that include values below a detection threshold, keep in mind three principles:

1. Never delete censored observations.

2. Capture the information in the proportions.

3. Never assume that you know more than you do.

This book is about what else is possible.

Introduction to the Second Edition: Invasive Data

In his satire Hitchhiker's Guide To The Galaxy, Douglas Adams wrote of his characters' search through space to find the answer to “the question of Life, The Universe and Everything.” In what is undoubtedly a commentary on the inability of science to answer such questions, the computer built to process it determines that the answer is 42. Environmental scientists often provide an equally arbitrary answer to a different question—what to do with censored “nondetect” data?

The most common procedure within environmental chemistry to deal with censored observations continues to be substitution of some fraction of the detection limit. This method is better labeled as “fabrication”, as it substitutes a specific value for concentration data even though a specific value is unknown (Helsel, 2006). Within the field of water chemistry, one-half is the most commonly- used fraction, so that 0.5 is used as if it had been measured whenever a <1 (detection limit of 1) occurs. For air chemistry, one over the square root of two, or about 0.7 times the detection limit, is commonly used. Douglas Adams might have chosen 0.42.

In addition to the environmental sciences where I work, the issue of correctly handling nondetect data has been of great interest in astronomy (Feigelson and Nelson, 1985), in risk assessment (Tressou, 2006), and in occupational health (Succop et al., 2004; Hewett and Ganser, 2007; Finkelstein, 2008; Krishnamoorthy et al., 2009; Flynn, 2010). We all deal with information overload, barely having time to read the relevant literature of our own discipline. It is next to impossible to keep up with work in other disciplines, even when they encounter the same issues as we do. Handling nondetect data is one example.

There is an incredibly strong pull for doing something that is simple and cheap, not to mention familiar. In 1990, I stated that techniques of survival analysis, statistical methods for handling right-censored data in medical and industrial applications, could be turned around and applied to censoring on the low end (Helsel, 1990). The 1990 article clearly states that substitution of values such as one-half the detection limit is generally a bad idea. Because I mention substitution in it, the article has since been referenced a myriad of times to justify using substitution! It makes me wonder whether they read the article at all. As I said, there is an incredibly strong pull for doing something simple and cheap.

The problem with substitution is what I have come to call “invasive data.” Substitution is not neutral, but invasive—a pattern is being added to the data that may be quite different than the pattern of the data itself. It can take over and choke out the native pattern. Consider the data of Figure i4, a straight-line relationship between two variables, Concentration (y) versus distance (x) downstream. The slope of the relationship is significant, with a strong positive correlation between the variables. Concentrations are increasing (perhaps with increasing urbanization) downstream. What happens when the data are reported using two detection limits of 1 and 3, and one-half the limit is substituted for the censored observations? The result (Figure i5) includes horizontal lines of substituted values, changing the slope and dramatically decreasing the correlation coefficient between the variables. Looking only at these numbers, the data analyst obtains the (wrong) impression that there is no correlation, no increase in concentration.

Figure i4 Original data prior to censoring. True correlation equals 0.81.

Figure i5 Data from Figure i4 after censoring at detection limits of 1 and 3 ppb and substituting ½ DL (shown as open circles). These invasive data form flat lines at one-half the detection limits, lowering the correlation to 0.55.

There are many published articles where substitution was used prior to computing a correlation coefficient. It is cheap and simple. Tajimi et al. (2005), as just one example, calculated correlation coefficients between dioxin concentrations and possible causative factors after substituting one-half the detection limit for all censored observations. A low correlation coefficient was considered evidence that the factor was not the likely cause of the contamination. They found no significant correlations. Was this because there were none, or was it the result of their data substitutions? When adding an invasive flat line to the original data, the original relationship may easily be missed. Thankfully, there are better ways.

Finkelstein (2008) re-examined a study that compared asbestos in the lungs of automobile brake mechanics to a control group. The original study decided that no difference in tremolite asbestos was evident between the two groups, based on visually comparing group medians. The study was faced with many censored observations in the two groups, and was not sure how to best incorporate them into a statistical test. Finkelstein used censored maximum likelihood (see Chapter) to test for differences, finding that concentrations of tremolite asbestos were indeed elevated in the mechanics' lungs. The message of his paper is clear—ignoring methods that incorporate censored data leads to wrong decisions both economically and for human or ecosystem health. In the introduction to the first edition, I used the flawed decision to launch the Challenger shuttle as the example. Finkelstein's example of missing the elevated levels of asbestos in the lungs of brake mechanics is equally compelling. Simple, cheap, easy but ineffective methods today can often lead to expensive, heart- breaking, difficult consequences later.

Here are at three recommendations to consider while reading this book:

1. In general, do not use substitution. Journals should consider it a flawed method compared to the others that are available, and reject papers that use it. The lone exception might be when only estimating the mean for data with one censoring threshold, but not for other situations or procedures. Substitution is NOT imputation, which implies using a model such as the relationship with a correlated variable to impute (estimate) values. Substitution is fabrication. It may be simple and cheap, but its results can be noxious.

2. We should all become more familiar with the literature on censored data from survival/reliability analysis. There should be more widespread training in survival/reliability methods within university programs in both the environmental and public health disciplines.

3. Commercial software should more easily incorporate left- and interval-censored data into its survival/reliability routines. For example, plots and hypothesis tests of whether censored data fit a normal and other distributions, as requested by Hewett and Ganser (2007), already exist in many commercial software packages. But they are sometimes coded to handle only right-censored data. They usually do not return p-values for the test. They often incorrectly delete the highest point prior to plotting (see Chapter). These and similar considerations will not change until software users in both environmental sciences and public health loudly request that they be changed.

Chapter 1

Things People Do with Censored Data that Are Just Wrong

Censored observations are low-level concentrations of organic or inorganic chemicals with values known only to be somewhere between zero and the laboratory's detection/reporting limits. The chemical signal on the measuring instrument is small in relation to the process noise. Measurements are considered too imprecise to report as a single number, so the value is commonly reported as being less than an analytical threshold, for example, “<1.” Long considered second-class data, censored observations complicate the familiar computations of descriptive statistics, of testing differences among groups, and of correlation coefficients and regression equations.

Statisticians use the term “censored data” for observations that are not quantified, but are known only to exceed or to be less than a threshold value. Values known only to be below a threshold (less-thans) are left-censored data. Values known only to exceed a threshold (greater-thans) are right-censored data. Values known only to be within an interval (between 2 and 5) are interval-censored data. Techniques for computing statistics for censored data have long been employed in medical and industrial studies, where the length of time is measured until an event occurs, such as the recurrence of a disease or failure of a manufactured part. For some observations the event may not have occurred by the time the experiment ends. For these, the time is known only to be greater than the experiment's length, a right-censored “greater-than” value. Methods for incorporating censored data when computing descriptive statistics, testing hypotheses, and performing correlation and regression are all commonly used in medical and industrial statistics, without substituting arbitrary values. These methods go by the names of “survival analysis” (Klein and Moeschberger, 2003) and “reliability analysis” (Meeker and Escobar, 1998). There is no reason why these same methods should also not be used in the environmental sciences, but until recently their use has been relatively rare. Environmental scientists have not often been trained in survival analysis methods.

The worst practice when dealing with censored observations is to exclude or delete them. This produces a strong bias in all subsequent measures of location or hypothesis tests. After excluding the 80% of observations that are left-censored nondetects, for example, the mean of the top 20% of concentrations is reported. This provides almost no insight into the original data. Excluding censored observations removes the primary information contained in them—the proportion of data in each group that lies below the reporting limit(s). And while better than deleting censored observations, fabricating artificial values as if these had been measured provides its own inaccuracies. Fabrication (substitution) adds an invasive signal to the data that was not previously there, potentially obscuring the information present in the measured observations.