Understanding Uncertainty, Revised Edition E-Book

Dennis V. Lindley

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Probability and Statistics
• Sprache: Englisch

Seitenzahl: 730

Veröffentlichungsjahr: 2013

Beschreibung

Praise for the First Edition "...a reference for everyone who is interested in knowing and handling uncertainty." --Journal of Applied Statistics The critically acclaimed First Edition of Understanding Uncertainty provided a study of uncertainty addressed to scholars in all fields, showing that uncertainty could be measured by probability, and that probability obeyed three basic rules that enabled uncertainty to be handled sensibly in everyday life. These ideas were extended to embrace the scientific method and to show how decisions, containing an uncertain element, could be rationally made. Featuring new material, the Revised Edition remains the go-to guide for uncertainty and decision making, providing further applications at an accessible level including: * A critical study of transitivity, a basic concept in probability * A discussion of how the failure of the financial sector to use the proper approach to uncertainty may have contributed to the recent recession * A consideration of betting, showing that a bookmaker's odds are not expressions of probability * Applications of the book's thesis to statistics * A demonstration that some techniques currently popular in statistics, like significance tests, may be unsound, even seriously misleading, because they violate the rules of probability Understanding Uncertainty, Revised Edition is ideal for students studying probability or statistics and for anyone interested in one of the most fascinating and vibrant fields of study in contemporary science and mathematics.

Leseprobe

Contents

Cover

Wiley Series in Probability and Statistics

Title Page

Copyright

Preface

Prologue

Chapter 1: Uncertainty

1.1 Introduction

1.2 Examples

1.3 Suppression of Uncertainty

1.4 The Removal of Uncertainty

1.5 The Uses of Uncertainty

1.6 The Calculus of Uncertainty

1.7 Beliefs

1.8 Decision Analysis

Chapter 2: Stylistic Questions

2.1 Reason

2.2 Unreason

2.3 Facts

2.4 Emotion

2.5 Normative and Descriptive Approaches

2.6 Simplicity

2.7 Mathematics

2.8 Writing

2.9 Mathematics Tutorial

Chapter 3: Probability

3.1 Measurement

3.2 Randomness

3.3 A Standard for Probability

3.4 Probability

3.5 Coherence

3.6 Belief

3.7 Complementary Event

3.8 Odds

3.9 Knowledge Base

3.10 Examples

3.11 Retrospect

Chapter 4: Two Events

4.1 Two Events

4.2 Conditional Probability

4.3 Independence

4.4 Association

4.5 Examples

4.6 Supposition and Fact

4.7 Seeing and Doing

Chapter 5: The Rules of Probability

5.1 Combinations of Events

5.2 Addition Rule

5.3 Multiplication Rule

5.4 The Basic Rules

5.5 Examples

5.6 Extension of the Conversation

5.7 Dutch Books

5.8 Scoring Rules

5.9 Logic Again

5.10 Decision Analysis

5.11 The Prisoners' Dilemma

5.12 The Calculus and Reality

5.13 Closure

Chapter 6: Bayes Rule

6.1 Transposed Conditionals

6.2 Learning

6.3 Bayes Rule

6.4 Medical Diagnosis

6.5 Odds Form of Bayes Rule

6.6 Forensic Evidence

6.7 Likelihood Ratio

6.8 Cromwell's Rule

6.9 A Tale of two Urns

6.10 Ravens

6.11 Diagnosis and Related Matters

6.12 Information

Chapter 7: Measuring Uncertainty

7.1 Classical Form

7.2 Frequency Data

7.3 Exchangeability

7.4 Bernoulli Series

7.5 De Finetti's Result

7.6 Large Numbers

7.7 Belief and Frequency

7.8 Chance

Chapter 8: Three Events

8.1 The Rules of Probability

8.2 Simpson's Paradox

8.3 Source of the Paradox

8.4 Experimentation

8.5 Randomization

8.6 Exchangeability

8.7 Spurious Association

8.8 Independence

8.9 Conclusions

Chapter 9: Variation

9.1 Variation and Uncertainty

9.2 Binomial Distribution

9.3 Expectation

9.4 Poisson Distribution

9.5 Spread

9.6 Variability as an Experimental Tool

9.7 Probability and Chance

9.8 Pictorial Representation

9.9 Probability Densities

9.10 The Normal Distribution

9.11 Variation as a Natural Phenomenon

9.12 Ellsberg's Paradox

Chapter 10: Decision Analysis

10.1 Beliefs and Actions

10.2 Comparison of Consequences

10.3 Medical Example

10.4 Maximization of Expected Utility

10.5 More on Utility

10.6 Some Complications

10.7 Reason and Emotion

10.8 Numeracy

10.9 Expected Utility

10.10 Decision Trees

10.11 The Art and Science of Decision Analysis

10.12 Further Complications

10.13 Combination of Features

10.14 Legal Applications

Chapter 11: Science

11.1 Scientific Method

11.2 Science and Education

11.3 Data Uncertainty

11.4 Theories

11.5 Uncertainty of a Theory

11.6 The Bayesian Development

11.7 Modification of Theories

11.8 Models

11.9 Hypothesis Testing

11.10 Significance Tests

11.11 Repetition

11.12 Summary

Chapter 12: Examples

12.1 Introduction

12.2 CARDS

12.3 The Three Doors

12.4 The Problem of Two Daughters

12.5 Two More Daughters and Cardano

12.6 The two Envelopes

12.7 Y2K

12.8 UFOs

12.9 Conglomerability

12.10 Efron's Dice

Chapter 13: Probability Assessment

13.1 Nonrepeatable Events

13.2 Two Events

13.3 Coherence

13.4 Probabilistic Reasoning

13.5 Trickle Down

13.6 Summary

Chapter 14: Statistics

14.1 Bayesian Statistics

14.2 A Bayesian Example

14.3 Frequency Statistics

14.4 Significance Tests

14.5 Betting

14.6 Finance

Epilogue

Subject Index

Index of Examples

Index of Notations

Wiley Series in Probability and Statistics

Established by WALTER A. SHEWHART and SAMUEL S. WILKS

Editors: David J. Balding, Noel A. C. Cressie, Garrett M. Fitzmaurice, Harvey Goldstein, Iain M. Johnstone, Geert Molenberghs, David W. Scott, Adrian F. M. Smith, Ruey S. Tsay, Sanford Weisberg

Editors Emeriti: Vic Barnett, J. Stuart Hunter, Joseph B. Kadane, Jozef L. Teugels

A complete list of the titles in this series appears at the end of this volume.

Copyright © 2014 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Lindley, D. V. (Dennis Victor), 1923-

Understanding uncertainty / Dennis V. Lindley, Minehead, Somerset, England.–Revised edition.

pages cm

Includes bibliographical references and index.

ISBN 978-1-118-65012-7 (cloth)

1. Probabilities. 2. Uncertainty–Mathematics. 3. Decision making–Mathematics. 4. Mathematical statistics. I. Title.

QA273.L534 2013

519.2–dc23

2013013977

Preface

There are some things that you, the reader of this preface, know to be true, and others that you know to be false; yet, despite this extensive knowledge that you have, there remain many things whose truth or falsity is not known to you. We say that you are uncertain about them. You are uncertain, to varying degrees, about everything in the future; much of the past is hidden from you; and there is a lot of the present about which you do not have full information. Uncertainty is everywhere and you cannot escape from it.

Truth and falsity are the subjects of logic, which has a long history going back at least to classical Greece. The object of this book is to tell you about work that has been done in the twentieth century about uncertainty. We now know that uncertainty has to obey three rules and that, once they are understood, uncertainty can be handled with almost as much confidence as ordinary logic. Our aim is to tell you about these rules, to explain to you why they are inevitable, and to help you use them in simple cases. The object is not to make you an expert in uncertainty but merely to equip you with enough skill, so that you can appreciate an uncertain situation sufficiently well to see whether another person, lawyer, politician, scientist, or journalist is talking sense, posing the right questions, and obtaining sound answers. We want you to face up to uncertainty, not hide it away under false concepts, but to understand it and, moreover, to use the recent discoveries so that you can act in the face of uncertainty more sensibly than would have been possible without the skill. This is a book for the layman, for you, for everyone, because all of us are surrounded by uncertainty.

However, there is a difficulty, the rules really need to be written in the language of mathematics and most people have a distaste for mathematics. It would have been possible for the book to have been written entirely in English, or equally in Chinese, but the result would have been cumbersome and, believe me, even harder to understand. The presentation cries out for the use of another language; that of mathematics. For mathematics is essentially another language, rather a queer one, that is unfamiliar to us. However, you do not, for this book, need to understand this language completely; only a small part of it will be required. It is somewhat like an English speaker needing about six characters from Chinese out of the many thousands that the language uses. This book uses part of the language of mathematics, and this part is explained carefully with, I hope, enough motivation for you to be convinced of its advantages. There is almost no technical use of mathematics, and what there is can be appreciated as easily as ordinary arithmetic.

There is one feature of our uncertain world that may either distress or excite you, I hope the latter, in that it does not always behave like common sense might suggest. The most striking example is Simpson's paradox, in Chapter 8, where a medical treatment appears to be bad for both the men and the women, but good for all of us. We will apply the ideas about uncertainty to the law, to science, to economics, and to politics with sometimes surprising results.

The prologue tells something about how this book came to be written. The final version owes a great deal to José Bernardo, Ian Evett, and Tony O'Hagan who read a draft and made many constructive proposals, almost all of which have been eagerly incorporated. In addition, Jay Kadane read the draft with a keen, critical eye, made valuable suggestions, and persuaded me not to ride too vigorously into fields where I had more passion than sense. The final version is much improved as a result of their kind efforts.

Preface to the Revised Edition

The principal change from the original edition is the inclusion of an additional Chapter 14, describing the impact the ideas of this book have on statistics, betting, and finance. The treatment of one problem (§§12.4 and 12.5) has been enlarged because of developments between the two editions. Efron's dice have been discussed because some readers have queried an important assumption. Minor changes have been made in the interests of clarity, several kindly suggested by Mervyn Stone. I would like to thank my daughter, Rowan, for help with the logistics, without which this new edition would not have been possible; and Stephen Quigley at Wiley for persuading me to undertake the revision.

Prologue

Almost all my professional life has been spent in academe as a statistician. In my first appointment in Cambridge, I was required to lecture for 6 hours each week during half of the year and personally to supervise some students. Admittedly, the preparation of new lecture courses took a lot of time, one occupying the whole of the 4 month summer vacation, but these duties did not constitute a reasonable workload. To fill the gap, one was expected to do exactly what I wanted to do, conduct research. As I moved to become professor and head of department, first in Aberystwyth and then at University College London, other duties, principally administrative, crowded in upon me and there was less time for research. But still it got done, because I wanted it to get done, often in conjunction with good, graduate students.

Research, at least in my case, consists of taking questions that interest one and to which you feel you might, given enough time and effort, be able to find an answer; working on them, producing an answer, which often turns out to be quite different from the form originally anticipated, and publishing the results for others to read. There are many aspects to this creative work but the one to be emphasized here is that the questions I chose to answer were selected by me. There was no superior, as there would have been in industry, posing me problems and expecting answers. There was no deadline to be met. This was freedom of thought in its true sense, requiring little more than a comfortable office, a good library, and, most important of all, time in which to think deeply about what interested you. Good answers produce rewards in promotion and more money but that is not the real motivation, which comes instead from the excitement of the chase, to explore where no one has been before, to think deeply, and to come up with something that is genuinely new. And all this free from the interference of others except those you wish to consult. That is true academic freedom that dictators hate so much.

At least during the first 20 years of my researches, I do not recall ever asking myself or being asked by others, whether what I was doing was worthwhile. Society paid me a salary that provided a comfortable living for myself and my family, giving me enough time to think and write, yielding appreciation from the few people who bothered to read my answers. I suppose if someone had asked me to justify my salary, I should have mumbled something about the training in statistics I had given to many students and the value of statistics in society. But nobody did ask and my conscience did not bother me; it was the chase that mattered. Later, however, as I began to sit on committees and come into more contact with life outside the university, I did wonder about the relevance to society of the answers I had given to questions I had chosen and, more widely, about the value of statistical ideas and methods produced by others. When I thought about this, the answers were not terribly encouraging, for admittedly the discovery of the harmful effects of smoking was mostly due to statistical analysis, and statisticians had played an important role in the breeding of new plants and animals, but I had had little to do with these activities and few had attempted to use the answers my research had provided, let alone succeeded. It had been a good life for me but had it been a worthwhile one from the viewpoint of society?

Research, especially in disciplines that use a lot of mathematics, is a young person's game and after early retirement I did little research but began to read more widely and consider problems that had not seriously entered into my comfortable research world. And I made a discovery. There were people out there, like politicians, journalists, financiers, lawyers, and managers, who were, in my opinion, making mistakes; mistakes that could have been avoided had they known the answers to the questions pondered in my ivory tower. In other words, what I had been doing was not just an exercise in pure thought, but appeared to have repercussions in the world that could affect the activities of many people and ultimately all of us. This is a phenomenon that has been observed repeatedly; namely that if people are given the freedom and opportunity to use their reasoning abilities to explore without any application in mind, what is termed pure research, they often come up with results that are applicable. Ivory towers can yield steel and concrete, produce food and shelter. This book is an attempt to explain in terms that motivated, lay persons can understand, some of the discoveries about uncertainty made in academe, and why they are of importance and value to them, so that they might use the results in their lives. In a sense, it is a justification for a life spent in academe.

The preceding paragraphs are too personal and for clarification it is necessary to say something more about scientific research. Research is carried out by individuals and often the best research is the product of one person thinking deeply on their own. For example, relativity is essentially the result of Einstein's thoughts. Yet, in a sense, the person is irrelevant, for most scientists feel that if he had not discovered relativity, then someone else would; that relativity is somehow “out there” waiting to be revealed, the revelation necessarily being made by human beings but not necessarily by that human being. This may not be true in the arts, so, for example, if Shakespeare had not written his plays, it would not follow that someone else would have produced equivalent writing. Science is a collective activity, much more so than art, and although some scientists stand out from the rest, the character of science depends to only a very small extent on individuals and what little effect they have disappears over time as their work is absorbed into the work of others. There are two lessons to be learnt from this as far as this book is concerned. First, my contribution to the results described herein is very small and is swamped by the work of others. It is as if I had merely added a brick or two to the whole building. Second, I have not thought it advisable in a book addressed to a general audience to attribute ideas to individuals. Our concern with individual scientists is often misplaced, because it is the collective wisdom that is important. The situation is made worse by the fact that the ideas are often attributed to the wrong individual. The ideas with which this work is usually associated are termed Bayesian, after Thomas Bayes, who had hardly anything to do with them. Generally, there is Stigler's law of eponymy that says that a scientific notion is never attributed to the right person; in particular, the law is not due to Stigler. Some scientists are named in the book because results are universally named after them—Bayes rule, for example, or de Finetti's theorem.

Here is a book about uncertainty, showing how it might be measured and used in your life, especially in decision making and science. It tells the story of great discoveries made in the twentieth century that merit dispersal outside the narrow community where they were developed. New ideas need new forms of exposition, so after a collection, in Chapter 1, of examples of where uncertainty impinges on our lives, Chapter 2 is concerned with certain stylistic questions, including the thorny subject of mathematics; it is only in Chapter 3 that the discoveries really begin.

Chapter 1

Uncertainty

1.1 Introduction

There are some statements that you know to be true, others that you know to be false, but with the majority of statements you do not know whether they are true or false; we say that, for you, these statements are uncertain. This book is about understanding uncertainty in this sense, about handling it, and, above all, about helping you to live comfortably with uncertainty so that you can better cope with it in your everyday life.

There are two comments that need to be made immediately. The first arises from the fact that the set of statements that you know to be true differs from my set, for you know things that I do not. Equally, things that are uncertain for you may be known to me; but there is more to it than that, for if we take a statement about which we are both uncertain, you may have more confidence that it is true than I do; we differ in our degrees of uncertainty. The upshot of these considerations is that uncertainty is a personal matter; it is not the uncertainty but your uncertainty. Admittedly, there are some situations where almost all agree on the uncertainty but these are rare and confined to special scenarios, for example, some aspects of gambling. Statements of uncertainty are personalistic; they belong to the person making them and express a relationship between that person and the real world about which a statement is being made. In particular, they are not objective in the sense that they express a property that is the same for all of us. It follows that throughout this book we will be referring to a person, conveniently called “you”, whose uncertainty is being discussed; it may sometimes be appropriate for you, the reader, to interpret it as referring to yourself but generally it applies to some unidentified person, or group of persons expressing a common opinion. You are uncertain about some aspect of the world and that uncertainty does not refer solely to you, or solely to the world, but describes a relationship between you and that world.

The second comment is to note that for any of us, for any “you”, the number of statements about which you are uncertain is vastly in excess of the number of statements for which their truth or falsity is known to you; thus all statements about the future are uncertain to some degree. Uncertainty is everywhere, so it is surprising that it is only in the twentieth century that the concept has been systematically studied and, as a result, better understood. Special types of uncertainty, like those arising in gambling, had been investigated earlier but the understanding of the broad notion, applicable to everyday life, is essentially a modern phenomenon. Because uncertainty is everywhere and affects everyone, a proper appreciation of it is vital for all persons, so this book is addressed to everyone who is prepared to listen to a reasoned argument about an ubiquitous concept. This book is for you, whoever you are. We begin with a collection of examples of uncertainty designed to demonstrate how varied, important, and numerous are statements where you genuinely do not know the truth.

1.2 Examples

Example 1. It will Rain Tomorrow

For all of us who live in climates with changeable weather, this statement is uncertain. It has become almost a classic example of uncertainty because weather is of interest, even importance, to many of us; because meteorologists have seriously studied the question of how to make forecasts like this; and because it is a statement whose uncertainty will be removed after tomorrow has passed, so that it is possible to check on the quality of the statement, a feature of which meteorologists are very conscious and which will be discussed in §5.12. Notice too, that you can change the degree of your uncertainty about rain by looking out of the window, by consulting a barometer, or by switching on the TV, and we will see in Chapter 6 just how this change may be effected.

A careful discussion here would require clarification of what is meant by “rain”; will a trace suffice, or is at least 0.01 cm in the rain gauge needed before rain can be said to have fallen? Which place is being referred to and where will the gauge be placed? What is meant by “tomorrow”—from midnight to midnight, or 24 hours from 7 A.M., as might be administratively more convenient? In this chapter we deal with illustrative examples and can be casual, but later, when more precision is introduced, these matters will assume some importance, for example, when the skills of meteorologists in predicting the weather are being assessed, or when the quality of mercy in a court of law is described. Again we return to the point in §5.12.