A Pocket Guide to Risk Mathematics E-Book

Matthew Leitch

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 235

Veröffentlichungsjahr: 2010

Beschreibung

This uniquely accessible, breakthrough book lets auditors grasp the thinking behind the mathematical approach to risk without doing the mathematics.

Risk control expert and former Big 4 auditor, Matthew Leitch, takes the reader gently but quickly through the key concepts, explaining mistakes organizations often make and how auditors can find them.

Spend a few minutes every day reading this conveniently pocket sized book and you will soon transform your understanding of this highly topical area and be in demand for interesting reviews with risk at their heart.

"I was really excited by this book - and I am not a mathematician. With my basic understanding of business statistics and business risk management I was able to follow the arguments easily and pick up the jargon of a discipline akin to my own but not my own."
—Dr Sarah Blackburn, President at the Institute of Internal Auditors - UK and Ireland

Leseprobe

Table of Contents

Title Page

Copyright Page

START HERE

Good choice!

This book

How this book works

The myth of mathematical clarity

The myths of quantification

The auditor’s mission

AUDITING SIMPLE RISK ASSESSMENTS

1 PROBABILITIES

2 PROBABILISTIC FORECASTER

3 CALIBRATION (ALSO KNOWN AS RELIABILITY)

4 RESOLUTION

5 PROPER SCORE FUNCTION

6 AUDIT POINT: JUDGING PROBABILITIES

7 PROBABILITY INTERPRETATIONS

8 DEGREE OF BELIEF

9 SITUATION (ALSO KNOWN AS AN EXPERIMENT)

10 LONG RUN RELATIVE FREQUENCY

11 DEGREE OF BELIEF ABOUT LONG RUN RELATIVE FREQUENCY

12 DEGREE OF BELIEF ABOUT AN OUTCOME

13 AUDIT POINT: MISMATCHED INTERPRETATIONS OF PROBABILITY

14 AUDIT POINT: IGNORING UNCERTAINTY ABOUT PROBABILITIES

15 AUDIT POINT: NOT USING DATA TO ILLUMINATE PROBABILITIES

16 OUTCOME SPACE (ALSO KNOWN AS SAMPLE SPACE, OR POSSIBILITY SPACE)

17 AUDIT POINT: UNSPECIFIED SITUATIONS

18 OUTCOMES REPRESENTED WITHOUT NUMBERS

19 OUTCOMES REPRESENTED WITH NUMBERS

20 RANDOM VARIABLE

21 EVENT

22 AUDIT POINT: EVENTS WITH UNSPECIFIED BOUNDARIES

23 AUDIT POINT: MISSING RANGES

24 AUDIT POINT: TOP 10 RISK REPORTING

25 PROBABILITY OF AN OUTCOME

26 PROBABILITY OF AN EVENT

27 PROBABILITY MEASURE (ALSO KNOWN AS PROBABILITY DISTRIBUTION, PROBABILITY ...

28 CONDITIONAL PROBABILITIES

29 DISCRETE RANDOM VARIABLES

30 CONTINUOUS RANDOM VARIABLES

31 MIXED RANDOM VARIABLES (ALSO KNOWN AS MIXED DISCRETE-CONTINUOUS RANDOM VARIABLES)

32 AUDIT POINT: IGNORING MIXED RANDOM VARIABLES

33 CUMULATIVE PROBABILITY DISTRIBUTION FUNCTION

34 AUDIT POINT: IGNORING IMPACT SPREAD

35 AUDIT POINT: CONFUSING MONEY AND UTILITY

36 PROBABILITY MASS FUNCTION

37 PROBABILITY DENSITY FUNCTION

38 SHARPNESS

39 RISK

40 MEAN VALUE OF A PROBABILITY DISTRIBUTION (ALSO KNOWN AS THE EXPECTED VALUE)

41 AUDIT POINT: EXCESSIVE FOCUS ON EXPECTED VALUES

42 AUDIT POINT: MISUNDERSTANDING ‘EXPECTED’

43 AUDIT POINT: AVOIDING IMPOSSIBLE PROVISIONS

44 AUDIT POINT: PROBABILITY IMPACT MATRIX NUMBERS

45 VARIANCE

46 STANDARD DEVIATION

47 SEMI-VARIANCE

48 DOWNSIDE PROBABILITY

49 LOWER PARTIAL MOMENT

50 VALUE AT RISK (VAR)

51 AUDIT POINT: PROBABILITY TIMES IMPACT

SOME TYPES OF PROBABILITY DISTRIBUTION

52 DISCRETE UNIFORM DISTRIBUTION

53 ZIPF DISTRIBUTION

54 AUDIT POINT: BENFORD’S LAW

55 NON-PARAMETRIC DISTRIBUTIONS

56 ANALYTICAL EXPRESSION

57 CLOSED FORM (ALSO KNOWN AS A CLOSED FORMULA OR EXPLICIT FORMULA)

58 CATEGORICAL DISTRIBUTION

59 BERNOULLI DISTRIBUTION

60 BINOMIAL DISTRIBUTION

61 POISSON DISTRIBUTION

62 MULTINOMIAL DISTRIBUTION

63 CONTINUOUS UNIFORM DISTRIBUTION

64 PARETO DISTRIBUTION AND POWER LAW DISTRIBUTION

65 TRIANGULAR DISTRIBUTION

66 NORMAL DISTRIBUTION (ALSO KNOWN AS THE GAUSSIAN DISTRIBUTION)

67 AUDIT POINT: NORMALITY TESTS

68 NON-PARAMETRIC CONTINUOUS DISTRIBUTIONS

69 AUDIT POINT: MULTI-MODAL DISTRIBUTIONS

70 LOGNORMAL DISTRIBUTION

71 AUDIT POINT: THIN TAILS

72 JOINT DISTRIBUTION

73 JOINT NORMAL DISTRIBUTION

74 BETA DISTRIBUTION

AUDITING THE DESIGN OF BUSINESS PREDICTION MODELS

75 PROCESS (ALSO KNOWN AS A SYSTEM)

76 POPULATION

77 MATHEMATICAL MODEL

78 AUDIT POINT: MIXING MODELS AND REGISTERS

79 PROBABILISTIC MODELS (ALSO KNOWN AS STOCHASTIC MODELS OR STATISTICAL MODELS)

80 MODEL STRUCTURE

81 AUDIT POINT: LOST ASSUMPTIONS

82 PREDICTION FORMULAE

83 SIMULATIONS

84 OPTIMIZATION

85 MODEL INPUTS

86 PREDICTION FORMULA STRUCTURE

87 NUMERICAL EQUATION SOLVING

88 PREDICTION ALGORITHM

89 PREDICTION ERRORS

90 MODEL UNCERTAINTY

91 AUDIT POINT: IGNORING MODEL UNCERTAINTY

92 MEASUREMENT UNCERTAINTY

93 AUDIT POINT: IGNORING MEASUREMENT UNCERTAINTY

94 AUDIT POINT: BEST GUESS FORECASTS

95 PREDICTION INTERVALS

96 PROPAGATING UNCERTAINTY

97 AUDIT POINT: THE FLAW OF AVERAGES

98 RANDOM

99 THEORETICALLY RANDOM

100 REAL LIFE RANDOM

101 AUDIT POINT: FOOLED BY RANDOMNESS (1)

102 AUDIT POINT: FOOLED BY RANDOMNESS (2)

103 PSEUDO RANDOM NUMBER GENERATION

104 MONTE CARLO SIMULATION

105 AUDIT POINT: IGNORING REAL OPTIONS

106 TORNADO DIAGRAM

107 AUDIT POINT: GUESSING IMPACT

108 CONDITIONAL DEPENDENCE AND INDEPENDENCE

109 CORRELATION (ALSO KNOWN AS LINEAR CORRELATION)

110 COPULAS

111 RESAMPLING

112 CAUSAL MODELLING

113 LATIN HYPERCUBE

114 REGRESSION

115 DYNAMIC MODELS

116 MOVING AVERAGE

AUDITING MODEL FITTING AND VALIDATION

117 EXHAUSTIVE, MUTUALLY EXCLUSIVE HYPOTHESES

118 PROBABILITIES APPLIED TO ALTERNATIVE HYPOTHESES

119 COMBINING EVIDENCE

120 PRIOR PROBABILITIES

121 POSTERIOR PROBABILITIES

122 BAYES’S THEOREM

123 MODEL FITTING

124 HYPERPARAMETERS

125 CONJUGATE DISTRIBUTIONS

126 BAYESIAN MODEL AVERAGING

127 AUDIT POINT: BEST VERSUS TRUE EXPLANATION

128 HYPOTHESIS TESTING

129 AUDIT POINT: HYPOTHESIS TESTING IN BUSINESS

130 MAXIMUM A POSTERIORI ESTIMATION (MAP)

131 MEAN A POSTERIORI ESTIMATION

132 MEDIAN A POSTERIORI ESTIMATION

133 MAXIMUM LIKELIHOOD ESTIMATION (MLE)

134 AUDIT POINT: BEST ESTIMATES OF PARAMETERS

135 ESTIMATORS

136 SAMPLING DISTRIBUTION

137 LEAST SQUARES FITTING

138 ROBUST ESTIMATORS

139 OVER-FITTING

140 DATA MINING

141 AUDIT POINT: SEARCHING FOR ‘SIGNIFICANCE’

142 EXPLORATORY DATA ANALYSIS

143 CONFIRMATORY DATA ANALYSIS

144 INTERPOLATION AND EXTRAPOLATION

145 AUDIT POINT: SILLY EXTRAPOLATION

146 CROSS VALIDATION

147 R (THE COEFFICIENT OF DETERMINATION)

148 AUDIT POINT: HAPPY HISTORY

149 AUDIT POINT: SPURIOUS REGRESSION RESULTS

150 INFORMATION GRAPHICS

151 AUDIT POINT: DEFINITION OF MEASUREMENTS

152 CAUSATION

AUDITING AND SAMPLES

153 SAMPLE

154 AUDIT POINT: MIXED POPULATIONS

155 ACCESSIBLE POPULATION

156 SAMPLING FRAME

157 SAMPLING METHOD

158 PROBABILITY SAMPLE (ALSO KNOWN AS A RANDOM SAMPLE)

159 EQUAL PROBABILITY SAMPLING (ALSO KNOWN AS SIMPLE RANDOM SAMPLING)

160 STRATIFIED SAMPLING

161 SYSTEMATIC SAMPLING

162 PROBABILITY PROPORTIONAL TO SIZE SAMPLING

163 CLUSTER SAMPLING

164 SEQUENTIAL SAMPLING

165 AUDIT POINT: PREJUDGING SAMPLE SIZES

166 DROPOUTS

167 AUDIT POINT: SMALL POPULATIONS

AUDITING IN THE WORLD OF HIGH FINANCE

168 EXTREME VALUES

169 STRESS TESTING

170 PORTFOLIO MODELS

171 HISTORICAL SIMULATION

172 HETEROSKEDASTICITY

173 RISKMETRICS VARIANCE MODEL

174 PARAMETRIC PORTFOLIO MODEL

175 BACK-TESTING

176 AUDIT POINT: RISK AND REWARD

177 PORTFOLIO EFFECT

178 HEDGE

179 BLACK-SCHOLES

180 THE GREEKS

181 LOSS DISTRIBUTIONS

182 AUDIT POINT: OPERATIONAL LOSS DATA

183 GENERALIZED LINEAR MODELS

CONGRATULATIONS

APPENDIX

INDEX

This edition first published 2010 by John Wiley & Sons, Ltd

Copyright © 2010 Matthew Leitch

Registered officeJohn Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should besought.

ISBN 978-0-470-71052-4

A catalogue record for this book is available from the British Library.

Set in 10/12pt Garamond Pro by Sparks—www.sparkspublishing.com

START HERE

Good choice!

This book is designed to do one job very well. If you read it as it is designed to be read, step by step, from start to finish, it will transform your understanding of risk and the mathematics involved, and will give you the confidence to tackle audits that would have been out of reach without it.

The book makes this possible by taking the pain out of the subject. Instead of bombarding you with symbols and formulae, it explains the ideas involved using mainly words and pictures so that you can understand it. In this way the book covers many more topics than a traditional mathematical textbook of the same size.

The book also focuses on the real needs of auditors. Auditors do not need to be able to do something themselves to audit it. They just need to understand the subject, know what problems to look for, and have some ideas about what to suggest. Ideally, their knowledge will be deep without necessarily being detailed. This book gives you those insights, without demanding that you spend months doing calculus.

Among other things, this book explains mathematical techniques that lead to systematic understatement of risk, so that you can suggest either changes in approach or better disclosure of the limitations of risk numbers. Some of these techniques are so simple and familiar they hardly deserve to be called mathematical, while others are more involved.

This is good for you and for your employer or clients, but it’s more than that. It’s good for society as a whole. Risk is a big issue these days, with important decisions influenced by assessments of risk ranging from gut feelings to the numbers produced by sophisticated computerized simulations. Overconfident reliance on these numbers can lead to lost lives and lost livelihoods, as the billions lost by banks during the 2007-2009 credit crunch showed.

Auditors need to play their part in stopping this from happening and, indeed, for years the audit profession, internal and external, has been positioning itself as a source of risk-focused assurance. It’s been saying that auditors are experts in risk—and so we are, in a sense—but what about the fundamental logic of risk? The truth is that most of us feel a bit uneasy about risk mathematics.

Most likely you learned some of the mathematics involved at school or university, but your knowledge is a bit rusty. If someone mentions ‘normal distribution’ you sort of know what that is, but vaguely.

And that’s a problem because some very big mistakes have been made by organizations that analysed risk badly, with no concerns raised by their auditors. I’m not talking about making a slip in the calculus, or going wrong somewhere in the algebra. The mistakes were big, conceptual mistakes that auditors could have understood and pointed out.

And things can get into an embarrassing muddle when it’s the auditors themselves doing the risk analysis. It’s not uncommon to see risk analyses done by auditors, or with their advice, that are let down by elementary mathematical errors.

This book

This book has been designed as the quickest, easiest, most convenient solution to that problem ever written.

Only the key concepts are explained, and always by building on ideas already covered. The emphasis is on having a confident, conceptual understanding and the ability to spot mistakes—a large number of which are explained and highlighted.

Obviously, reading this book requires time and effort, it’s not a comprehensive guide to risk mathematics and its mistakes, and you still won’t be able to do the mathematics yourself. However, this book is so much easier than the traditional alternatives that it changes what auditors can do, fundamentally.

While travelling to work perhaps, spend 20 minutes a day for as long as it takes to read through this book and your expertise will be transformed. Your audit colleagues will be staggered by your knowledge, and interesting reviews with risk at their core will be yours to choose.

And when a risk quant (i.e. risk mathematician) tries to blind you with technicalities you can smoothly reply with something like, ‘I see your point about the Gaussians, but you still haven’t explained your approach to model uncertainty.’ Imagine the blood draining from the face of your unsuspecting tormentor.

Not sure you want to make the investment? Give it a try and you’ll be amazed at the practical importance of the stuff you didn’t know.

How this book works

Filling your mind with the basic concepts and terminology of a discipline is the fastest way to start becoming an expert and a great way to become a good amateur. That’s why this book is made up of small, digestible chunks, each focusing on an idea and its terminology.

Many years ago I had to make myself an expert in telecommunication businesses and to get started I bought a book called Pocket Telecommunications that had alphabetically ordered entries explaining industry jargon. I read it every day on my train journeys to and from work. After about two weeks I had finished the little book and I felt different. I was different. I could read books and articles about the industry and understand what they meant. I could talk to people with years of telecommunications experience and get away with it.

Most amazing of all was that I found I was more of an expert than most people I worked with, many of whom were supposed to have years of industry experience behind them. It’s astonishing what a little homework can achieve when it is focused on the right things.

This book is designed to help you transform yourself in that way. It introduces a series of ideas and terms about the mathematics of risk.

It builds them up systematically, starting with things you will already know and teaching you what you need to understand in order to go further. All the terms have been analysed to find the dependencies and make sure you are not blocked by explanations that make no sense.

As far as possible I’ve used plain English and kept formulae to a minimum. The examples are simple ones because it is concepts that you need to understand most.

The ideas are grouped by topic, and there’s an alphabetical index too. Each idea is identified by a word or phrase and where that is used in the text it is in bold to remind you to think of its technical meaning.

To keep you motivated the book is littered with ‘audit points’—specific issues to look for and ideas for what to recommend. There are also opportunities to test how your knowledge is growing, chapter by chapter, item by item.

In writing this book I consulted several mathematical dictionaries and many other sources, and of course they didn’t agree with each other. I’ve tried to select definitions that are logical, easy to understand, and easy to define without lengthy theoretical groundwork.

In a few cases I’ve had to invent suitable terminology to label ideas that don’t have an established name.

You should read a bit of the book every day, starting at the beginning and reading carefully. If you’re not sure about a section then read it again, but it’s not necessary for you to understand every point to gain a huge boost in confidence and competence.

Every new idea you absorb will raise your ability to understand and critically appraise risk mathematics you read. You may even find you have knowledge that some professional quants lack, perhaps because they have focused on the mechanics of what they do and not had time to look around at the bigger picture.

When you’ve finished the book it will be time to test yourself by trying to read documents about risk mathematics, preferably from inside your own organization. See how much more you can understand, at least in principle, from the text. (You won’t be much better at understanding the formulae.)

This is a vast subject area, so there will still be lots that you don’t recognize. Gradually you will get better at spotting where people are using terms incorrectly, or have not explained something they should have, and you will be tripped up by fewer details.

Be sensible about which audits you can take on using your new powers. Auditing key pricing models constructed by leading mathematicians is still likely to be out of your reach. If mathematicians try hard to cover up the weaknesses in their work you may struggle. And it still won’t be sensible to try to check the algebra or computer code yourself.

However, everyday risk assessments by non-specialists will look very different to you and they will be an easy target. Somewhat quantitative risk work by pseudo-mathematicians who barely understand what they are doing will also be your happy hunting ground. And when the big models are reviewed by quant review teams you will understand far more of what their review covered and what it did not and perhaps should have.

The myth of mathematical clarity

Mathematics does something quite brilliant. By reducing ideas to small, easy-to-write symbols it makes it possible to say a lot in a small area. Manipulating the ideas becomes dramatically easier, making possible deductions that would be desperately tiring and slow using words alone.

Mathematical thinking about risk and uncertainty is far ahead of the muddle that most of us have in our heads. Understanding its fundamentals will make many things clear to you, perhaps for the first time.

However, it’s a myth that mathematics is clear. A lot of mathematical writing is diabolical. Never assume you can’t understand some mathematics because you are ignorant or stupid. It’s very likely that the main reason you can’t understand, even after a sincere effort, is that it is written poorly.

Common problems include misleading terms, lack of type declarations (introducing a symbol without even saying what kind of thing it is), assuming the reader already knows what the writer is trying to say, using the same notation to mean different things, and using obscure letters from the ancient Greek alphabet that many people cannot name. Do you know what η is? Me neither, I had to look it up. (It’s eta.)

A great deal is usually left to the reader to know from the context, as if by magic. (Mathematics that is written to be checked by a computer program has to be written very differently.)

All this is harder still if the writer’s command of English is weak and the format is PowerPoint.

Another reason that mathematical writing is often confusing is that mathematics changes over time. New ways of doing things, new terminology, and new notation get invented and gradually permeate the literature. So what we see today is a mixture of ideas and styles from different eras.

Finally, and perhaps most importantly, mathematics uses many ordinary, familiar words but in very specific and unfamiliar ways. The scope for misunderstandings in everyday situations is frightening.

For example, imagine a room full of managers being told that ‘the expected value of this project is £3.2 million.’ Many of those managers will think this means that the project will deliver £3.2 million (perhaps give or take a few thousand) and that this is more likely to happen than not. In ordinary language that’s what ‘expected’ means in this context. What the speaker actually meant was that the project’s value is unknown but the probability weighted average of his guesses so far is £3.2 million. I hope you can see that there’s a big difference!

So, never assume that not understanding some mathematical document is your fault. Make a sincere and patient effort to understand written mathematics. Reread the text. Be wary even of familiar words and phrases. Look carefully for things that are not made clear and identify them specifically.

Then ask, just as if it was any other audit. You are entitled to a clear explanation and if someone cannot give it then their understanding must be doubted.

The myths of quantification

Understandably, many people think a mathematical approach to risk necessarily means a quantitative approach, and that a quantitative approach must have data to support it. It is true that in some famous applications of mathematics to risk (in financial markets and weather forecasting, for example) the approach is quantitative and supported by massive amounts of data. It is also true that mathematical risk analysis is at its best when quantified and supported by data.

However, it is not true that mathematics is restricted to quantitative analysis, and not true that it must have plentiful data.

Mathematics is a commitment to logical thinking. It can help us squeeze the most learning from limited data. If the only data we have are our gut feelings it can help us make best use of them. Given things we are confident we can guesstimate, it can be used to calculate things that defy intuition. It can show us where our gut feelings are logically inconsistent and could lead to us losing money.

The essence of the mathematical approach to risk is not quantification or data, but something much simpler and more familiar. It is building a model of a problem, and recognizing uncertainty around that model. You may have experienced this centuries-old method at school when tackling the topic of probability, or on a business course, or while studying for an accountancy qualification. Your thinking encompassed the whole problem or system, with ‘risk’ recognized in the form of uncertainties about values in the model, usually shown using probabilities.

Contrast this with the language and procedures of ‘risk management’ as it has sprung up over just the last few decades, where ‘risks’ are talked of as if they are physical phenomena, out there, to be found and listed, unrelated to other thinking. This perspective inevitably leads to ‘risk’ being separated from other management activities, with separate risk management meetings, risk reports, risk teams, and so on—all cut off from the main action of management.

A mathematical approach is a step forward from this, even without quantification or data.

The auditor’s mission

What is the auditor’s mission in auditing risk assessments and models? There’s plenty to go for.

Over-reliance on flawed (usually optimistic) risk assessments has led to innumerable failed projects and other business ventures, and was at the heart of the 2007—2009 credit crunch. Obviously we’d like to spot the mistakes before they can cause damage, if we can, and recommend better practices to use in future.

To be successful in this there is one thing auditors should focus on above all else when reviewing risk assessments: our human tendency to underestimate and ignore our uncertainty.

This is such a powerful syndrome of behaviours, and so universal, that it seems at times like a vast conspiracy.

The risk analyst wants to make assumptions that are easy to work with and wants to present results that seem precise and trustworthy. He sees no need to go on about the many simplifying assumptions made, or the flawed data used, or the dangers of extrapolating into the future. Why should he when his audience, managers, so often appear not to want to hear that kind of material? They want numbers that will support the actions they already want to take.

Similarly, the analyst sees no need to mention the bits of his own work he doesn’t fully understand, or the bits he knows to be wrong in small and probably insignificant ways. (Or so he thinks.) Why should he when nobody else knows enough to find him out?

Enter the auditor. Already secretly fearing that the conversation will quickly get confusing, the auditor dives for the easy tests he can understand. Were the documents signed off by people in authority? Have they been reviewed by someone with experience and credentials? Was the data used taken from a computer system that’s already been audited? (Thinking: ‘Crikey, did he just say “general linear model” and is a “Kalman filter” something else in the model or an invitation to take a coffee break? It’s time to finish this meeting and get out of here.’)

It doesn’t have to be this way. With a bit of knowledge and reassurance it should be like any other audit of something the experts know better than the auditor. Ask questions, listen carefully, don’t be afraid to probe, make an effort to grasp the issues, and close in when people get defensive.

Time and again you will find that what people have done has led to a misstatement of risk, usually an understatement. Consequently, risk is not being taken as seriously as it should be. If you can get the risk analysed or presented more fairly then you can change how people respond to it and perhaps prevent some of the disasters of the past from happening again.

AUDITING SIMPLE RISK ASSESSMENTS

This chapter introduces the most basic ideas of probability and risk and shows how they can help us audit simple risk assessments.

These are the sort of casual risk assessments that pop up in conversation and on risk registers. Even at this simple level you will find a lot of surprises and helpful insights.

To start with, in the world of business, ‘risk’ has a high profile and ‘probability’ is a word a lot of people try to avoid. In the world of mathematics the situation is reversed, with ‘probability’ the undisputed king and ‘risk’ an afterthought, sneaking in from theories about investment portfolios.

As you read on, remember how this book is designed. It’s a series of concepts and terms, each of which will help you in your work. Tackle them in order, patiently and carefully. Your objective is to learn as much as you can, not to finish the book as quickly as possible.

1 PROBABILITIES

A lot of ideas about probabilities are controversial among theorists or take a while to understand, but what we know for certain is that probabilities work. There are people who talk about and benefit from using probabilities and this has been true for hundreds of years.

One of the great pioneers of the mathematics of probability was Frenchman Pierre-Simon Laplace (1749—1827). In the introduction to his book, Théorie Analytique des Probabilités, he wrote that ‘que la théorie des proba bilités n’est, au fond, que le bon sens réduit au calcul,’ which means ‘the theory of probability is just common sense reduced to calculation.’

‘what we know for certain is that probabilities work’

Probabilities are stated about things that might happen or, more broadly, about things that might be true. For example, consider the statement ‘the probability that Happy Boy wins the 3.15 p.m. race at Kempton Park is 0.12.’ The thing that might happen is Happy Boy winning. The statement that might be true is that ‘Happy Boy will win’.

It is also generally agreed that probabilities are numbers between 0 and 1 inclusive and that a probability of 0 means something is considered certainly not true or not going to happen, while a probability of 1 means it certainly is true or certainly will happen.

Sometimes probabilities are expressed as percentages between 0 and 100%. Sometimes they are given as odds, as in ‘3:1 against’, which translates to a probability of 0.25, or 25% if you prefer. Sometimes they are given as proportions as in ‘one in four’, which is also a probability of 0.25.

Take care when translating between different styles. In the song ‘Five to One’ by the Doors, Jim Morrison equates ‘five to one’ with ‘one in five’, but of course that should be one in six.

2 PROBABILISTIC FORECASTER

It is also clear that probabilities come from many sources, which I’ll call probabilistic forecasters. Mostly they come from people (e.g. weather forecasters, tipsters, research companies, managers in companies), from mathematical formulae, and from computer systems. Some of these probabilistic forecasters restrict themselves to a very narrow topic, while others are prepared to give probabilities for a wider range of propositions or outcomes.

One question of great interest to auditors and many others is how good the probabilities from a particular probabilistic forecaster are.

3 CALIBRATION (ALSO KNOWN AS RELIABILITY)

How can you assess the probabilities provided by a probabilistic forecaster? There are two ways:

1. Look at how the probabilities are worked out (which includes looking at any data used).

2. Compare the probabilities to reality and see how well they match up.

The second method is the easiest to understand and is easy to do if you have enough data. You can’t make any assessment from just one example unless the probabilistic forecaster says something is certain and turns out to be wrong.

However, if you have lots of probabilities from the same source and you know what actually happened or what the truth was then you can calculate various scores that show how good the source is.

There are two main qualities that good probabilities must possess, and one of them is calibration.

If a probabilistic forecaster of some kind is well calibrated