The Hidden Half E-Book

Michael Blastland

• Herausgeber: Atlantic Books
• Kategorie: Bildung
• Sprache: Englisch

Seitenzahl: 423

Veröffentlichungsjahr: 2019

Beschreibung

Why does one smoker die of lung cancer but another live to 100? The answer is 'The Hidden Half' - those random, unknowable variables that mess up our attempts to comprehend the world. We humans are very clever creatures - but we're idiots about how clever we really are. In this entertaining and ingenious book, Blastland reveals how in our quest to make the world more understandable, we lose sight of how unexplainable it often is. The result - from GDP figures to medicine - is that experts know a lot less than they think. Filled with compelling stories from economics, genetics, business, and science, The Hidden Half is a warning that an explanation which works in one arena may not work in another. Entertaining and provocative, it will change how you view the world.

Leseprobe

The Hidden Half

‘Extraordinary and excellent. The antidote to books that explain everything.’

Sir David Spiegelhalter – author of The Art of Statistics

‘Brilliant. Blastland provides an explanation of the need for humility in the face of the inevitable limits to knowledge.’

Diane Coyle – Bennett Professor of Public Policy, Cambridge University

‘Fascinating... As John Wooden said, it’s what you learn after you know it all that counts.’

Andrew Gelman – author of Red State, Blue State, Rich State, Poor State

‘Beautifully written and often very funny.’

Dame Frances Cairncross – Chair, Executive Committee of the Institute for Fiscal Studies

‘Packed with fascinating examples of our shaky understanding of ourselves and the world.’

Bobby Duffy – Director, the Policy Institute at King’s College London

‘A terrific book! I read it quite literally in one sitting.’

Nick Chater – Professor of Behavioural Science, Warwick Business School, and author of The Mind is Flat

About the Author

Michael Blastland is a writer and broadcaster. He was the originator of BBC Radio 4’s More or Less, the long-standing, authoritative guide to numbers and evidence in public argument. He is the author with Andrew Dilnot of the bestselling The Tiger That Isn’t: Seeing Through a World of Numbers, a guide to interpreting numbers in public argument, and co-author with Professor David Spiegelhalter of The Norm Chronicles, about risk. He writes, teaches and advises widely on risk, evidence and data.

The Hidden Half

The Unseen Forces ThatInfluence Everything

Michael Blastland

First published in hardback and trade paperback in Great Britain in 2019 by Atlantic Books, an imprint of Atlantic Books Ltd.

This edition published in 2020.

Copyright © Michael Blastland, 2019

The moral right of Michael Blastland to be identified as the author of this work has been asserted by him in accordance with the Copyright, Designs and Patents Act of 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of both the copyright owner and the above publisher of this book.

10 9 8 7 6 5 4 3 2 1

A CIP catalogue record for this book is available from the British Library.

The following images are reproduced with kind permission of the following parties: Figure 1 © Günter Vogt; Figure 9 (left) © Shutterstock Images; Figure 9 (right) © Alamy Images.

Paperback ISBN: 978 1 78649 639 3

E-book ISBN: 978 1 78649 638 6

Printed in Great Britain

Atlantic Books

An imprint of Atlantic Books Ltd

Ormond House

26–27 Boswell Street

London

WC1N 3JZ

www.atlantic-books.co.uk

For Alan, for the rockets

The great menace to progress is not ignorance but the illusion of knowledge.

Daniel Boorstin, Cleopatra’s Nose: Essays on the Unexpected

Contents

Prologue

The marmorkrebs and the hidden half

1. Bill is not Ben

Hidden influences on the path through life

2. I am not constant

Unsuspected causes of belief and choice

3. Here is not there, now is not then

Obscure differences that subvert knowledge

4. One path is not enough

What ‘finding out’ through research doesn’t see

5. The principle isn’t practical

Big ideas and the small print

6. Big is not small

The hidden limits of what’s probable

7. Big is not clear

Hidden complexity in simple stories

8. The ignorant chicken

On ignoring the hidden half

9. What to do

Postscript

Why the hidden half?

Acknowledgements

Endnotes

Index

Prologue

The marmorkrebs and the hidden half

It ain’t what you know that gets you into trouble, it’s what you know for sure that just ain’t so.

Mark Twain (possibly)1

In the mid 1990s, rumour ran through the small world of aquarium owners and traders in Germany of a peculiar new beast, previously unknown to science. There was no hint of it in the wild, so how this creature turned up in the German aquarium trade, no one’s sure. One day it didn’t exist, the next it did – in a fish tank.

The marmorkrebs, as they were later named, were a new species of crayfish, similar to other crayfish – apart from one conspicuous fact: lone females had spontaneously started to lay eggs that hatched without fertilization, the process known as parthenogenesis. These crayfish no longer needed to mate. The mother’s offspring were natural clones.2

‘People wondered, it’s only females, where are the males?,’ said Frank Lyko, an epigeneticist, adding that normal evolution of a new species can take thousands of years.3

No other crayfish or related creature, none of the approximately 15,000 species of decapod including crabs, shrimps, prawns and lobsters was known to do this. No one seems to know how it came about, other than to speculate on a spontaneous mutation, one day, in just one crayfish in one fish tank, the marmorkrebs’ Eve.

All this was peculiar and wonderful enough, but the story gets better: having startled us with their out-of-the-blue existence, the marmorkrebs were about to ruffle a mass of human presumptions.

First, they caught the eye of scientists who, in a short report in the journal Nature in 2003, revealed them to the research community with a storyteller’s flourish:

It has been rumoured that an unidentified decapod crustacean, a crayfish of marbled appearance and of uncertain geographical origin that was introduced into the German aquarium trade in the mid-1990s, is capable of unisexual reproduction (parthenogenesis). Here we confirm that this marbled crayfish (‘marmorkrebs’) is parthenogenetic under laboratory conditions.4

Cloning helps to make marmorkrebs a menace in the wild, where they are said to be invasive, as the release of just one can establish a whole population. They are also ‘robust and highly fecund’ as the research put it: quick to mature and laying a lot of eggs. ‘You put them into your aquarium and a year later you have hundreds,’ said Frank Lyko in 2018, after Eve’s offspring became briefly famous for scuttling free and over-running Madagascar.5 It was a headline writer’s dream: ‘Invasion of the mutant crayfish’.

But the greater interest of these cloned creatures to science lay in an altogether different direction: in the marmorkrebs’ potential to help with that old and thorny question of the balance of forces between nature and nurture, as researchers realized they had stumbled on an ideal experimental subject.

Ordinarily, it’s hard to tease apart why things turn out the way they do. If you fall ill with heart disease, there is a tangle of potential causes, genetic and environmental; the fault could be inherited, dietary, linked to exercise (too little/too much), stress, some combination, and so on. By holding constant the role of genes, clones make it easier to tease out other influences. Whatever happens to them, one compared with another, pure difference of genes will not be the explanation.6 These cloned crayfish were a research godsend.

So it was that, one day in Germany, researchers chose two founder females to be the great mothers of two laboratory lineages, naming them A and B (such are the names of legends when you’re a crayfish). They popped A’s and B’s offspring into tanks of water to see what happened. Each of the two lineages was genetically identical, naturally. This was not merely assumed; they were checked for genetic consistency.

But the research went further. The marmorkrebs were also all hatched and raised in exactly the same lab-controlled environment. Now, every influence on their development was as consistent as possible. They were fed the same food (Tetra Wafer Mix, since you ask), checked regularly for disease, and reared in simplified tanks containing room-temperature tap water. It was even arranged for the same person to examine them on every occasion. The aim as far as humanly possible was to eliminate every variation that anyone could think of. They were born into the most boring uniformity humans could contrive.

What were they like, these clones in an identical environment? Take a moment, hazard a guess. About the same? Maybe exactly the same?

After all, we know everything we know to be knowable about these crayfish, and what we know is the same for each and every one of them. Both genes and environment – the two great forces in life, two titans of human understanding between which there is a perpetual war of explanatory power – are identical for all.

But look at them. Figure 1 shows a group of laboratory-bred marmorkrebs from the same batch of eggs.7 It appeared in a research paper written by members of the German lab, published in 2008 and it’s one of the great pin-ups of genetics, or should be.8 Evidently, the marmorkrebs are very far from the same. Among identical batch-mates in the same conditions, one crayfish grew to be twenty times the weight of another.

Figure 1. Genetically identical marmorkrebs from the same batch of eggs in the same environment.

The visible differences are stunning, and size is only the first. Every single marmorkrebs of several hundred studied had a unique pattern of marbled markings. There were visible, physical differences in their sensory organs, differences in internal organs, differences too in how they moved and rested: some sat under shelter, others lay on their backs. Another big variation was lifespan, which ranged from 437 to 910 days. The onset of reproduction differed hugely too, as did the quantity of eggs and number of batches. While laying, some fed, others didn’t. Some marmorkrebs moulted in the morning, others at night.

There were more differences in how they socialized. Brought together in one tank, they soon fell into hierarchies, some submissive, some dominant. Some were solitary, others liked a crowd. They were physically different and behaviourally different. They were genetically identical, in a world where everything was as far as possible identical, and they were chalk and cheese.

When the same is not the same

From expectation to outcome is such a non sequitur it’s as if whitewash came out striped. But if this difference out of sameness is the first shock, two other thoughts follow. The first is that whatever we thought we knew about genes and environment suddenly becomes – can it be true? – in need of serious revision. Normally, we say that if it isn’t genes it must be environment; if it isn’t environment it must be genes. But this seems in some way to be neither. And with that, a pile of presumptions goes up in smoke and we sit scratching our heads. The laws of development most of us think we know about ought to imply no difference, yet the differences are stark.

The next thought is equally disconcerting: that if this story is true (it is), there must be something else, something we’re missing, some other potent but hidden influence that causes these differences – an influence, by the way, that continues throughout life. The researchers discovered that every marmorkrebs is capable of ‘changing randomly. . . during all life stages’. Why? How? They are the same thing in the same environment. Having straitjacketed the two big causes of everything, what makes the results so disorderly?

The short answer is: we don’t know. There is nothing approaching a full account of where the variation comes from. Big as the differences are between the marmorkrebs, we’re hard-pushed to explain them in any but the most general terms, and these are little help identifying the path of their development.

One tempting explanation is epigenetics. This is how genes are switched on and off – for example, to cause cells with the same genes to make eyes, kidneys, heart, etc. Epigenetics is also used to describe the interplay of genes and the environment (GxE) to produce stable effects that persist as cells divide. But this just pushes the question back: where do these varied epigenetic effects originate when everything we know about the marmorkrebs and their environment is the same? Epigenetics might explain how the influences are mediated – and for that reason it’s fascinating – but that doesn’t tell us where the influences come from. What flicks a marmorkrebs’ epigenetic switch one way and not another to cause so much variety? We don’t know.

Another tempting explanation is short-term gene–environment interaction. Genes do not directly determine how a creature turns out, but code indirectly for proteins throughout life, a process under potentially continuous outside influence. This leaves plenty of room for gene–environment interaction. It’s not the same as epigenetics, since it needn’t produce long-term stable effects, but more to the point it still doesn’t address the problem that every input to this GxE interaction in the marmorkrebs’ case is, as far as we can tell, the same.

Finally, although it makes sense for clones to find a way to be different, to spread their evolutionary bets so that at least one is more likely to survive changing conditions, that doesn’t help either to explain how they do it.

In short, we’re stumped. And perplexed. When I first show people the marmorkrebs’ picture, their reaction often begins with a ‘But. . .?’ as if there must be a simple objection. And then they stop, lost for words, and stare. You watch bafflement play on speechless faces. Old convictions rattle. So it’s not. . . but if it’s not. . . then what. . .? Surprised to find that differences like this exist to be explained, they struggle – like all of us – to imagine what might be missing. But something sure is.

This is what I like to call the shock of ignorance. It’s a good moment, a forced recalibration. It reminds us how easily we can be satisfied with established ideas, and what amazement might lie around the corner.

Out of that moment, we’re forced to start thinking. Because the marmorkrebs’ variation must have a cause, mustn’t it? That’s a statement of the obvious. Though you might begin to wonder as the cause seems to come from nowhere. It’s tempting to speculate that one crayfish – eventually the biggest – grabbed the first food, ate more, grew bigger, then threw its weight or energy around to take the lion’s share and grow bigger still, so an initial advantage was compounded. Except that they all had more than enough to eat, all the time; the researchers made sure of it.

The genetic make-up of an individual is called the genotype. How it turns out to look and behave is the phenotype. The emergence of multiple, varied phenotypes from a single genotype in the same environment is one of the latest, greatest of nature’s curiosities to crave attention.

The secret life of causes

We’re left to wonder if the causes lie beyond discovery, maybe hidden in the minute detail of the marmorkrebs’ experience or development. You find yourself groping for obscure possibilities, like which one was first to sense the sunlight through the window in the morning, which was nearest the lab door or the air-con. The researchers write, teasingly, that what they call ‘microenvironmental influences’ were reduced to a minimum by the experimental design ‘but can never totally be excluded’. It’s hard to know what a micro-environmental influence might look like to a marmorkrebs, but again you wonder: are these unknown micro-experiences and influences the source of major differences, as if the tiniest, random nudges, invisible to science, feed back and amplify the marmorkrebs’ idiosyncratic development? Or, as the research puts it, there are ‘non-linear, self-reinforcing circuitries involving behaviour and metabolism’ beginning who knows where, leading to who knows what. In short, is hidden trivia capable of growing magically into a whole forest of consequence?

Curiously, put the marmorkrebs in groups and one group shows a different spectrum of variation to another, as if the particular chatter in each group alters everyone in it. This is despite the fact that every creature, across every group, is genetically identical. Could simple, spontaneous interaction be part of the answer? This may partly be the case for those kept together, though how exactly it could account for so many diverse effects is mind-boggling. But what about differences – also evident – in those reared separately? Maybe, even though they all had more than enough food, once they were in the same tank there was a race to the first morsel that left one crayfish feeling like a winner, another a loser, all hanging on nothing more than who was closest to that first morsel when it sank. Was this enough to establish a pecking order? Maybe the biggest crayfish just decided to eat more, independently of any other influence, according to whatever you believe about the possibility of crayfish free-will. Or maybe all these differences originate in an element of pure, lawless randomness at a primitive stage of development, buffeted by more randomness along the way – if such a thing as pure randomness exists, here or anywhere. You can tell I’m guessing. We all are.

Whatever the cause or causes, this is a good moment to remind ourselves that great ideologies have been built on competing beliefs about genes versus environment, nature and nurture. Millions have been slaughtered in the name of claims that the differences between us are in the blood or can be socially designed. Science has moved on from these simple polarities, but they remain a source of bitter and still bloody argument. Yet here’s evidence that neither explanation as traditionally conceived is adequate, not even when they’re combined.

Clearly, genes matter: the marmorkrebs’ offspring are still marmorkrebs. Clearly, environment matters: if they had no food at all, their lives would all be short. These are big forces. Equally clearly, something not quite either, as normally understood, is heavily in the mix. But what?

Like the rest of us, the marmorkrebs researchers scratched their heads, then named this elusive factor ‘intangible variation’. All that ‘intangible variation’ means is that, even when everything seems the same, something out there makes things vary, but we don’t know what – the source is intangible. So the phrase is intriguing, but ultimately frustrating. Another term the researchers use is ‘developmental noise’, which sounds even less helpful, certainly not the sort of thing you’d want to waste time studying. ‘Noise’ in research is supposed to be the irrelevant stuff, to be screened out in the quest for a consistent signal. Who wants to listen to noise?

Forgive me for a moment, but sod the consistency. What we have to explain here is the inconsistency. The so-called ‘noise’ is what’s so arresting. These creatures are different and we don’t know why.

And they are not alone. This has become a recurring puzzle in a variety of animal studies when attempts to standardize everything have failed to suppress significant variation.9 The types of variation in each case are different – we don’t always see a huge range of sizes, for example – but there is always variation, always unexplained.

In fact, this is turning out to be such an extensive problem that some researchers say we should recognize a third source of developmental difference.10 They don’t know what this third source is, exactly, but they do know there is a substantial hole for it. Something must explain these differences. And whatever that something is has a power in some cases to equal or eclipse the other forces combined: a whole hidden half of explanation for how nature turns out.11 This isn’t detail, in other words, it’s fundamental. Yet even the existence of this large unknown is itself largely unknown. Almost no one I talk to about it is aware of its power.

The noise is the signal

There is another question, more fundamental again: what if ‘noise’ – the research world’s irritating aside – is the great, avoided point? By that, I mean to question the habit of talking of ‘noise’ – as I think we do – as if it’s the intellectual dross left over after our genius has discerned life’s vital patterns. What if we thought of noise instead as a pervasive, positive force for disorder, as vital as any other? As I say, it’s not the regularity here that’s most striking; it’s the irregularity. Treating it as a leftover that we assume we’ll get around to explaining away some day won’t do. We need to face the possibility that big influences are not as orderly or consistent as we expect, that the way things turn out is bound less by observable laws, forces or common factors than by the mass of uncommon factors, the jumble of hidden, micro-influences. Our habit of thinking of this as ‘noise’ – and then thinking of ‘noise’ in turn as an annoying residual – diminishes one of life’s most magical elements.

According to the marmorkrebs researchers, the problem of unexplained variation has been ‘largely untouched’. That seems incredible, given its implications. So beguiled are we by chasing hints of order that it seems we’ve not looked properly into the forces of disorder. What is this intangible force for difference, seemingly so surprising, so strong but so little studied and so vaguely described? Where else might similar problems arise?

You have to admire the marmorkrebs, going about their scuttly business, rattling our cages. By finding their way into the wild where they became a pest in places, they are a model for my ambition: to make the questions they inspire equally irritating.

We need to consider seriously the case for another, hidden force; for a re-appraisal of what we dismiss as ‘noise’ and push to the margins of our attention despite its power. We also need to reflect more on the disruption it causes. If even clones in the same environment – where the problem is as simplified and controlled as humanly possible – are not the same, owing to the power of intangible variables, how reliably can we expect to pinpoint the sources of difference between people, businesses or policies, in all their infinitely messy complexity?

At the very least, we could pay more attention to how subtle and unexpected variation can subvert what we think we know. Since knowing when one thing is like another (and will therefore behave in a consistent way) is the root of claims to understand and control anything, what if similar, irregular forces exert equally underestimated influence over other aspects of life: in politics, business, crime, education, economics, over how we make decisions, and much besides?

After all, it is the most basic property of knowledge (if it’s to be socially or scientifically useful) that it must travel. Knowledge must generalize wherever we want to use it, otherwise it’s not knowledge. Failures of knowledge often become clear when it doesn’t travel as we expect. We thought we knew something, thought we’d seen our knowledge at work, thought we understood why it worked; then we tried to apply it again, perhaps in only a slightly different context, expecting it to work again, and it didn’t. Only then might we reluctantly concede that we didn’t really know what was going on quite so well as we thought we did.

Among the marmorkrebs, even the Herculean forces of genes and environment don’t travel as expected. Despite holding constant the two influences that seem between them to leave no room for irregularity, these crayfish are breathtakingly irregular. We’re forced to conclude that genes and environment, traditionally understood, are not the complete forces we thought.12

Like many, I like to think I’m as rational as they come. I’m pro-science to my second anorak. But the marmorkrebs remind us all how much we can miss, how much remains hidden, and what dangerous fools we can be whenever we think that we know.13

Enigmatic variation

This book advances three ideas or arguments. The first is that we need to face up more readily to the many mysteries and surprises that humble human understanding – like the mystery of the marmorkrebs. We’ll look especially at cases – again like the marmorkrebs – that raise doubts about how reliably we discern patterns and order. Although we can be amazingly good at this, too many of us are reckless about the limitations of our cleverness. There is, all too often, a hidden half of understanding that we are bound to miss and don’t even like to acknowledge. The evidence of this problem, I think, is abundant, and the book presents it.

But how do we explain it? This brings us to the second idea.

One approach we’ve heard already. This is simply to call the element that causes all the trouble ‘noise’ or ‘chance’ – and be done with it. So, we might just say that the marmorkrebs are different because of ‘chance factors’, then shrug, and move on. This would not be wrong, but it is not enough. There are compelling reasons to treat the problem in a different way. Chance can feel like a whim of the Gods, noise like a tedious irritation – abstractions both, lacking substance. They can be too resigned, and too dismissive at the same time. That’s the wrong reaction in the face of such amazement. I think we can do better and bring these abstractions to life as a positive force for disorder.

Another approach to explaining why we know less than we think we know would be to invoke human irrationality or cognitive biases: the limiting effect on our understanding of the systematic ways that we distort or frame reality which lead to misjudgement and error. These undoubtedly play a part, but I will largely ignore them. That’s partly because cognitive biases have already received much excellent attention. But I also have a mild doubt about the current emphasis on people’s cognitive limitations: that it might suggest all we need do to overcome them is become a little smarter – which if you’re a reader of books about cognitive bias of course you will be. Whereas if – as I argue – a large part of the problem is an obdurate property of the world at large, rather than (primarily) in other people’s psychology (not ours, we’ve read the books), then flattering ourselves about our own exceptional genius will get us nowhere. People do take mental shortcuts, and they do go wrong, and although this is undeniably a problem partly to do with our own thinking, one reason that we take shortcuts is the sheer complexity of what we’re grappling with. The nub of a problem is often out there, and insurmountable.14 You could be the rational paragon of philosophers’ dreams and still be not one jot closer to discovering why the marmorkrebs are so different. So, although it may be true that we’re riddled with cognitive faults that mean we don’t see the world as it is, we have to ask to what extent this world is so irreducibly awkward as to be beyond even the most rational understanding. At any rate, we should think harder about the nature of its awkwardness.

The second idea then, is to put aside questions of rationality for a moment, and also put aside labels like chance and noise. Instead, we will dig into this hidden half of enigmatic variation, see it as a positive force for disruption, and try to discover a little more about how it works to undermine what we think we know. Above all, I’ll aim to make its characteristics indelibly vivid. The language will be that of difference, disorder, variation and irregularity. This is to make explicit the connection between knowledge based on expectations of regularity, and the frustration of these expectations by irregularity, so that we see clearly how one opposes the other.

The key to this concept is that difference means ignorance. Things that we expect to be the same – and therefore think we understand – are often not really. Do enigmatic differences like those between the marmorkrebs exist elsewhere? I believe they exist just about everywhere. We dream of laws and general truths; the practicality is often a patchwork of unexpected anomalies.

Run with these ideas, apply them more widely, and you begin to conceive a world bustling with powerful but enigmatic differences that we just don’t see. Whether in politics and policy, business, medicine, economics, psychology, human development, science generally, or elsewhere, we’ll look at evidence that the laws, principles, lessons, forces, findings that we hold up as knowledge often don’t travel as well as we think they will from experience or theory to practice, or from one instance of real life to another. And in each case, thinking about the hidden half can help us see why.

Facing our limitations

These first two thoughts – less knowledge, more awkwardness – demand the third: bluntly, how to cope. If we know (and can know) less than we think, if the world falls into line less readily than we suppose, what do we do?

For some, this is the most unwelcome part of the argument. When I try it on audiences of journalists and others, the reaction often starts with denial – who are you to you tell us we don’t know or words to that effect – or maybe, but we always get there in the end, what’s your problem? Then, as we consider more evidence of the kind we’ll consider in this book, the mood shifts to defeatism, sometimes laced with panic:

So, we don’t know anything? What are we supposed to do, give up?

Or:

What are you saying? It’s all pointless, nothing’s true, nothing works?

But slowly they adapt. They work out strategies that depend less on bullish assertions that we simply know, more on coping with only half an understanding. They come around to what’s still possible. As the economist John Kay has put it: ‘Instead of saying “I really need to know this,” we need to say, “I can’t know this and how do we manage in a world when we don’t.”’

Because the good news is that the case is far from hopeless. There is no need for nihilistic despair. The third idea, then, summarized in the final chapter, is what to do – and we can do a great deal – in a world of daunting uncertainty. As Daniel Boorstin wrote in what serves as the epigraph to this book, it’s not ignorance that’s the greatest obstacle to progress, but illusions of knowledge. We are in sore need of dispelling some of those illusions so that we see the way ahead more clearly. A keener sense of the world’s subtle intrigues might help.

Mansions of straw

I’m a journalist, a hack. What do I know? That’s not modesty; it’s the feeling of being intimidated. This is a hard and ubiquitous problem, and we will have little choice but to skate over oceans of scholarship.

Two things. First, because I’m a journalist, I’m free to follow ideas regardless of specialism, and foolish enough to trample across boundaries to look for them and combine them.

Second, I’m interested in how ideas show up in public debate. Whatever the theory (which by the way I enjoy), the public discussion (which by the way I don’t) of what we know is failing to get things done. Might more pragmatic, intuitive ways of talking about why we often don’t know what we claim to know help us to get things done a little more reliably? With the right way of conceptualizing the problem, it should be possible to cut down some of the pretending, change a few habits, and maybe encourage a little more constructive realism. If I’m right, this is partly a question of imagination: how we can picture the problem more vividly so we’re more alert to it. That’s something maybe a foolhardy generalist can attempt.

More striking than what I think is how many others say something similar: that we have somehow over-reached, and now wake up to the fact that life is not the shining edifice of robust understanding that our mass of research findings suggests. These findings are prolific, to be sure. But they have started to fall over at an alarming rate, failing to replicate as scientists seek to repeat each other’s work. You might have heard talk of a replication crisis, even a crisis of expertise, or research-credibility crisis. Take a moment to absorb that phrase: ‘research’ faces a ‘credibility crisis’. We’re not sure what to believe even from people whose purpose is finding out what to believe. If the knowledge factory, of all places, can’t be relied on for knowledge, we know we’re in trouble. By one estimate, most published research is false.15 We have become, said one genuinely respected researcher, prone to building mansions of straw, rather than sturdy houses of brick.16

The crisis narrative can be overdone, partly because we have little idea whether the problem is actually any worse than it used to be. It might have been like this for decades. In any case, science is not broken. It still produces an immense amount of good work. But it is unquestionably going through a new and intense period of soul-searching about its methods and reliability. To any outsider, that is a shock. Even defenders seem to agree that we have recently laid bare huge scope for improvement.

Controversy about experts flared up at the time of the UK’s Brexit referendum after a comment by a politician, Michael Gove, widely reported as: ‘people have had enough of experts’. To be fair, he didn’t simply throw a bucket of scorn at all experts. He was more careful than that, and actually said: ‘People have had enough of experts from organisations with acronyms saying that they know what is best and getting it consistently wrong.’ Which, if that is what these experts did, would be reasonable enough. He didn’t say whether people had also had enough of politicians, and I suspect at the time he had little idea of the research credibility or replication crisis, which is more serious and remains under-reported.

This book is written against that more serious background – of anxiety about failures of knowledge in the social and human sciences, coupled with a movement to raise the game. It has various names, this movement – ‘meta-science’ is one you might come across if you haven’t already. Some people say it can be a force for good, in that we are at last becoming aware of the problem, which means it can be addressed, which in turn promises better science. Positive or not, the message seems to be that we have been too confident of our ability to see patterns, detect regularity and further knowledge, and too careless of the perverse incentives for researchers to produce flashy but false findings. The overwhelming feeling as you dig into the background is distress and confusion among the many talented and well-intentioned people who have been caught up in an institutional convulsion. But as you read across disciplines you can’t ignore other feelings: embarrassment, shock and frustration, anger, sometimes denial, and more often a determination to improve. Overall, there’s a whiff of revolution.

Those of us looking on from other walks of life should take a deep breath and reflect frankly on the provenance of our own understanding. Because the rigour in research is usually better than elsewhere, and still they somehow unintentionally screwed up on a worrying scale. Do the rest of us think we are immune?

It seems unlikely. Mansions of established wisdom have gone down like slum clearance lately. Take the many astounding events in politics: the elections of relative outsiders Donald Trump in the US and Emmanuel Macron in France; or the rise of a populist political right in many parts of Europe. Or consider the UK’s Brexit referendum, or the rise of Jeremy Corbyn, a left-wing Member of Parliament whose chances of the leadership of his party were rated zero by most observers, who then took firm control. Even democracy, some say, is suddenly in retreat.17 Political assumptions have fallen to the extent that one professor of politics wondered publicly if it was time she and colleagues tore up their lecture notes and started afresh.18

Then throw in economics and business, with their global banking and financial crises, deep recession, slow recovery and an enduring low-productivity puzzle in many OECD countries,19 plus a historically unprecedented stutter in the growth of average earnings in some leading economies – and all in all you have to wonder if this is a time for a large helping of what philosophers call epistemic humility (intellectual humble pie to the rest of us). If we thought we knew, we were kidding ourselves.

The redemption for research at least is that it can study its own errors – and through movements like meta-science it is working out what went wrong and what to do differently. The physicians begin to heal themselves. This enables the book to steal from their insights on the way to a re-evaluation of how people should try to find things out, argue, analyse evidence and decide what to do – with lessons for every walk of life.

Of course, despite its stumbles, knowledge does advance. I’d even go along with those who say that recent setbacks have in a roundabout way moved us on. My argument is not that we are bound to fail. Nor that trying is somehow misguided. It’s that we can’t help turning up our pattern-making instinct to 11 – when life offers only a 5. Too often, we make bold claims about big forces with law-like effects, but with culpable overconfidence that leads us to waste time, money, talent and energy, and detract from real progress. I’d like to dent that overconfidence. I’d like our claims to be more proportionate to the awkwardness of the task. Every new generation needs reminding of the overconfidence of every previous generation, of how much there is still to know and do, and, above all, how resistant the raw materials of life can be.

One proviso. If you are tempted to see this book as an excuse to resist evidence that doesn’t suit you, or to think that because we are shrouded in ignorance anyone’s opinion is as good as anyone else’s, stop here. I’ve no desire to dismiss or discourage genuine, careful and humble efforts to understand, and no desire either to knock down robust houses of brick alongside the mansions of straw. It would be easy, but deluded, to see this book as part of an anti-science cynicism that says everything is uncertain, and therefore nothing can be done. I reject that view entirely. On the contrary, I want more robust evidence precisely so that our decisions and actions can be more reliable. I sympathize entirely with how difficult it is to do that well. I applaud those who devote themselves to the problem conscientiously and carefully. This is why we must recognize our limitations, try to understand how they arise, tread more carefully and test what we know vigorously. It was once said that at certain times the world is over-run by false scepticism, but of the true kind there can never be enough.20 This book aspires to the true kind. The goal is not cynicism; it is to do better.

In which case, what on earth is going on in that hidden half?

1. Bill is not Ben

Hidden influences on the path through life

Pause you who read this, and think for a moment of the long chain of iron or gold, of thorns or flowers, that would never have bound you, but for the formation of the first link on one memorable day.

Charles Dickens, Great Expectations, 1861

You’re sceptical. The marmorkrebs are surprisingly irregular, that’s true, and leave a lot of unanswered questions, but they’re just a weird species of crayfish. They’re not people and might not be anything like people. We don’t know how the marmorkrebs think – so far as they think at all – we can’t interrogate them to learn their reasoning, or see closely enough what happens to them to pick out patterns of decisive influence, cause and effect. It’s unsurprising that the source of the variation in their lives is hidden – at least from us.

Then let’s turn to people, in whom we are compulsive pattern spotters. Maybe in people we can uncover at least some of the missing causes we can’t see in crayfish, not least because people – unlike marmorkrebs – can talk about how their experiences shape them.

The question we seek to answer as we look for patterns of cause and effect in human experience is how far these patterns are real. To what extent are they regular and consistent, and therefore examples of useful knowledge that will travel? Or is there, between people as between cloned crayfish, a hidden half of enigmatic variation that subverts our understanding?

To find out, we’ll look in this chapter at a variety of human cases where you might expect the patterns and order to be clear, then see how well that order holds up. And we’ll begin with the life of Mike Tyson, the former boxer who has a lot to say about how he was shaped by his childhood, which might help us to think about the patterns of cause and effect that make people turn out the way they do.

It’s all because. . .

Mike Tyson was born on the wrong side of the tracks. His early life included a father who wasn’t really his father who soon fled the coop anyway, serial homes in condemned buildings and a heavy-drinking mother who could be violent to anyone who crossed her, children included.

In a profile in Rolling Stone magazine, Tyson is described as ‘finding his identity’ by robbing houses, beating people up and doing drugs. He first snorted coke when he was 11 and, by 13, had been arrested 38 times. He finished up in the Tryon School for Boys, an institution described elsewhere as New York’s most infamous juvenile prison. As he sums it up in his autobiography: ‘I did a lot of bad sh**’.1

Is it any wonder Mike Tyson grew up to become the self-styled baddest man on the planet, with a rap sheet that kept growing? As an adult, he bit off part of another boxer’s ear and was imprisoned for rape. But how else was he likely to turn out? He is among the first to say that his entire life, even beyond boxing, was defined by violence. Except, there is also the case of Mike Tyson’s brother.

‘Once, my mother was fighting with this guy, Eddie’, said ‘Iron Mike’ Tyson himself in another interview, ‘and it’s barbaric. Eddie knocked out her gold tooth and me and Denise [his sister] are screaming. But my mother’s real slick. She puts on a pot of boiling water. The next thing I know she’s pouring boiling water over Eddie. He was screaming, his back and face covered in blisters.’

The former heavyweight world champion regards these influences as formative: ‘I used to look upon women as equal in physical confrontation. I didn’t grow up around frightened women. If you sleep they might kill you.’ Without excusing Tyson’s behaviour, we might say this is how it is if your life begins in brutality: history is destiny. That is, we might be tempted to say so, except for Mike Tyson’s brother.

Mike Tyson’s brother, Rodney Tyson – who features in stories of childhood theft and arrest in Mike’s autobiography where he is described confronting Mike with a gun2 – became a specialist surgical assistant in a hospital trauma department in Los Angeles, where his job included helping to patch up the victims of crime.3

How does that affect our view of the power of a criminal childhood to shape a young person’s future? Maybe it changes little. Maybe we start compiling a mental list of ifs and buts that might explain the difference. Maybe we change tack and wonder if Rodney Tyson is also haunted by his past but, unlike Mike, reacted against it, rejecting the norms of his background, rejecting the violence and crime in his home and neighbourhood, committing his life to trying to heal wounds. You see, he was shaped by his past too, like his brother, but in an opposite, compassionate direction.

But if the same background causes brothers to become so different, how do we say which way that background points? Misogynist thug or vocation to help? When paths that begin in the same extreme place end so far apart, how far can we say the same beginning in crime has any consistent effect at all?

There are other complicating facts. In his teenage years Mike Tyson was taken under the wing of a paternalistic boxing coach who showed him a life of privilege and high expectation. Apparently, he lived in a large Victorian house described in places as a mansion. His life became disciplined, if still motivated by vicious intent. The Mike Tyson story is easy to fit – if we’re inclined – into a world of big causes. But which is the precise cause and effect now?

Maybe you say that by the time he met his boxing coach it was too late. Maybe you say it was the very privilege, suddenly conferred, that did the damage, or that the coach’s motivational fighting talk made things worse. Maybe dreams of a world of wealth and women at his teenage feet put the last touches to a monster. Or maybe this pigeon-fancying, self-proclaimed shy and frightened child grew up misunderstood. And after all that, maybe you still insist that, nine times out of ten, anyone who grew up like Mike Tyson would remain in some way like Mike Tyson.

While there might be truth in some of this, let’s also consider the limitations, because doubt also creeps into speculations like these. How much is storytelling after the fact? Are we really spotting order, a pattern of cause and effect, or are we weaving together selective details to justify whatever pet theory we hold about child and adolescent development?

A question then, to test the limits of our knowledge: if a boy is delinquent, how reliably can we tell whether he will be a criminal adult? Is there a type that persists, and a type that desists? Can anything in their background help us? To be clear, we are looking not at how young people get into crime, but at how that most extreme of early experiences – a criminal childhood – shapes adult life.

You are, by the way, welcome to consider even the most high-rate chronic offenders. If being the worst-case kid on the block gives any indication of whether or when someone will go straight as an adult, or if there is anything else distinctive in the lives of those who are still burgling houses in their 60s compared with those who stop 40 years earlier, we can say there’s at least some semblance of pattern. Can we?

The extent to which the patterns are real is explored in an extraordinary study of delinquent boys by John Laub and Robert Sampson, called Shared Beginnings, Divergent Lives: Delinquent Boys to Age 70.4

The story goes that these two professors of social science happened one day on some dusty boxes of data in the basement of the Harvard Law School in the US that contained studies of 500 men who had been in serious trouble in childhood – plus another 500 who despite similar backgrounds had not. It was a wonderful find. The original study5 followed the men from birth in Boston between 1928 and 1930, up to age 32. John Laub and Robert Sampson analysed that data and then picked up where it left off to complete their own study, tracking down as many of the men as they reasonably could for the longest life-course study of criminal behaviour ever conducted. The crimes, thousands of them, ranged from fraud to armed robbery and serious physical violence. The first recorded arrest was at age 7; the last was at age 69.

Simply finding the men was a detective story. The last known addresses were 35 years old. Social security records were patchy, phone numbers scarce and usually useless. Some men turned up through guesses that they’d Americanized their names – found by changing the search from Pasquale to Patrick, for instance. In the end, the researchers enlisted the help of a lieutenant detective on Boston Police homicide department’s cold-case squad.

When found, some men didn’t want to talk. For a few it was too long ago. For others, it was too current. One – thought to have connections with organized crime – told them in plain language to stop bothering him. Some had died. They interviewed the men in their homes, in burger bars, in ‘Daddy’s Donuts’, in the Kennedy library, in ‘a beat-up old, brown Ford that reeked of cigarette smoke’, and in prison.

This huge effort, remarkable in itself, was motivated by their earnest belief that they’d be able to identify precisely the influences that might guide attempts to turn criminals straight. If we know the mechanism, maybe we can use it for good.

But for all their efforts, they hit a problem. They realized – ‘to our fascination and simultaneous horror’6 – that they simply couldn’t tell from the mass of data at their disposal whether any particular person would continue to be criminal in adulthood. Nothing turned up. While it seemed to be true that some tended to become set on certain paths, there was abundant evidence that others could change. But who would become set, who would change and when, was another matter. As they compared the life-courses of criminals with others whose early lives were much the same but later became law-abiding, and they scoured those divergent lives and experiences for clues about the future, they found they couldn’t pinpoint anything that would have told them which path anyone would take. Some delinquents stopped, some carried on, some zig-zagged. Nothing in their background suggested why. Hence the title of the book: Shared Beginnings, Divergent Lives. All that we know is that most desist eventually. But the gradient of giving up is about the same for any group you care to define, while that part of the story for any individual seems unpredictable.

Their conclusion was stark, and complete:

[We] reject the idea that childhood experiences such as early involvement in antisocial behaviour, growing up in poverty, and woeful school performance are sturdy markers for predicting long-term patterns of offending. . . that individual ‘traits’ such as poor verbal skills, low selfcontrol, and difficult temperament can explain long-term patterns of juvenile delinquents. . . [and] that offenders can be neatly grouped into distinct categories, each displaying a unique trajectory and etiology of offending.