An Anthology of Global Risk E-Book

SJ Beard

• Herausgeber: Open Book Publishers
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Veröffentlichungsjahr: 2024

Beschreibung

This anthology brings together a diversity of key texts in the emerging field of Existential Risk Studies. It serves to complement the previous volume The Era of Global Risk: An Introduction to Existential Risk Studies by providing open access to original research and insights in this rapidly evolving field. At its heart, this book highlights the ongoing development of new academic paradigms and theories of change that have emerged from a community of researchers in and around the Centre for the Study of Existential Risk. The chapters in this book challenge received notions of human extinction and civilization collapse and seek to chart new paths towards existential security and hope.


The volume curates a series of research articles, including previously published and unpublished work, exploring the nature and ethics of catastrophic global risk, the tools and methodologies being developed to study it, the diverse drivers that are currently pushing it to unprecedented levels of danger, and the pathways and opportunities for reducing this. In each case, they go beyond simplistic and reductionist accounts of risk to understand how a diverse range of factors interact to shape both catastrophic threats and our vulnerability and exposure to them and reflect on different stakeholder communities, policy mechanisms, and theories of change that can help to mitigate and manage this risk. Bringing together experts from across diverse disciplines, the anthology provides an accessible survey of the current state of the art in this emerging field.


The interdisciplinary and trans-disciplinary nature of the cutting-edge research presented here makes this volume a key resource for researchers and academics. However, the editors have also prepared introductions and research highlights that will make it accessible to an interested general audience as well. Whatever their level of experience, the volume aims to challenge readers to take on board the extent of the multiple dangers currently faced by humanity, and to think critically and proactively about reducing global risk.
 

Leseprobe

An Anthology of Global Risk

An Anthology of Global Risk

Edited by SJ Beard and Tom Hobson

https://www.openbookpublishers.com

©2024 SJ Beard and Tom Hobson

Copyright of individual chapters is maintained by the chapter’s authors

This work is licensed under an Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). This license allows you to share, copy, distribute and transmit the text; to adapt the text for non-commercial purposes of the text providing attribution is made to the authors (but not in any way that suggests that they endorse you or your use of the work). Attribution should include the following information:

SJ Beard and Tom Hobson (eds), An Anthology of Global Risk. Cambridge, UK: Open Book Publishers, 2024, https://doi.org/10.11647/OBP.0360

Copyright and permissions for the reuse of many of the images included in this publication differ from the above. This information is provided in the captions and in the list of illustrations. Every effort has been made to identify and contact copyright holders and any omission or error will be corrected if notification is made to the publisher.

Further details about CC BY-NC licenses are available at http://creativecommons.org/licenses/by-nc/4.0/

All external links were active at the time of publication unless otherwise stated and have been archived via the Internet Archive Wayback Machine at https://archive.org/web

Digital material and resources associated with this volume are available at https://doi.org/10.11647/OBP.0360#resources

ISBN Paperback: 978-1-80511-114-6

ISBN Hardback: 978-1-80511-115-3

ISBN Digital (PDF): 978-1-80511-116-0

ISBN Digital eBook (EPUB): 978-1-80511-117-7

ISBN XML: 978-1-80511-119-1

ISBN HTML: 978-1-80511-120-7

DOI: 10.11647/OBP.0360

Cover image: Javier Miranda, Alien planet, June 18, 2022, https://unsplash.com/photos/nc1zsYGkLFA

Cover design: Jeevanjot Kaur Nagpal

Contents

Introduction

I. History, Concepts, and Norms

1. Ripples on the Great Sea of Life: A Brief History of Existential Risk Studies

SJ Beard and Emile P. Torres

2. Democratising Risk: In Search of a Methodology to Study Existential Risk

Carla Zoe Cremer and Luke Kemp

3. Classifying Global Catastrophic Risks

Shahar Avin, Bonnie C. Wintle, Julius Weitzdörfer, Seán S. Ó hÉigeartaigh, William J. Sutherland and Martin J. Rees

4. Governing Boring Apocalypses: A New Typology of Existential Vulnerabilities and Exposures for Existential Risk Research

Hin-Yan Liu, Kristian Cedervall Lauta and Matthijs Michiel Maas

5. Existential Risk, Creativity and Well-Adapted Science

Adrian Currie

II. Methods, Tools, and Approaches

6. An Analysis and Evaluation of Methods Currently Used to Quantify the Likelihood of Existential Hazards

SJ Beard, Thomas Rowe and James Fox

7. Scanning Horizons in Research, Policy and Practice

Bonnie C. Wintle, Mahlo N. C. Kennicutt II and William J. Sutherland

8. Exploring Artificial Intelligence Futures

Shahar Avin

9. Accumulating Evidence Using Crowdsourcing and Machine Learning: A Living Bibliography About Existential Risk and Global Catastrophic Risk

Gorm E. Shackelford, Luke Kemp, Catherine Rhodes, et al.

10. The Mortality of States (MOROS) Dataset

Luke Kemp

11. Enabling the Participatory Exploration of Alternative Futures With ParEvo

Rick Davies, SJ Beard, Tom Hobson and Lara Mani

III. Risk Drivers and Impacts

12. Global Catastrophic Risk From Low Magnitude Volcanic Eruptions

Lara Mani, Asaf Tzachor and Paul Cole

13. Re-Framing the Threat of Global Warming: An Empirical Causal Loop Diagram of Climate Change, Food Insecurity and Societal Collapse

C. E. Richards, R. C. Lupton and J. M. Allwood

14. Existential Change: Lesson from Climate Change for Existential Risk

SJ Beard and Luke Kemp

15. A Fate Worse Than Warming? Stratospheric Aerosol Injection and Catastrophic Risk

Aaron Tang and Luke Kemp

16. Bioengineering Horizon Scan 2020

Luke Kemp, Laura Adam, Christian R Boehm, et al.

17. Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI

Carla Zoe Cremer and Jess Whittlestone

IV. Policy, Institutions, and Impacts

18. Pathways to Linking Science and Policy in Global Risk

Clarissa Rios Rojas, Catherine Richards, Catherine Rhodes and Paul Ingram

19. The Cartography of Global Catastrophic Governance

Catherine Rhodes and Luke Kemp

20. The Stepping Stones Approach to Nuclear Disarmament Diplomacy

Paul Ingram

21. It Takes a Village: The Shared Responsibility of “Raising” an Autonomous Weapon

Amritha Jayanti and Shahar Avin

22. Representation of Future Generations in United Kingdom Policy-Making

Natalie Jones, Mark O’Brien and Thomas Ryan

23. Financing Our Final Hour

Luke Kemp, Haydn Belfield, Ellen Quigley, Julius Weitzdörfer and SJ Beard

Contributors

Index

Introduction

© 2024 SJ Beard and Tom Hobson, CC BY-NC 4.0 https://doi.org/10.11647/OBP.0360.00

Would that a lion had ravaged mankind; rather than the flood,Would that a wolf had ravaged mankind; rather than the flood,Would that famine had wasted the world; rather than the flood,Would that pestilence had wasted mankind; rather than the flood

—

The Epic of Gilgamesh

, tablet 11

Humans have been living under the shadow of global catastrophe for a very long time. For most of our history, the risk of catastrophe was understood to be supernatural, serving to make it more threatening and fearsome. By some accounts the expected global catastrophe would strike down only the sinful and wicked and bring the blessed to a new life in a better world. By other accounts it may have been seen as part of the natural cycle of life, a cosmic extension of the cycles of birth and death, spring and autumn, rise and fall. Invariably global catastrophes were not the final word for humanity. Although global catastrophes have never been the final word for humanity, always accompanied by promises of salvation and renewal, they nevertheless were maintained as awesome prospects to be feared.

This anthology centres on very different kinds of risk: naturalistic, disastrous, and potentially final calamities. However, that does not mean we should not be concerned with these tales from our past. Old myths about the world’s end (from the Christian apocalypse to the Norse-pagan Ragnarök) remain key touchstones for our society, culture, and even politics. Perhaps the most influential of all of these myths is also the oldest and most universal — the deluge. The story of a great flood that was once sent to Earth by an angry god or gods to wipe out humanity is a story told by many cultures around the world. Usually, one human being is forewarned of this impending disaster, however, and is able to escape by building a boat to carry him, his family, and some selection of plants and animals to repopulate the earth. This story has been found across the Mediterranean Basin and throughout South and Southwest Asia, with comparable stories being told in many other parts of the world. For readers of this anthology, the story might be most familiar by the role it plays in major world religions, including Judaism, Christianity, and Islam (as the story of Noah) and Hinduism (as the story of Vaivasvata Manu). It remains a very popular story for young children all over the world, and is almost certainly the most common introduction most people reading this book will have received to the idea of a global catastrophic event. Yet it is a story that predates all of these religions, with the earliest recorded versions (like the one quoted as part of the Epic of Gilgamesh above) dating back over 4,000 years. It may even be the very oldest story to have been passed down to us today.

So, what does this ancient myth tell us about global catastrophes? We are told that the world was nearly destroyed by a single disastrous event (the flood), caused by an exogenous force (the gods), but which happened as a direct result of the faults and failings of humanity. In the story of the great flood, humanity survived because one individual was granted foreknowledge of this catastrophe and was able to take action to save themselves as well as a sufficient number of people, plants, and animals to repopulate the world.

It may be that the story originated in experiences with catastrophic flooding in the river valleys where early civilisations tended to form (and that in places where such flooding was less common, such as in Eastern Iran, the story would survive but with a different agent of disaster, such as a hard winter). It might also be the case that the story reflects the religious sensibilities of the age, in which centralised religion demanded increasingly strict adherence to its laws and requirements on pain of divine punishment. The fact that a story can be so widespread, culturally established in our oral tradition through decades of retelling, suggest that — regardless of origin — certain elements make for a narrative that can withstand the test of time.

It is for this reason that the anthology begins with a discussion of the flood myth. Clearly a compelling story, its elements have been reproduced time after time when we come to think about the end of the world, from speculation about an AI that, due to the imprudent haste with which it was developed, is indifferent to human values and thus chooses to eliminate us, to the hope that we might survive a nuclear or volcanic winter in a bunker or on an island refuge; from a tendency to talk about climate change as an exogenous force that is punishing humanity for our misdeeds, to a desire to predict exactly what kind of biological catastrophe is most likely to bring about a global catastrophe. Our vision of extreme global risks in the early 21st century seems to eerily mirror the stories of our ancestors, even when translated through our present-day claims to rationality and objectivity.

Stories serve to pass down knowledge, ideas, and judgements about how the world is and what it might become; indeed, as Chapters 1 and 8 of this volume describe, they have played, and continue to play, important roles in the development of Existential Risk Studies. However, they also serve as sense-making tools, providing ways to interpret the world around us, its immutability or transience, and the futures we might aspire to or fear. The ability to tell stories, or at least the ability to propagate them and have others listen, is also bound up in social relations and takes place within the material contexts of a given historical moment. Stories do not emerge, fully formed, into the world. Stories are told, heard, retold. Their narratives are reshaped and their endings reimagined. The evolution of the flood narrative over time should also be understood as being shaped by the social relations from which each successive iteration emerges.

To be clear, this does not mean that the resulting ideas are misguided or misinformed. However, it does mean that we should approach them with due care, knowing that we ourselves have been shaped by ancient myths which give meaning and power to certain world perspectives. It is quite possible to tell very different stories about global catastrophes: stories in which humanity is damaged by long-term, slow-moving processes that are endogenous factors in our socio-technological systems, arising from blameless aspects of human nature, or stories in which survival is achieved via a broad awareness of many possible disasters, causing us to increase resilience for all of humanity. It is just that these are not such good stories, and they are never going to capture people’s attention in the same way.

The chapters in this volume all contribute to the development of a truly secular approach to extreme global risk, in that they show how we can make significant advances in understanding and managing risk, as well as how we can challenge traditional catastrophe narratives, and create new ones to fit the evidence we are gathering.

To begin with, we can broaden the ways in which we think about extreme global risk. Chapter 1, Ripples on the Great Sea of Life, examines the history of how our understanding of this risk has developed over time. The first naturalistic accounts of human extinction and other global catastrophes came from artists and speculative fiction authors looking for new and interesting stories to tell about the end of the world. However, as science and technology developed rapidly through the 19th and 20th centuries, an increasing range of scientists expressed concern that this was a real possibility coming our way. What drew these diverse concerns together, at the dawn of the 21st century, was initially a group of transhumanists who feared that uncontrolled advances in artificial intelligence threatened not only the realisation of their own vision of technological utopia, but also the very survival of humanity. This prompted the establishment of an interdisciplinary community of researchers who saw their goal as charting a safe passage through this “time of terrors” without triggering an existential catastrophe whilst still advocating for further research into artificial intelligence and other technologies so that they might reach the end state they desired. This initial group has been enlarged and diversified by subsequent events, most notably the emergence of the Effective Altruist movement as a substantial source of both additional resources and researchers, and the entrance of, and engagement with, researchers from outside of this community who agreed that extreme global risk was an important problem, if not for the same reasons. The legacy of this history can still be seen in many aspects of the field. Existential risk research still maintains an (arguably disproportionately) strong focus on hazards that could emerge from technologies that many people in the field also see as highly worth developing like AI and biotech; and much of the research in the field remains guided by a common set of ethical and epistemological commitments underpinned by ethical consequentialism and Bayesian epistemology, even though these are not directly related to existential risk.

The remaining chapters in Section 1 all grapple with, build on, and challenge this legacy in a variety of ways. A common theme among these chapters is the need to move away from the most direct and straightforward kinds of existential catastrophe (the naturalistic equivalents of Noah’s flood) and towards complex risk assessment that considers a far wider range of possibilities and factors. Chapter 2, Democratising Risk, offers a critique of the original paradigm of Existential Risk Studies, what it refers to as the Techno-Utopian Approach. It argues that it is elitist and methodologically limited, so should be replaced with new, more participatory and democratic ways of thinking, which focus instead on complex risk assessment and are transparent about their commitments. Chapter 3, Classifying Global Catastrophic Risk Scenarios, provides a framework that helps to meet some of these goals by understanding global catastrophe scenarios from a systemic perspective, moving away from individual scenarios in order to consider convergent risk factors, including systemic interdependence and mitigation fragilities. Chapter 4, Governing Boring Apocalypses, provides a complementary framework that rejects a hazard-centric approach to risk, and moves towards considering vulnerabilities (i.e. aspects of humanity and the systems we rely on that make us susceptible to being harmed by hazards) and exposures (i.e. the ways in which hazards and vulnerabilities come into connection with one another); not merely to better understand the full nature of risk, but also because these often provide additional mitigation opportunities. Finally, Chapter 5, Existential Risk, Creativity, and Well-Adapted Science, asks fundamental questions about what kind of science is best suited to studying extreme global risk. The chapter makes a strong case that Existential Risk Studies needs to be creative, in the sense of exploring a wide range of hypotheses, rather than seeking to exploit a smaller range of more likely hypotheses, and that as a field it operates within incentive structures that tend to push science towards being more conservative. Countering this in order to achieve the kind of science that is best adapted to its purpose requires exactly the kind of reflexive work that the chapters in this section, and elsewhere in the volume, set out to provide.

Section 2 turns from broad questions about the nature of Existential Risk Studies as a field to consider the methodologies, tools, and approaches for studying it. Some key themes from these chapters include a focus on the value of rigorously implementing methodologies rather than jumping to judgement, even if this is well informed, and the importance of making use of foresight tools that explore a wide range of possible futures rather than trying to forecast the most likely or dangerous among these. As existential risk researchers we have found that one of the most common questions we get asked is ‘what should we be most worried about?’, following the Noah narrative that successfully surviving a global catastrophe requires us to predict exactly what it is going to be. However, these methodologies provide far more expansive and inclusive ways of studying extreme global risk that avoid this way of thinking entirely.

The first three chapters in this section survey a wide range of different methodologies that can be applied within Existential Risk Studies. Chapter 6, An Analysis and Evaluation of Methods Currently Used to Quantify the Likelihood of Existential Hazards, provides a wide-ranging survey and evaluation of methods that have been used for the quantification of risk. It argues that there is no perfect methodology in the field but that it could benefit from a greater degree of methodological pluralism. More importantly, however, the chapter also argues that methodologies need to be applied more transparently and rigorously in order for researchers to engage critically with the limits and interpretations of whatever methods they are using. Chapter 7, Scanning Horizons in Research, Policy, and Practice, provides a more focused survey of horizon-scanning techniques. These are structured expert elicitation techniques that both combine information from diverse communities of practice and allow these same communities to sort, verify, and analyse this information to produce better collective judgements, generally aiming at identifying emerging threats, issues, and questions for further research. Chapter 8, Exploring Artificial Intelligence Futures, focuses on different methods, that are accessible to researchers from the humanities, for exploring futures of AI. These range from engaging with science fiction and the work of individual disciplines such as philosophy, economics, and risk analysis to participatory methods for bringing diverse groups together. The chapter argues that there is significant potential for more work to be done on the formation and use of participatory role-play scenario tools in particular.

The final three chapters in this section turn to describing three specific methodological tools that have been developed or improved by scholars in this field to better study extreme global risk. Chapter 9, Accumulating Evidence Using Crowdsourcing and Machine Learning, describes the creation of TERRA, a semi-automated literature review tool designed to expand the evidence base for Existential Risk Studies. Chapter 10, The Mortality of States (MOROS) Dataset, provides an example of using historical data about the lifespan of political states, MOROS, to study societal collapse and to better understand political institutions that are highly relevant for extreme global risk. Finally, Chapter 11, Enabling the Participatory Exploration of Alternative Futures, discusses the ParEvo technique, which enables groups to participate in the construction, exploration, and evaluation of divergent narratives about different possible futures using an evolutionary process.

Section 3 provides examples of how these developments in Existential Risk Studies have been used to produce new insights about the causes and consequences of extreme global risk. The chapters provide insights on a range of risk drivers, from volcanoes to AI. However, it is suggested that these should not be understood as exogenous factors that are out there trying to get us, but simply as the result of processes we are currently struggling to understand.

For instance, Chapter 12, Global Catastrophic Risk From Low Magnitude Volcanic Eruptions, argues that traditional accounts of Global Catastrophic Volcanic Risk focus too much on the explosivity of potential future eruptions. However, the relationship between the size of an eruption and the amount of damage caused is neither straightforward or linear. By plotting active volcanoes alongside critical global infrastructure such as manufacturing and transportation pinch points, the chapter shows how we are especially vulnerable to volcanic eruptions in particular localities, due to the placement of key infrastructure in areas where eruptions could easily damage or disrupt it. Hence, it is possible for even a relatively low explosivity volcanic eruption from the right/wrong volcano to cause harm at the global scale. Chapter 13, Re-Framing the Threat of Global Warming, looks at the risk from climate change and provides an empirical evidence base for studying how this could be mediated through food insecurity and societal collapse. By conducting an extensive literature review, the chapter constructs an empirical causal loop diagram that describes the systemic cascades that could be triggered by future climate change, and that are created by the ways we have designed national and international institutions and systems around current climatic expectations. Chapter 14, Existential Change, builds on this with a theoretical exploration of what Existential Risk Studies can learn from climate change more broadly, highlighting how the tendency to ask questions such as ‘is climate change an existential risk?’ misunderstands the nature of risk and fails to learn lessons from other researchers, who have studied climate change, about its likely effects. As a result, we need to move away from thinking about “climate change” as a single force and towards thinking through a diversity of different climate scenarios. Chapter 15, A Fate Worse Than Warming?, turns to consider one of the elements of future climate scenarios: the potential use of Stratospheric Aerosol Injection, a technology that injects sulphates into the upper atmosphere, deflecting sunlight and providing a global cooling effect. This has been touted as a possible means of mitigating risks from climate change but it is a risky technology in its own right, and the chapter assesses what these risks are, what we know about them, and how they might be weighed against the potential benefits. The chapter concludes that it is unlikely that we can conclusively ever say whether the Stratospheric Aerosol Injection is “good” or “bad” as so much will depend upon other features of any scenarios in which it is deployed. Chapter 16, Bioengineering Horizon Scan 2020, uses horizon-scanning techniques like those described in Chapter 7, to look at emerging issues in bioengineering. It identifies 20 issues including technological, societal, and governance changes that could emerge over a range of time spans and are of highest priority for further research. Finally, Chapter 17, Artificial Canaries, shows how we can combine a variety of methods to identify early warning signs (or “canaries”) that Artificial Intelligence may be on the brink of increased transformative potential. These take account of both what experts currently know about the possibilities for future AI and where there is currently most uncertainty, so that the warning signs give sufficient room for anticipatory governance frameworks to be put in place to manage this transformation for good rather than ill. By focusing on a broad concept of what transformative AI might be like and how we can learn more about what kind of future trajectory we might be on, this chapter once again highlights the importance of exploring different possible futures and understanding how we continue to shape these through present and future choices.

Finally, Section 4 considers mechanisms for reducing the level of extreme global risk by improved policy-making. These chapters are grouped according to their shared concerns with shaping policies, institutional behaviours, and governance priorities. They take a variety of approaches to undertaking this task, emphasising dialogue, collaboration, equity in representation, and the importance of linking policy-making to scientific expertise. However, they are all clearly focused on prevention, rather than survival, and on spreading power to more people who can use it to collectively achieve common goals, not prioritising the interests of elite latter-day Noahs. The chapters also pose important questions about how we might go beyond reactive engagements with risks from hazards that are considered as already imminent or intrinsic, to instead proactively fostering social, political, and economic conditions more amenable to human and planetary survival.

Chapter 18, Pathways to Linking Science and Policy for Global Risk, proposes that engagement with policy-making is a necessary and core component of existential risk research, as an action-oriented discipline. The chapter provides an overview of some of the policy shaping work undertaken by researchers at CSER and highlights promising approaches that scholars might take in the future. Chapter 19, The Cartography of Global Catastrophic Governance, takes a more macro-level approach to charting the efficacy and concentration of different GCR governance efforts, proposing a typology that allows for comparison based on risk focus, institutional arrangement, and effectiveness of implementation, highlighting the gaps scholars are best positioned to fill. This chapter provides a map of governance efforts for different GCRs at the time of writing, whilst additionally presenting an analysis of what kinds of action might serve to best increase resilience to GCRs, even in the face of complexity and uncertainty. The remaining chapters focus on more specific contexts, ranging from national policy and institutional design to international diplomacy and private sector investment. Chapter 20, The Stepping Stones Approach to Nuclear Disarmament Diplomacy, marks another change of focus and level of analysis as the author provides a reflective account of efforts to build dialogue, and embraces the potentialities for radical change that might be catalysed by even modest incremental improvements in diplomatic relations towards nuclear disarmament. Chapter 21, It Takes a Village: The Shared Responsibility of Raising an Autonomous Weapon, considers a specific policy area, defence policy and military procurement towards Lethal Autonomous Weapons Systems (LAWS). It explores how this can be improved by simulating an inquiry that might take place following a LAWS-initiated fatality, and uses the results to show how narrow policies shaped by restrictive notions of “human control” are likely to be insufficient to govern these systems. In Chapter 22, Representation of Future Generations in United Kingdom Policy-Making, the focus shifts towards representation and we are prompted to consider how, and why, we might seek to ensure equitable representation of future generations in the national policy-making processes of today. The final chapter of this volume, Financing Our Final Hour, provides readers with an empirically grounded analysis of how different modes of pressure and advocacy can influence institutional investors to take seriously the responsibility they have to people and the planet to reduce the re-production of catastrophic hazards in their investment practices.

These chapters also serve to further emphasise the point that extreme global risks, and the means of reducing or preventing them, are never ex machina. Rather, they are shaped through an ongoing process of interactions: interpersonal, international, and technological relationships come to the fore in the sections’ analyses of how researchers might shape policy and practice in this field. For many of us, these processes can seem very remote, and it is important not to forget how concentrated much of the power over risky scientific, technological, and economic development really is. However, these chapters prove that many of us are already enmeshed within institutions, from parliaments to pension funds, that have the power to influence them. These chapters also promote us to think positively about what better institutions and policies might look like. For instance, Chapter 20’s stepping-stones approach starts by drawing on radical visions of how security could be achieved without weapons of mass destruction, while Chapter 23 makes a strong case that large institutional investors, known as universal owners, have strong ethical, legal, and financial reasons to reconceptualise themselves as responsible stewards of the entire economy, and should use their power for collective goods like the reduction of global risk.

Together, the prospect of distributed power and responsibility, and reflection on positive possibilities for how existing institutions could be used in the service of creating positive futures, opens the door to very different ways of thinking about humanity’s 21st-century predicament. We are not only living in an age of extreme global risk, including existential risk to the future of humanity, but also living with the possibility of existential hope. That humanity may be heading for extinction is of very limited interest in the cosmic scheme of things, as extinction is ultimately the fate of all species. However, if we rise to the challenges of our age then it is possible that humanity may be the first species in the long history of our planet to have created the conditions for our own extinction and then chosen to do something else. That seems like a project worth pursuing. As Martin Luther King Jr famously put it in his final speech, the night before his assassination in 1968:

And another reason that I’m happy to live in this period is that we have been forced to a point where we’re going to have to grapple with the problems that men have been trying to grapple with through history, but the demands didn’t force them to do it. Survival demands that we grapple with them.

1

We do not wish to claim that Existential Risk Studies has yet earned the right to say that we are delivering Dr King’s dream of a world in which people truly face up to the reality of such problems. However, we do share his view that one can be happy to live in a time when the ancient fears of global catastrophes may finally be leading us to at least think about how this work may be done.

Our contention is that this anthology signals something special, the establishment of an entirely new field of study, Existential Risk Studies, and we hope that the chapters within it, and the conversations between them, show how this field is developing. The chapters engage with an issue that is of great concern to many and examine its meaning and foundations, developing methodologies to study it responsibly, revealing new insights about its nature and impacts, and advocating for meaningful change to make it less concerning. They show how we are moving away from the speculative and alarmist and towards the proactive, rigorous, transparent, and accountable. In doing so, it is suggested by the authors that we can move away from the deep myths that have defined our past, towards a creative and engaged science that can help us build a better future.

1 King Jr, M. L. ‘I see the promised land’, in J. M. Washington (ed.), A Testament of Hope: The Essential Writings and Speeches of Martin Luther King, Jr. HarperOne (1986), pp. 279–86.

I. History, Concepts, and Norms

The chapters in this volume all take as their subject the study of extreme global risks, in most cases extreme to the point of involving either a global or existential catastrophe. As the first chapter in this section, Ripples on the Great Sea of Life: A Brief History of Existential Risk Studies by Beard and Torres, suggests, this is not a particularly new research area. Existential risk has been a subject of speculation and research going back at least into the 19th century. However, the notion of a transdisciplinary field of Existential Risk Studies, and the growth of a research community dedicated to it, are far more recent developments. The chapters in this volume emerge from this nascent area of research, and represent many of the key debates in the field. Indeed, this contributions to this volume show, perhaps above anything else, that the key concepts and norms of this community are still very much points under discussion.

To open this volume, therefore, we offer five chapters that provide a range of sympathetic perspectives on this emerging field and how it is developing. It is possible to see these chapters as providing the basis for their own paradigm of existential risk research and/or to see them as substantially critiquing certain views within the field. There are certainly points on which they all agree; such as the importance of systemic risk and looking beyond the most direct and explosive forms of catastrophe, the need for pluralism and interdisciplinarity, and the importance of integrating risk assessment and mitigation in research. However, we hope that by bringing them together the points of divergence and discussion between these chapters becomes clearer, so that the reader can more easily appreciate that they are united not be a shared idea of what existential risk is and how to reduce it, but by a shared commitment to more expansive, open, and reflexive ways of working to answer that question.

The first chapter of this section provides a historical overview of the study of existential risk. It presents an account of the ways existential risk was thought about during the 19th, 20th, and 21st centuries and proposes a possible genealogy of the emerging field of Existential Risk Studies. This chapter emerged from a desire by the authors, one of whom is co-editor of this volume, to see a history of this field written, not simply as an intellectual exercise but also a way of helping the field develop and improve. The authors envisioned their history as making three main contributions: 1) a way of introducing new researchers to the breadth of ideas within the field, and where they had come from, 2) an opportunity to learn lessons from how this work has been done in the past that could be applied to doing it better in the future, and 3) to inspire people to think about how the field might be developing and what its possible future trajectories could be. That the work was successful in fulfilling these objectives is perhaps best demonstrated by the scale and scope of further discussions it has been taken as a starting point for. Scholars — including Thomas Moynihan,1 Daniel Zimmer,2 Apolline Taillandier,3 and Matthew Connelly, among others — have all made contributions to an evolving dialogue, painting different pictures of the trajectory of this field, allowing for greater learning and stimulating more ideas about where it may be heading.

In this particular history the trajectory of the field is told in terms of three successive “waves” of development. The first of these, which occurred largely during the 2000s, saw a small group of researchers conceptualise the idea of existential risk from a specifically transhumanist perspective. These researchers, such as Nick Bostrom and Eliazer Yudkowsky, were committed to bringing about a transition to a post human state that they felt would be better than our present condition but also argued that, along the way, the technologies we were developing were creating risks with the potential to threaten both the lives of presently existing human beings and the potentiality of this “ideal” future. The field of Existential Risk Studies can be seen, in many ways, as these researchers both seeking to produce a coherent research agenda for understanding and mitigating these specific threats as well as to place their work in dialogue with that of many concerned scientists and technology developers who had long worried about humanity’s potential to wipe ourselves out. A second wave emerged in the late 2000s and early 2010s, when these researchers sought to increase the power and impact of their field by engaging with broader communities while, at the same time, the burgeoning Effective Altruism movement, whose aim was to identify how individuals could produce the greatest quantity of value with their actions, started to wonder about existential risk mitigation as a priority cause. These two movements saw the field expand and diversify rapidly and lead to ethics, and in particular the forms of consequentialist ethics most common among Effective Altruists, becoming deeply embedded in the methods and concepts of the field. Finally, a third wave has emerged since the late 2010s, in which this expanded field has been influenced by the diverse perspectives of new researchers, many of whom are not interested in, or are even actively hostile to, transhumanism and/or consequentialist ethics and who have also introduced new methods, approaches, and perspectives, many of which relate to systems thinking and complexity.

The second chapter, Democratising Risk by Crèmer and Kemp, considers some of the same issues but moves from a historical analysis to a reflection on what the present situation of Existential Risk Studies implies about the field’s strengths and weaknesses. In particular, the chapter critically assesses the legacy of pioneering work in the field that shared transhumanist, utilitarian, and longtermist assumptions. The authors characterise this as the Techno-Utopian Approach. One of the implications this has had for the field is that standard definitions of existential risk are concerned less with death or harm suffered by people in a global catastrophe, but rather more with the significant loss of “value” (according to a particular set of assumptions about what is valuable) that this would represent, not merely in terms of actual harm but especially quantity of potential future value in future lives that would never come into existence as a result. The authors argue that a definition like this is problematic because it will invariably tie our understanding of existential risk to the value system of a particular group of people and that this is both philosophically tenuous — for a field that claims to speak on behalf of humanity as a whole — and practically dangerous, in that it carries the potential of justifying the values of this group being imposed on everyone else. They thus argue that the field should separate work on existential and extinction ethics (what is good or bad in relation to existential risk and the future of humanity, how to prioritise work in this area, how to make decisions under uncertainty, and other related questions) and the study of human extinction and other global catastrophes. They also point to a range of other methodological weaknesses in the Techno-Utopian Approach, including a techno-deterministic view of the future and a focus on simplistic, threat based, models of risk assessment and mitigation. Furthermore, they argue that these features of the field carry the potential to generate negative consequences and “response risks” if its recommendations were to disproportionately influence future decision makers. As a result, they call for a diversified and democratised field that is open to a wider range of assumptions and approaches and seeks to both listen to, and engage with, a far wider community of stakeholders.

While both these chapters present a somewhat critical engagement with the history and possible future trajectories of the field of Existential Risk Studies, there are a number of points of divergence between their analyses. For instance, while the first views Existential Risk Studies as presently evolving through a process of diversification and maturation, the second argues that there are important forces currently seeking to stifle this and ensure that the field does not deviate too far from the ethical frameworks it was founded around. Similarly, the first chapter argues that certain individuals, as well as the field as a whole, can be seen as transitioning between the various waves of existential risk research (indeed, in some ways, the three waves can be seen as typified by three papers from Nick Bostrom in particular — Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards,4Existential Risk Prevention as Global Priority,5 and The Vulnerable World Hypothesis6 — although each wave, and especially the third, go far beyond the work of this one prominent scholar), the second views different perspectives within Existential Risk Studies as far more bound up with the interests of competing groups and argues for the necessity of democracy and justice to rectify this. These, and other differences, should definitely not been seen as requiring there to be a right or wrong answer about the nature of the field; however, they point to different understanding of it, and we invite readers to reflect on this and draw their own conclusions.

The remaining three chapters in this section move from a consideration of the field of Existential Risk Studies to thinking about existential (and other forms of extreme global) risk and how to study and prevent them. Chapter 3, Classifying Global Catastrophic Risks, by Shahar Avin and a team of (then current) CSER researchers, sought to provide an analytical framework for thinking about different kinds of global catastrophe scenario. Rather than drawing upon people’s immediate thoughts about what kinds of mechanism could bring about a global catastrophe, it approaches the subject by thinking about the nature of such catastrophes themselves. It shows that in all cases a global catastrophe involves at least one critical system on which humanity depends being pushed beyond the safe limits for supporting our survival and flourishing, one or more spread mechanisms that would cause this effect to be experienced globally (and or prevent it being contained locally) and, crucially, the fragilities in human decision making that caused us to fail to prevent this from happening. This framework serves as both a methodology for thinking clearly about how particular global catastrophe scenarios, from pandemics or volcanic eruptions to environmental or technological catastrophes, might play out, whilst also understanding all of the factors that might trigger such a catastrophe, and hence the avenues we might have for preventing it. This chapter has been foundational to much of the contemporary thinking about the systemic components of Global Catastrophic and existential risk, at the CSER and elsewhere.7

Chapter 4, Governing Boring Apocalypses by Hin-Yan Liu, Kristian Cedervall Lauta and Matthijs Michiel Maas, provides a complementary analytical framework for thinking about the drivers of risk (rather than the catastrophes that might emerge from it). The chapter argues that when people think about the causes of Global Catastrophic Risk they often focus only on the “hazard” or “threat” that precipitated the catastrophe, such as an asteroid, pathogen, or unaligned Artificial Intelligence. While hazards are important however, decades of work in disaster studies has shown that they are not the only drivers of risk. For a catastrophe to occur, a hazard needs to be combined with two other features: a vulnerability (a factor that makes humanities subject to the harm this hazard might cause) and an exposure (the medium by which the hazard and vulnerability meet). For instance, earthquakes, a quintessential hazard, seldom kill people on their own. The harm that earthquakes do comes about because we create a vulnerability by building and living around structures that an earthquake might cause to collapse upon us, and also because we expose ourselves by doing this in areas where earthquakes happen.8 The chapter goes on to provide a complete classification of existential vulnerabilities and exposures that, once again, both helps to explore the nature of risk drivers that are already being examined within the field, as well as drawing attention to the possibility of discovering new risk drivers that have not yet been considered. However, perhaps even more importantly, the authors point out that a framework like this not only helps us think about new ways in which things might go wrong but also, by extension, to understand the full suite of tools at our disposal for preventing them. In particular, they argue that once a full inventory of vulnerabilities and exposures has been produced, we can appreciate that technological solutions are not only far from the sole options at our disposal for preventing existential risk but that they might also be harmful. For instance, they can feed into cultural vulnerabilities and exposures, reduce societies’ resilience, and also risk breeding a false sense of security; we take comfort in a few easy solutions (such as rapid vaccine development) and this causes us to lose sight of more complex problems (such as inadequate provision of public health).

It is notable that these two chapters first appeared at the same time, and they may appear to be offering the same kind of output, an analytical framework for thinking more expansively about existential and Global Catastrophic Risk. However, they are not the same, nor are they in competition with one another. For many years, people in Existential Risk Studies have talked about existential risks, labelling things such as AI, biorisk, climate change, and nuclear war. However, while these things might be (usefully at times) labelled and understood as “risks”, they are also technologies, processes, trends, and events. These two chapters do not seek to classify different risks but global catastrophe scenarios (the catastrophic events themselves) and the drivers of risk (the processes and phenomena that come together to precipitate events such as this). It is important to note that these two things cannot be matched up on a one-to-one basis; for instance in Classifying Global Catastrophic Risk Scenarios the authors note that the global catastrophe of a darkening of Earth’s atmosphere leading to mass starvation could equally be caused by a range of mechanisms, from an asteroid impact to a nuclear war. Similarly, Governing Boring Apocalypses highlights how the same kinds of vulnerability and exposure (such as just in time food delivery or short-term political decision making) can be involved in precipitating many kinds of catastrophe. We therefore need to apply both of these frameworks in order to fully understand both the drivers of risk and catastrophe scenarios, but also to understand that in doing so we are assessing only one, more or less unified, phenomena of existential and Global Catastrophic Risk, not many distinct existential and Global Catastrophic Risks. This is a conceptual innovation that is still developing within the community.

The final chapter in this section, Existential Risk, Creativity and Well-Adapted Science, by Adrian Currie, looks at the kind of science that would be best suited to studying existential and Global Catastrophic Risk. The chapter considers many of the questions raised by previous chapters, such as the value-laden nature of science and the difficulty of studying the interaction of complex systems, and argues that for a field such as Existential Risk Studies to be well-adapted to this situation, it is important for researchers to show a high degree of creativity, raising, exploring, and testing many different kinds of solution, rather than the conservative strategy of trying to identify only those solutions most likely to succeed and not exploring more widely. In this aim, however, the chapter notes that the field runs up against many different features of science that currently work against precisely this kind of creativity. Achieving more creativity requires the field to be multi-disciplinary, pluralistic, and opportunistic, but most of all it requires researchers within the field to identify the sources of maladaptation and ask which of these we might do something about. This means that we need a creative engagement with both existing norms and practices in the field, and also with those taken as best practice within science more broadly, from competitive peer-reviewed funding to the institutionalisation of scientific research. We cannot safely assume that the norms and incentives we are developing will provide the field with the creativity it needs to succeed at its aims.

Clearly, this chapter points us right back to the beginning of this section, and the need to interrogate what the field is and how it came to be this way. However, we hope that it also provides an opening into thinking about the remaining chapters of this book, in which we turn from engaging with the field to thinking more about the problems it aims to solve. In the next section we will look at a variety of methods, tools, and approaches that have been developed by researchers in this community to study existential and Global Catastrophic Risk. In their own way, all of these seek to promote more creative work while also upholding standards of transparency and rigour. However, in moving forward from the groundwork of this section, we suggest that readers proceed to the next section, and indeed to any other work in this field, with some key reflective questions to hand. For instance:

How did this method, tool, approach, or idea come to be, what community created it and what assumptions, norms, and values shaped it, are there other versions of the same method, tool, approach, or idea that might reflect different ways of thinking about the same problem?

How do these ways of thinking about existential and Global Catastrophic Risk fit with particular individuals, institutions, or paradigms and the power that they hold? Do they play a role in concentrating resources (not merely economic but also social, cultural, and epistemic) or distributing them more fairly?

Does this method, tool, approach, or idea allow for a complete analysis of all aspects of a problem, or does it tend to focus on one element that is most readily observed or easy to study and, if so, what is left out?

How can this idea be used not merely to understand risks better but also to manage them?

Is this a helpful way of exploring the maximum possible range of possibilities or ideas? Is it a way of exploiting one subset of possibilities and ideas most fully? If it makes trade-offs between these two, how does this work and who gets to decide?

Questions such as these can easily be dismissed as naval gazing by those who wish to jump straight into thinking about a problem. However, one thing that we take from the chapters of this section is that this is a mistake. While reflection can be difficult, and critique can be even more so, understanding the context in which one works, and how one’s work fits with the larger field of Existential Risk Studies, is of vital importance if we are to make sure that the community as a whole is finally making progress towards the long-held desire to ensure the safety of humanity, now and in the future.

Notes and References

1 Moynihan, Thomas. X-Risk: How Humanity Discovered Its Own Extinction. MIT Press (2020).

2 Zimmer, Daniel. Essence, Process, System: Human Extinction in Political Thought from Aristotle to the Anthropocene (in preparation).

3 Taillandier, Apolline. ‘From boundless expansion to existential threat: Transhumanists and posthuman imaginaries’, in Sandra Kemp and Jenny Andersson (eds), Futures. Oxford University Press (2021).

4 Bostrom, Nick. ‘Existential risks: Analyzing human extinction scenarios and related hazards’, Journal of Evolution and Technology, 9 (2002).

5 Bostrom, Nick. ‘Existential risk prevention as global priority’, Global Policy, 4(1) (2013): 15–31.

6 Bostrom, Nick. ‘The vulnerable world hypothesis’, Global Policy, 10(4) (2019): 455–76.

7 However, it is worth noting that the chapter itself builds upon previous work, notably the planetary boundaries framework for thinking about anthropogenic environmental disruption and the Boundary Risk for Humanity in Nature framework developed by Seth Baum and Itsuki Handoh, who brought this concept into dialogue with ideas about Global Catastrophic and existential risk. See Rockström, Johan, Will Steffen, Kevin Noone, Åsa Persson, F. Stuart Chapin III, Eric Lambin, Timothy M. Lenton et al. ‘Planetary boundaries: exploring the safe operating space for humanity’, Ecology and Society, 14(2) (2009); Baum, Seth D. and Itsuki C. Handoh. ‘Integrating the planetary boundaries and global catastrophic risk paradigms’, Ecological Economics, 107 (2014): 13–21.

8 See Hörhager, Elisa and Julius Weitzdörfer. ‘From natural hazard to man-made disaster: The protection of disaster victims in China and Japan’, Protecting the Weak in East Asia. Routledge (2018), pp. 139–65.

1. Ripples on the Great Sea of Life: A Brief History of Existential Risk Studies

SJ Beard and Emile P. Torres

© 2024 SJ Beard and Emile P. Torres, CC BY-NC 4.0 https://doi.org/10.11647/OBP.0360.01

Research highlights:

While thoughts about naturalistic human extinction can be traced back to the latter 19th century among both speculative artists and concerned scientists, the field of Existential Risk Studies (ERS) only emerged in the last two decades and can be characterised by three distinct “waves” or research paradigms.

The first was built on an explicitly transhumanist and techno-utopian worldview and the risks associated with it.

The second grew out of an ethical view known as “longtermism”, closely associated with the Effective Altruism movement, and is concerned with creating the most value possible.

The third emerged from the interface between ERS and other fields that have engaged with existential risk, such as Disaster Studies, Environmental Science and Public Policy.

In adumbrating the evolution of these paradigms, together with their historical antecedents, the authors offer a critical examination of each and speculate about where the field may be heading in the future.

This chapter sketches the history of Existential Risk Studies up to the year 2020. Chapters 3 and 4 provide some of the key original sources for the shift to global systems thinking described here as the third wave of ERS. The continuing influence of speculative fiction on ERS and wider social perceptions around AI and Existential Risk are discussed in Chapter 8.

But if some poor story-writing man ventures to figure this sober probability in a tale, not a reviewer in London but will tell him his theme is utterly impossible. And, when the thing happens, one may doubt if even then one will get the recognition one deserves

— H. G. Wells,

The Extinction of Man

(1897)

A colleague of mine likes to point out that a Fields Medal (the highest honor in mathematics) indicates two things about the recipient: that he was capable of accomplishing something important, and that he didn’t. Though harsh, the remark hints at a truth

— Nick Bostrom,

Superintelligence

(2014)

There is an emerging scientific consensus that, due to the multiplicity of risks with the potential to cause global catastrophes, Homo sapiens is now in the most perilous moment of its 300,000-year history. We face global challenges of such magnitude that, by comparison, all the set-backs and tragedies of human history are “mere ripples on the surface of the great sea of life”.1 Yet if all that we have ever known are such ripples, how can we understand, let alone stop, the tidal waves that threaten to engulf us?

It is thus hardly surprising that a new field focused on the long-term survival of our species is emerging. This has variously been referred to as “Existential Risk Studies” (ERS), “Existential Risk Research” and “Existential Risk Mitigation”. For the present purposes, we will use the acronym “ERS”, to fit with related fields such as Futures Studies, Science and Technology Studies and Disaster Studies. The aim of this chapter is to explore the historical development of ERS and, in doing so, to identify points of convergence and divergence between different researchers studying existential risk. We argue that there have been multiple ERS paradigms or “waves”, i.e., sets of concepts and practices in the sense of Thomas Kuhn (1922–1996). These can be distinguished according to the following issues: (i) definitions of key terms, (ii) motivating values, (iii) classificatory systems, and (iv) methodologies.

We break these paradigms into four groups, which we consider in successive sections of this work. The next section deals with the history of thinking about existential risk that preceded the emergence of ERS as a unified field of study in the early 2000s, looking at how the topic has been explored by speculative fiction authors and concerned scientists. Section 2 considers the forces that helped to unify ERS in the first decade of the 21st century, which arose from a specifically transhumanist or techno-utopian world view. Section 3 explores a second paradigm, connected with a significant expansion of both interest in and support for this field, related to the growth of the “Effective Altruism” (EA) movement after 2009 and its promotion of ethical longtermism. Section 4 examines a third paradigm, which has emerged in recent years both within certain centres of ERS research and among the scientists from other fields who are beginning to engage with it, that focuses more on global systems and is comparatively less interested in ethics. Finally, Section 5 offers some speculation about the possible future trajectories of ERS and the developments that will drive them: increased scrutiny and public attention, the growing list of existential threats to humanity, and the diversification of the field.

In breaking down the paradigms of ERS into successive waves we do not claim that these represent cohesive social groups or schools of thought; it is notable that many individual scholars have passed between many of them and would not necessarily identify any strong change of mindset in doing so. Nor do we mean to imply that successive waves have succeeded or replaced each other. However, we do claim that roughly combining the work of scholars into these waves tells an interesting and useful story that helps to illustrate and explain the development of the field of ERS. Even more importantly, we hope that it helps to identify how the seemingly disparate and even contradictory claims of scholars can be understood as offering complementary perspectives on a common problem, and thus that our work will help to ensure that ERS remains a coherent field of study as it continues to diversify.

Section 1: The Prehistory of ERS

People in many cultures throughout history have speculated about the possibility of global catastrophes, up to and including the “apocalypse” or “end of the world”. Indeed the first story ever to have been written down may well have been the Mesopotamian “flood myth”, which tells of a flood that wiped out all but two humans and is familiar to most in the west through its inclusion in the Bible as the story of Noah.2 However, such speculation has largely been bound up with religious beliefs and invariably ends with the survival of humanity, either on Earth, in an afterlife or via an eternal cosmic cycle of rebirth.3 In contrast, the notion of existential risk is both absolute (humanity’s extinction or ruination is both total and irreversible) and naturalistic (the fate of humanity is to be brought about in accordance with scientific laws of nature). Concern about this kind of catastrophe has been far less common. Indeed, the very idea of human extinction is a recent invention. The four primary reasons4 for this are that:

The scientific community largely rejected the possibility that species could go extinct until the French zoologist Georges Cuvier (1769–1832) demonstrated that elephantine bones unearthed in Siberia and North America belonged to mammoths and mastodons.

The belief that an ontological gap separates humans from nature, which was prominent at least until Charles Darwin’s

On the Origin of Species

,

5

convinced the scientific community that evolution is a fact about the history of all Earth-originating life, metaphysically integrating humanity into the natural order.

Religious eschatologies monopolised thinking about the fate of humanity until the 19th century; it wasn’t until the 1960s that the “Age of Atheism” commenced, to borrow a term from Gerhard Ebeling.

6

There was no agreement within the scientific community about the existence of potential kill mechanisms (other than the second law of thermodynamics) that could annihilate humanity until the second half of the 20th century.

Yet, over the past two centuries, several historical precedents for the modern field of ERS have emerged, and it is worth considering these before turning to the history of this field.

Speculative fiction

Some of the earliest thinking about human extinction in a naturalistic sense are found among artists in the early 19th century. For example, in works by Lord Byron (1788–1824), the infamous romantic poet and father of computer pioneer Ada Lovelace. Lord Byron is reported to have been interested in comets and concerned that humanity would someday perish as a result of a comet impact, while his 1816 poem “Darkness” imagines a future in which Earth becomes lifeless (probably inspired by the after-effects of the 1815 eruption of Mount Tambora).7 Mary Shelley (1797–1851), Byron’s friend and the founder of science fiction, published The Last Man in 1826.8 This tells the story of Lionel, who witnesses the death of all other human beings in the last few decades of the 21st century from a series of apocalyptic events, most notably a worldwide plague, and must come to terms with the fate of the world. Shelley was likely influenced by the loss of her husband (Percy) and many friends, including Lord Byron, in the preceding years. However, she may also have been influenced by the work of her parents, William Godwin and Mary Wollstonecraft, who envisioned utopian futures of social equality and progress, which Mary’s own life had often failed to realise.