22,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Discover the next major revolution in data science and AI and how it applies to your organization
In Causal Artificial Intelligence: The Next Step in Effective, Efficient, and Practical AI, a team of dedicated tech executives delivers a business-focused approach based on a deep and engaging exploration of the models and data used in causal AI. The book’s discussions include both accessible and understandable technical detail and business context and concepts that frame causal AI in familiar business settings.
Useful for both data scientists and business-side professionals, the book offers:
- Clear and compelling descriptions of the concept of causality and how it can benefit your organization
- Detailed use cases and examples that vividly demonstrate the value of causality for solving business problems
- Useful strategies for deciding when to use correlation-based approaches and when to use causal inference
An enlightening and easy-to-understand treatment of an essential business topic, Causal Artificial Intelligence is a must-read for data scientists, subject matter experts, and business leaders seeking to familiarize themselves with a rapidly growing area of AI application and research.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 481
Veröffentlichungsjahr: 2023
Ähnliche
Table of Contents
Cover
Table of Contents
Title Page
Foreword
Preface
Introduction
CHAPTER 1: Setting the Stage for Causal AI
Why Causality Is a Game Changer
Causal AI in Perspective with Analytics
Analytical Sophistication Model
Scope of Services to Support Causal AI
The Value of the Hybrid Team
The Promise of AI
Understanding the Core Concepts of Causal AI
Summary
Note
CHAPTER 2: Understanding the Valueof Causal AI
Defining Causal AI
The Origins of Causal AI
Why Causal AI Is the Next Generation of AI
The Business Imperative of a Causal Model
The Importance of Augmented Intelligence
The Importance of Data, Visualization, and Frameworks
Getting Started with Causal AI
Summary
Notes
CHAPTER 3: Elements of Causal AI
Conceptual Models
Process Models
Collaboration Between Business and Analytics Professionals
The Fundamental Building Blocks of Causal AI Models
The Elements of Visual Modeling
Summary
Notes
CHAPTER 4: Creating Practical Causal AI Models and Systems
Understanding Complex Models
Causal Modeling Process: Part 1
Causal-Based Approach: Part 2
Summary
Notes
CHAPTER 5: Creating a Model with a Hybrid Team
The Hybrid Team
Defining Roles
The Basics Steps for a Hybrid Team Project
An Overview of Model Creation
Summary
CHAPTER 6: Explainability, Bias Detection, and AI Responsibility in Causal AI
Explainability
Detecting Bias and Ensuring Responsible AI
Summary
Note
CHAPTER 7: Tools, Practices, and Techniques to Enable Causal AI
The Causal AI Pipeline
The Importance of Synthetic Data
Current State of Tools and Software in Causal AI
Summary
CHAPTER 8: Causal AI in Action
Enterprise Marketing in a Business-to-Consumer Scenario
Moving from Strategy to Building and Implementing Causal AI Solutions
Summary
CHAPTER 9: The Future of Causal AI
Where We Stand Today
Foundations of Causal AI
The Causal AI Journey
Integrating Causal AI and Traditional AI
The Imperative for Managing Data
Ensembles of Data
Generative AI Is Emerging as a Game Changer for Causal AI
The Future of Causal Discovery
The Emergence of Causal AI Reinforcement Learning Will Accelerate Model Training
Causal AI as a Common Language Between Business Leaders and Data Scientists
The Emergence of Probabilistic Programming Languages
The Predictable Model Evolution Cycle
The Emergence of the Digital Twin
Improving the Ability to Understand Ground Truth
The Development of More Sophisticated DAGs
Visualizing Complex Relationships in the DAGs
The Merging of Causal and Traditional AI Models
The Future of Explainability
The Evolution of Responsible AI
Advances in Data Security and Privacy
Integration Will Be Between Models and Business Applications
Summary
Glossary
Appendix: Causal AI Tools and Libraries
Selected Resources
Acknowledgments
About the Authors
About the Contributor
Index
Copyright
Dedication
End User License Agreement
List of Tables
Chapter 3
Table 3.1 Basic Entities in an Entry-Level Causal Model
Chapter 4
Table 4.1 Definition of Common Effects in a Causal Model
Table 4.2 Refined and Extended Effects in a Causal Model
Chapter 5
Table 5.1 Key Roles in the Causal AI Team
List of Illustrations
Chapter 1
FIGURE 1.1 Analytical sophistication model
FIGURE 1.2 The collaborative process begins by articulating the problem bein...
FIGURE 1.3 Causal model demonstrating the relationships between Product Qual...
Chapter 2
FIGURE 2.1 A replica of the Broad Street water pump.
FIGURE 2.2 The Ladder of Causation indicates the stages of Pearl's view of c...
FIGURE 2.3 An example of a simple causal model of a marketing campaign.
Chapter 3
FIGURE 3.1 The core correlation-based AI model begins by identifying raw dat...
FIGURE 3.2 Causal-based AI model, part 1. A causal AI model models the proce...
FIGURE 3.3 A sample DAG describes the basic relationships in a model.
FIGURE 3.4 In Wright's 1921 paper, “Correlation and Causation,” he used caus...
FIGURE 3.5 A sample DAG illustrating the relationship between stress and hea...
FIGURE 3.6 A DAG with weights illustrated
Chapter 4
FIGURE 4.1 The process of creating a causal AI model is an iterative process...
FIGURE 4.2 The DAG modified to include price as a treatment
FIGURE 4.3 The DAG modified to include a confounding variable
FIGURE 4.4 A mediator variable can unlock the relationship between variables...
FIGURE 4.5 A DAG illustrating the chain path type
FIGURE 4.6 A DAG illustrating the fork path type
FIGURE 4.7 A DAG illustrating the inverted fork path type
FIGURE 4.8 A DAG with an unobserved variable
FIGURE 4.9 Causal-based AI model: part 2
Chapter 5
FIGURE 5.1 A basic causal model to determine the cause and effect of a marke...
FIGURE 5.2 An indication of the causal strengths of variable relationships i...
Chapter 6
FIGURE 6.1 Explainability requires an understanding of the data, modeling re...
Chapter 7
FIGURE 7.1 A typical DevOps infinity loop diagram
FIGURE 7.2 The iterative causal AI pipeline
FIGURE 7.3 The cycle isn't just a continuous loop; instead, there are epicyc...
Chapter 8
FIGURE 8.1 DAG for the DDCo annual pricing review and update cycle
FIGURE 8.2 DAG for the DDCo semi-annual product review cycle
Guide
Cover
Table of Contents
Title Page
Copyright
Dedication
Foreword
Preface
Introduction
Begin Reading
Glossary
Appendix: Causal AI Tools and Libraries
Selected Resources
Acknowledgments
About the Authors
About the Contributor
Index
End User License Agreement
Pages
i
ii
v
xix
xx
xxi
xxiii
xxiv
xxv
xxvi
xxvii
xxviii
xxix
xxx
xxxi
xxxii
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
299
300
301
302
303
304
305
306
307
308
309
310
311
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
331
332
333
335
336
337
339
340
341
342
343
344
345
346
347
348
349
350
351
352
vi
vii
353
Praise for Causal AI
I am delighted to welcome this exceptional book on causal AI, written by Judith Hurwitz and John Thompson, which bridges the gap between conventional methods of business analytics and modern techniques of causal analysis and machine learning. Business leaders and data scientists who are trained in prescriptive statistics, simulation, and optimization techniques will be happy to discover how critical problems in their field, commonly left to intuition or contentious judgment, can now be systematically conceptualized, articulated, and solved using simple techniques of causal artificial intelligence. These include predicting the effects of potential interventions, finding explanations for unexpected outcomes, testing the validity of one's assumptions, and combining data from several sources. The lucid exposition of this book and the variety of practical examples provided promise to make business analytics the new beneficiary of the causal revolution.
—Judea PearlUCLA chancellor professor emeritus in computer science, recipient of the A.M. Turing Award
As the authors describe in detail, the vast majority of AI today is based on statistical association rather than causality. Under many circumstances those results aren't sufficiently clear-cut to drive action. The world will eventually move toward causal AI, and this book will provide a head start.
—Thomas H. Davenportdistinguished professor at Babson College; author of All In on AI, Working with AI, and The AI Advantage
Causality is no longer a purely academic sport. From Amazon to Spotify and Netflix, companies worldwide are investing in building causal systems to capture value where traditional machine learning and statistics fall short. Judith and John take a unique perspective on causality, focusing on the business side of things and shedding light on practical aspects of causality, which are no less than essential to any modern decision-maker.
—Aleksander Molakauthor of Causal Inference and Discovery in Python, community leader at CausalPython.IO
Judith's and John's book turns over all the proverbial stones revealing deep truths about causal AI and AI more generally. While they explore relevant topics like avoiding bias in AI, the realities of data federation, and many more, I was captivated by the care taken to relate the human and AI intersection. In this exploration, the authors illustrate how experts are important for AI developments to move past the “science fair stage” to tooling that reliably produces trustable outcomes with easy-to-understand utility.
—Michael Haydigital infrastructure portfolio leader at Hitachi Vantara
The causal AI revolution is starting to have a huge impact on digital transformation. Being able to determine why events happened and how best to leverage or mitigate them has the potential to help management make better decisions and grow revenue. The vast amount of available technical knowledge and open-source tools means that the commercialization of causal AI is becoming a reality. This book provides an invaluable resource for business teams to gain insights into the way causal AI can transform business.
—Stuart FrostCEO at Geminos Software
Causal Artificial Intelligence
The Next Step in Effective Business AI
Judith S. Hurwitz
John K. Thompson
Foreword
Dear Reader,
It is with great pleasure and enthusiasm that I introduce you to this exceptional book on causal AI, written by the brilliant Judith Hurwitz and John Thompson. As I embarked on my journey through its pages, I found myself intrigued by the depth of knowledge, clarity of explanations, and practical insights it offers. “Causal Artificial Intelligence: The Next Step in Effective Business AI” is a true gem for those seeking to comprehend the intricate world of causal artificial intelligence.
In today's rapidly evolving technological landscape, the concept of causal AI has emerged as a source of radical innovation and a catalyst for transformation. It holds the key to unlocking the true potential of artificial intelligence, empowering us not only to predict and classify but also to understand the causal mechanisms that underlie complex systems. Through the lens of directed acyclic graphs (DAGs) and structural causal models (SCMs), pioneered by the eminent Prof. Judea Pearl, this book guides us on a remarkable journey of discovery.
One of the book's greatest strengths is its accessibility, making it a perfect companion for those who are eager to grasp the essence of causal AI but lack a formal background in the field. Whether you are a seasoned professional in the realm of business analytics or someone just stepping into this fascinating domain, Judith Hurwitz and John Thompson expertly demystify the core concepts, rendering them comprehensible to a wide audience. Their ability to translate complex theories into practical applications is commendable, making “Causal Artificial Intelligence” an invaluable resource for both novices and experts alike.
As you delve into the pages that follow, you will encounter crucial topics that lie at the heart of causal AI. Exploring the realms of explainability, you will discover how causal models offer an unparalleled understanding of why certain decisions are made by AI systems. In a world increasingly driven by algorithms, the book masterfully examines the biases that can be inadvertently embedded within these systems and provides valuable insights into mitigating such biases, ensuring fairness and equity in algorithmic decision-making.
Another significant facet the book illuminates is the vital aspect of robustness in AI systems. Through a meticulous exploration of DAGs and SCMs, you will gain a deep understanding of how to construct models that are not only accurate but also resilient to unforeseen perturbations and uncertainties. This robustness is crucial in building trustworthy and reliable AI systems that can be confidently deployed in real-world scenarios.
The book strikes a perfect balance between theory and practice. While it delves into the fundamental principles and theories that underpin causal AI, it never loses sight of the practical applications and real-world implications. The numerous case studies and examples provided throughout the book reinforce its relevance and enable you to connect the dots between theory and its practical implementation.
I have no doubt that this exceptional work will become an indispensable resource for professionals in business analytics roles. The knowledge imparted within these pages has the potential to revolutionize the way we approach artificial intelligence, enabling us to harness its power responsibly and ethically. Whether you are a data scientist, a decision-maker, or simply an enthusiast seeking to expand your understanding, this book will serve as an invaluable guide on your journey to master the intricacies of causal AI.
In conclusion, I would like to express my deepest appreciation to Judith Hurwitz and John Thompson for writing this remarkable book. Their passion for the subject matter is evident, and their ability to distill complex ideas into clear and relatable explanations is truly remarkable. Each chapter takes you on a journey, unraveling the intricacies of causal AI and imparting knowledge that is both insightful and actionable. Prepare to embark on a transformational voyage, unlocking the power of causal AI and embracing a future where we not only predict and observe but truly understand.
Enjoy the journey!
Sincerely,
Paul Hünermund
Assistant Professor of Strategy & Innovation
Copenhagen Business School
Preface
In my view, causal AI is the next stage in the evolution of software because it is focused on being able to understand the causes and effects of events. As we discuss in this book, what has caused a marketing campaign to achieve the revenue objectives? Is the problem the campaign itself, or are there underlying issues that are impacting results? Is the cause of the disappointing marketing campaign because of a sudden competitive threat? Is there a problem with the company's reputation? What would the impact on revenue if the product price was reduced by 10 percent? Would a different type of marketing campaign result in better results? The underlying casual technology needed to address these problems is complex, and the approach is instrumental for business leaders to understand the potential impact. Therefore, unlike some earlier evolutions of AI, the value of a causal AI approach can have a direct and profound effect on business outcomes.
A plethora of books and articles already address causal inference—a field that must recognize Judea Pearl as a pioneer and visionary in causality. So, why write yet another book on the topic? The reason is straightforward—this book is written for technology-focused leaders who are not developers but are responsible for bringing new technology into their companies to gain a competitive edge. In writing this book, I have spent countless hours speaking with leaders in the field and reading many articles and books. The goal of this book is to provide an understanding of why the field of causal AI is so important. It has the potential to truly transform how we use artificial intelligence to digitally transform business.
My journey through the complex world of software started more than 35 years ago. My experience in technology began when I joined a financial services company and was tasked with introducing emerging technology to various business units. The goal was to evaluate how the technology could help transform the competitiveness of the business. From that beginning, I went on to spend many years as a developer, strategy IT consultant, industry analyst, thought leader, and writer. Most recently, I joined Geminos Software, a causal AI company, as their chief evangelist. I credit my ability to begin to understand this amazing and complex technology to the insights and wisdom of the Geminos team.
While I have spent years delving into some of the most complex technologies, I have always put solutions in perspective by focusing on the needs of the business organization. No matter what position I have been in, I always asked some variation of the same questions: What is the purpose of a software platform, and how does it help the business flourish? Why is the technology important?
Since I have always focused on those key issues, it is not surprising that I have paid particular attention to some of the most complex emerging technologies. During my pursuit of learning and understanding the value of new offerings, I have coauthored 10 books and dozens of customized e-books all focused on explaining complex technologies to both business and technical audiences. My goal has long been to bridge the gap of how business and technology leaders must collaborate to be able to succeed. I have always believed that customers will not buy technology that they do not understand. Topics of the books I have coauthored include service-oriented architecture, big data, machine learning, and cloud computing. My two most recent books focused on cognitive computing and augmented intelligence. Both books have informed my journey to an exploration of causal AI.
As with any emerging technology, causal AI will evolve over the coming decade. The goal of this book is to provide guidance and an understanding for a business audience of the foundation of this important technology. As a participant in the world of emerging technologies, I felt it was the right time to put causal AI in perspective.
—Judith Hurwitz
May 2023
While writing this book on causal AI, generative AI burst onto the market with great excitement, fanfare, and disruption. I was asked by more than a few people who knew that I was involved in writing a book on causal AI if I should put this book on hold and focus my current efforts on generative AI. As with all reasonable suggestions and questions, I considered the change in direction. My conclusion was that while generative AI is transformative in relation to how people are employed, how work will be executed, the impact on productivity, and more, generative AI is not a new field of AI. Generative AI is an extension of, and a new way of combining, neural networks, unsupervised learning, supervised learning, reinforcement learning, and much larger models than we have seen before, but it is not a new field of AI, not the way causal AI is. Hence, my conclusion was that while my day job is dominated by determining how to design, leverage, govern, deploy, and use generative AI in an enterprise environment, this book on causal AI was still needed to raise the awareness of the power, value, and transformative nature of causal AI.
My main motivation for writing this book was to put an original book into the market that takes the dialogue relating to causal AI in a new direction—a direction that begins to draw the business, technical, and analytical communities into the dialogue.
In my research to expand my fundamental understanding of causal AI and the stage of development of this completely new field of AI, before the writing process began, I read nearly 100 pieces of original writing. All of the books, research papers, most of the blogs, and more, on causal AI immediately dove into the details of the calculus and related math underlying causal AI. I refreshed my understanding of calculus that I learned in graduate school. My knowledge of calculus was extended, sharpened, and revived, but I knew that this type of writing was a barrier to broadening and deepening my understanding of causal AI. I also knew that if it was a high barrier for me, then it was a complete showstopper for most people.
I knew that the audiences that I felt needed to know about causal AI were not, for the most part, going to wade through even a 10th of what I had read. I became excited about the opportunity to be among the first people in the field of data, analytics, and AI to develop and carry the message forward that causal AI was being developed, was a powerful new tool, and would be a significant advance in our arsenal of tools in our quest to document, model, and understand our world in a more complete manner.
I wrote Building Analytics Teams (BAT) after having built multiple analytics teams over the previous 37 years as a technologist and an AI practitioner. One of my goals, and my primary objective, in writing BAT was to help people from all walks of life who have more than a passing interest in being part of the fields of data, analytics, and AI to understand the real-world environment, the environment in the majority of enterprise-class organizations, and the real constraints and opportunities that are at play in working in the field of analytics. I wanted to help new college graduates to understand what working in analytics really looked and felt like. I wanted new managers to have a “how to” book on how to design, build, manage, and grow, their analytics teams, and I wanted, most of all, to help analytics professionals to not make the same mistakes that I made. I wanted to make their lives and journeys better. In BAT, I accomplished that goal.
My primary goal in writing this book is to help draw the business, technical, and analytical communities into an exploration of the emerging field of causal AI. I want those practitioners to buy and read this book to understand what is coming next. I want them to engage with the content to fire their imaginations about what they can do with causal AI and how causal AI is an entirely novel and new approach to AI that expands their toolset and puts the power of AI in the hands of the business users. In that respect, putting the power of AI in the hands of business users, causal AI has some similarities to generative AI, but only at a conceptual level.
I recognized that causal AI was a completely new field of AI, and I wanted to be part of the evolution, to be a messenger that raises the awareness of this impressive new area. I knew, and know, that once causal AI moves beyond the research phase into the early adopter phase, there will be a flurry of activity enabling early-mover companies to build and maintain a defensible and significant competitive advantage. This book is a call to action for those early-stage enterprise-class innovators to take notice of causal AI and to begin their process of investigating the potential of this technology and approach.
One of the early epiphanies that I experienced in researching the topic was that the underlying causal approach could be applied to any process. Historically, the causal approach was applied to agriculture, healthcare, and specialty use cases such as dog breeding. But, as I looked back in time, all the way to ancient Greece, and then forward again to ages like the Renaissance and the Reformation, it was clear that philosophers, mathematicians, and academics of all types were touching on causality and slowly but consistently adding to the global corpus of knowledge related to causality.
This aggregation of knowledge reached an acceleration point in the past century, and causal AI gained a dedicated and devout following that drove the development of casual AI to a new level. Once I realized that the field of causal AI was racing forward, I wanted to write this book.
So, why did I write this book, or atleast my part of the book? I wanted to contribute to the understanding, adoption, and use of this incredible new toolset and technologies that we refer to as causal AI.
I hope that you enjoy reading and learning about causal AI as much as I did.
—John K. Thompson
May 2023
Introduction
Why this book, and why now?
We have spent decades exploring, researching, writing, and working with the most important emerging technologies. We have seen hundreds of innovative and novel technologies come and go, each promising to turn human knowledge into packaged solutions that are easy to understand and implement. The history of technology solutions has proven repeatedly that there are no simple solutions to complex problems. However, each technological solution takes us a step further to addressing the core issues. For the past 10 years, the focus of AI and advanced analytics has been on analyzing massive amounts of data to understand the answers to difficult problems. Big data was the silver bullet that offered some success but did not go far enough. In fact, often beginning with big data created correlations that sent businesses in the wrong direction.
One of the problems with leveraging complex technology solutions is that they are multifaceted, interconnected, and complex. It is possible that the data scientist can understand all the ins and outs of the underlying math and technology, but to be successful, the data team must work in collaboration with IT and business to anticipate customer needs and to plan for what's next. In most cases, business leaders do not understand emerging technologies, the data, or the underlying math; hence, they don't know what questions to ask to determine if the technology is well suited to solving their specific operational challenges. We have seen this knowledge gap and mismatch in understanding multiple times. Therefore, one of our primary goals in writing this book is to bridge the knowledge and communication gap between data scientists and the business leaders so that a door can be opened to facilitate a conversation and create a venue for collaboration.
However, there is no silver bullet. Many companies are either adopting or evaluating artificial intelligence-based solutions to automate processes and to determine what specific changes can be implemented to improve their businesses. The promise of AI is tantalizing—organizations can use algorithms to analyze their data in context to anticipate changes in customer requirements and prepare for the future. In competitive markets, it is imperative to understand what is happening within the industry and how to ensure that revenue can grow. When looking into the future, organizations need to be able to understand the impact of decision-making. What happens if a product price is reduced by 10 percent? Will this cause more customers to buy? If revenue suddenly decreases, does management understand why this has happened and what to do to change things? Are customers leaving because of a quality issue with a new supplier or because of an emerging competitor? Understanding the cause and effect from processes and data is the goal and the reason that causal inference is suddenly becoming such a critical approach.
How is causal inference different from other types of artificial intelligence? Simply put, causal inference and the resulting causal AI solutions focus on the assumptions we make about the world and specifically business and the underlying processes that are executed each day. The goal of causal inference is to be able to understand the “why” in the story of the data.
We wrote this book because we believe that causal AI is going to open the door to solving many critical problems in business, engineering, manufacturing, and science. While the idea of causal inference as a topic has been around for centuries, it is only now becoming the lynchpin of addressing the most complex problems facing us today. One of the benefits of causal AI is that it assumes that there is a hybrid group of professionals who collaborate to find the cause and effect from data. This hybrid team consists of data scientists, subject-matter experts, data experts, technologists, business managers, and executives.
This book is intended to provide guidance to all the members of this hybrid team. For example, for the data scientist, we will provide deep technical information as well as the type of information needed to collaborate with subject-matter experts. For the subject-matter expert, we will provide explanations that help to converse with the data scientists. These teams need to be able to work with experts who understand the business data within their organizations so they can be part of the process. Business executives and managers must be able to direct the hybrid team based on the direction that the organization wants to take and the problems that need to be solved. You will therefore be able to select sections that apply to your knowledge level.
We have been working in the intersection of business and technology for decades. We have both written numerous books and have been part of the management team of several companies. Our goal with this book is to bring an understanding and context for this important transition in artificial intelligence.
We are in an interesting and complicated transition in the evolution of artificial intelligence. While the focus of many traditional AI solutions is on data engineering, there is an interesting and revolutionary trend emerging. This revolution is called causal AI. This is a sophisticated technology, but it is also a transformational technology.
To summarize the main point to be made, causal inference is the science of why, as explained so very well by Judea Pearl in The Book of Why. Dr. Pearl states, “Some tens of thousands of years ago, humans began to realize that certain things cause other things and that tinkering with the former can change the latter.” His point is that while we can't know all the answers, we can ask why events happen and the cause and effect of a business situation we are trying to solve. As Dr. Pearl accurately sums up the promise of causal inference, “The ideal technology that causal inference strives to emulate resides within our own minds.”
We hope you enjoy the book and that the content fires your imagination to learn more about causal inference and causal AI.
CHAPTER 1Setting the Stage for Causal AI
The ability to understand information in the context of solving complex problems is not new. From the earliest days of artificial intelligence, scientists and mathematicians have tried to find new ways to understand the world through models and data. The promise of artificial intelligence (AI) is to reach the point where machines could think and provide answers to some of the most challenging problems of our world. There are a huge number of sophisticated analytics tools that provide significant help in understanding what has occurred in the past and predict a possible future from that data. However, one element that has been missing from the analyses is understanding the cause and effect of the observed and unobserved interactions. The dynamic of understanding why events happen and what can be done to change the outcomes is the power and opportunity of causal AI. This chapter will put causal AI in perspective and set the stage for our exploration of the evolution of the field of AI.
Why Causality Is a Game Changer
Why is there a sudden explosion in interest in causal AI? The answer is both complex and simple. Causal AI enables us to move beyond the predictive modeling capabilities of traditional AI to understand and predict causal relationships between variables in a system. Here are some of the most salient topics that outline the value of causal AI:
Understanding causality
:
Traditional AI models can make predictions based on observed correlations between variables but cannot tell us why a particular outcome occurred. Causal AI, on the other hand, can identify the causal relationships between variables and help us understand why a particular outcome occurred. Causality and understanding the dynamics of causality can be particularly important in fields such as healthcare, where understanding the causal relationships between risk factors and health outcomes can help identify new interventions and treatments.
Identifying interventions
:
Causal AI can help us identify interventions that can change outcomes. For example, causal AI provides a graphical technique to pinpoint the most relevant variables needed to understand specific objective or estimate the consequences of a given intervention. The goal of causal AI is to help an organization assess the possible cause and effects of various policy actions. Causal AI has the potential to enable a team to have a common understanding of a problem so that they can work together to determine why a situation has occurred and establish a plan to arrive at the best next actions.
Predicting counterfactuals
: Causal AI can predict the effect of a particular variable on an outcome of interest in an alternative scenario. This is especially useful when the variable of interest is not directly observable or measurable, as it allows the estimation of the causal effect on the outcome. For example, it can help predict what would have happened if a particular intervention or policy had not been implemented.
Avoiding bias
:
Traditional AI models can be biased if they are trained on biased data or if they do not account for all relevant variables. Causal AI, on the other hand, can help avoid bias by identifying and accounting for all the relevant variables in a system. This can help ensure that the predictions and decisions made using AI are fair and unbiased.
Improving decision-making
:
Causal AI can help make better decisions by providing a deeper understanding of the causal relationships between variables. This can be particularly useful in fields like business, where understanding the causal relationships between different variables can help businesses make more informed decisions and achieve better outcomes. Causal AI provides us with a deeper understanding of the causal relationships between variables in a system and can help us identify interventions, predict alternative choices and actions, avoid bias, and make better decisions.
The next generation of artificial intelligence can benefit from a deeper level of collaboration between data experts, business leaders, and subject-matter experts. While AI has long been used to solve complex problems, in this new era of expanded AI, hybrid teams of business and analytics professionals can include an examination of why problems happen and what alternate approaches can help a business move forward to gain a sustainable and measurable advantage when faced with increasingly sophisticated and aggressive competition. Therefore, we are in an interesting and complicated transition in the evolution of artificial intelligence. While the focus of many traditional AI approaches focuses on data and feature engineering, there is a revolutionary trend emerging. Causal AI uses causal inference as the underlying math of cause and effect. The focus of causal AI is on business outcomes. Causal inference is the science of why, as explained so well by Judea Pearl in The Book of Why: “The ideal technology causal inference strives to emulate resides within our own minds. Some tens of thousands of years ago, humans began to realize that certain things cause other things and that tinkering with the former can change the latter.”1
While it is time-consuming and challenging to examine all the possible answers, we can easily ask why events happen and what are the primary the cause-and-effect factors of a problem we are trying to solve.
We have many years of experience working with business and technology leaders who are grappling with some of the most complex problems that our current and traditional technologies are designed to solve. We have seen hundreds of emerging technologies come and go that promise to turn human knowledge into packaged solutions that are easy to understand and implement. The history of technology has proven repeatedly that there are no simple solutions to complex problems. However, each technology takes us a step further to addressing issues. For the past 10 years the focus of AI and advanced analytics has been on analyzing massive amounts of data to understand the answers to difficult problems. Big data was the silver bullet that offered some success but did not go far enough.
One of the biggest stumbling blocks to making AI and advanced analytics solutions work effectively is the complexity of the underlying technologies. Typically, business leaders want to be able to visualize the outcomes from the data buried inside applications and from both internal and external data sources. Business managers and leaders want to not only understand what the data tells them about their current situations but what actions they can take to protect and advance their future goals and objectives. Business leaders look to data scientists who employ statistical and computational techniques to determine insights from big data. Many data scientists use correlation and machine learning techniques to identify patterns and anomalies to predict outcomes. Increasingly, business leaders are beginning to understand that there is tremendous potential to leverage AI to solve complex business problems. The greatest potential for AI is to create a way to abstract the complexity from the underlying technology so that data scientists, subject-matter experts, data specialists, and business leaders can collaborate to solve business problems. Therefore, one of our goals with this book is to bridge the gap between the data scientist and the business leader so that it opens the door to use the power of causal AI and traditional AI to create a competitive advantage.
However, there are no silver bullets or simple answer. Many business and technology leaders are either adopting or evaluating AI-based solutions to automate processes and determine how to improve their businesses. The promise of AI is tantalizing—organizations can use algorithms to analyze their data in context to anticipate changes in customer requirements and prepare for the future.
In competitive markets, it is imperative to be able to understand what and why situations are happening within the business. How can leadership within a business ensure that revenue can grow? When looking into the future, organizations need to be able to understand the impact of the decisions they make. What happens if a product price is reduced by 10 percent? Will a lower price cause more customers to buy? Will the price increase entice more new customers to buy? If revenue suddenly decreases, does management understand why this has happened and what can be done to change the current course of business? Are customers leaving because of a quality issue triggered because the business began using a new supplier or because of an emerging competitor? Understanding the cause and effect from processes and data is one of the primary reasons why causal AI is emerging as such a critically importatnt approach across the fields of academia and business.
The most common techniques that have been used by data scientists are correlation-based techniques that are common in the field of traditonal AI. While correlation and causality are related approaches, they are not the same. In brief, correlation is a technique for establishing the statistical relationship between variables. In contrast, causality refers to how one variable has an impact on other variables. In the case of causality, one variable might have a direct impact on a second variable, there could be an indirect effect, or there could be a confounding effect. Causal AI is the art and science of understanding the myriad of relationships between variables that drive relevant causes and effects in a system that we are seeking to understand and manage.
Understanding the difference between correlation-based statistical analysis and causality-based analytics is key to being able to employ and deploy the power of causal AI. Therefore, in the next section we will explore the broad area of analytics. Applying a causal AI approach to analytics will help guide organizations to focus on the assumptions and knowledge that we have about how the world works. If we can answer the complex questions about why an issue occurs, we can adapt our approach to solve problems. The goal of causal AI is to be able to understand the story of the data.
Causal AI in Perspective with Analytics
Analytics is one of the most widely used, and often misused, terms today. The discussion of analytics is widespread. At a conceptual level, correlation-based AI and causal-based AI approaches have the same roots; they come from the same family/branch/category of advanced analytics. So, before we move forward with our discussion and description of causal AI, let's define the broader term analytics to ensure that we have a common understanding as we move forward in our dialogue.
The term analytics often refers to dashboards and historical reports. In addition, analytics includes collections of data, information, applications, and analytical models related to work with descriptive statistics. Analytics encompasses work products resulting from predictive, prescriptive, simulation, and optimization projects and programs.
There are many different perceptions and assumptions about what it means to conduct an analytics project. To make matters worse, there is likely to be a different vision for how to approach analytics. Typically, organizations and departments will be trying to solve very different problems depending both on the problems they need to address and on the stage of their projects. Approaching analytics in the context of causal AI requires a common understanding of the types and approaches to analytics.
Routinely, we talk about analytics with academics, researchers, government officials, university administrators, scientists, business executives, subject-matter experts, data scientists, and more. So, it is not surprising to see confused expressions on people's faces when someone talks about analytics. One person is talking about a historical dashboard, while others are trying to convey how a simulation model might work in the same operational area of the business. It can be frustrating for people not to be able to connect on a topic in which they are all deeply interested.
We have found that beginning a discussion by defining the analytical maturity model or framework to be used in the ongoing dialogue helps in reducing confusion and ensures that everyone in the conversation has a clearer understanding about what is being said or what others are trying to convey given the wide range of definitions individuals hold in their minds about the term and areas of analytics.
Analytical Sophistication Model
Various analytical maturity models have been developed, including one by the staff at Gartner. Using an approach similar to Gartner's, we have created an extended sophistication model to clarify and further define analytics enablers, analytics, and advanced analytics (see Figure 1.1).
FIGURE 1.1 Analytical sophistication model
Analytics Enablers
In this model, the initial categories to the left of Descriptive Statistics (e.g., Raw Data, Clean Data, Standard Reports, and Ad Hoc Reports & OLAP) are related to data, dashboards, and historical reporting. These categories are not included in our definition of analytics. These categories are enablers of analytics but not analytics in and of themselves.
Analytics
In Figure 1.1, analytics begins with Descriptive Statistics. Analytics, as we are defining the term in this model, extends to, and ends at, the vertical dotted line encompassing Predictive Projects. Descriptive statistics provide insights into historical and current data and are valuable and useful tools in analytics projects and work. When an analytics team creates an exploratory data analysis (EDA) to investigate and gain a deeper understanding of a problem space, descriptive statistics play an integral role in examining and outlining how a business scenario operates in the real world and how that operation is described through data. An EDA and the descriptive statistics used provide a view into the factors that are at work and the relationship of those factors.
Analytics includes prescriptive projects as well. Organizations begin to experiment with their journey into more sophisticated analytics via prescriptive projects. These projects may illustrate that an organization has an appetite for advanced analytics, and it may show that this type of work is too costly, complicated, and difficult for the organization to embark upon as a sustained activity.
Advanced Analytics
Beginning at the vertical dotted line in the diagram and continuing to the right, all of the subsequent areas (i.e., predictive programs, prescriptive programs, simulation, and optimization) are encompassed in the category of advanced analytics. For the purposes of our discussion, all of AI, including both families of correlation and causal AI, exists in the portion of the analytical maturity model that is to the right of the vertical dotted line. In our discussion, advanced analytics and AI are synonymous.
Scope of Services to Support Causal AI
Causal AI must be viewed as part of the overall computing infrastructure for businesses. Therefore, it needs to be a consideration in the approaches to managing data and cloud services.
The hybrid cloud is a critical asset when moving to causal AI. One can argue that analytics can be managed locally. However, in the complex world of advanced analytics and more specifically causal AI, you have to assume that relevant data will come from a variety of sources—some will be third-party data sources managed in a public or private cloud. Other data may be generated by the Internet of Things (IoT) devices at the edge of the network. Creating a federated data approach that takes into account the fact that there are myriad data sources needed is critical to the success of advanced analytics in the context of causal AI.
In any discussion of advanced analytics, the scope of the data is critical to successfully approaching causal AI. For the past decade, data scientists have assumed that if you can collect enough data to create a model, you can make well-informed decisions. However, one of the distinctions between a data-first approach and a model-first approach is the ability to anticipate the type and scope of the data needed to understand relationships between variables and the data that supports establishing causal relationships. Being successful with causal AI requires the ability to discover the relevant data whether it is internal or external. Some of these data sources may be extremely large, and it would be impractical to move that much data to an internal data center.
Some of the early experiments with causal AI have failed because the business simply did not collect enough of the right data to answer why a problem occurred and what approaches will help to understand appropriate next actions. The data that you will need to create a causal AI solution will be varied. For example, there is considerable data that may be stored in data lakes. There will be data from systems of record as well as third-party data sources that are relevant to market trends. In some situations, your organization may have correct information, but there is simply not a large enough corpus of data to make relevant decisions. If an analytics team using a small data set draws conclusions from that data, the answers may be incorrect.
Keep in mind that building a causal AI model and solution is an iterative process requiring incrementally adding data that corresponds to variables and relationships modeled. For example, what data might you discover that is relevant to understanding a complex problem? If you were going to try to determine (in an agricultural setting) the optimal time to plant crops, you would need a variety of data ranging from the chemical construct of the soil, the changing weather patterns, and the past planting history. You might also need to understand what crops are selling better in a changing economic environment. If one of these factors is ignored, the resulting analysis will be useless. Data discovery and incrementally adding newly discovered data is critical to building effective causal AI solutions.
The hope has been that if we are able to analyze this data, we will be able to understand our world and understand how to transform our businesses. However, harnessing and making sense of data is not simple. As with any emerging technology innovation, moving from the idea of managing data to the reality of discovering solutions to complex problems is harder than anyone could have imagined.
There are many practical techniques for analyzing massive amounts of data that has helped organizations in many ways. Tools are available that help bring together many different types of structured and unstructured data to better understand the meaning of information, which has been a tremendous help to businesses. Data warehouses and data marts, for example, have enabled organizations to analyze business data to help make decisions, such as tracking transactions and managing operational data.
The Value of the Hybrid Team
While taking advantage of a team of professionals when successfully managing a data-focused project is critical to any project, there are requirements when approaching causal AI. We will cover the process of collaboration in a causal AI project in Chapter 5. Many early AI projects have failed when data scientists work in isolation from representatives from different areas of the business. It is particularly important in causal AI to involve subject-matter experts very early in the design process. These professionals understand the content and details of the operation that will become instrumental in creating a model that matches the problem as well as having hands-on experience with the most important data sources. Business strategists will help focus the team on the most important business problems that the company needs to address.
The Promise of AI
While AI was conceived of more than 60 years ago, it has been slow to realize true benefits for industry and business. The promise of artificial intelligence for decades has been the ability of machines to mimic the capacity of the human brain to understand the context of data and make quick decisions even when all the data you need is not available. One aspect of cognition that sets humans apart from machines is that humans typically maintain a mental model of our environment. Creating mental and computing models is one of the key processes to transforming data into a meaningful understanding of phenomena that we seek to explain or understand. A goal of AI is to be able to predict the behavior of individuals. It is no wonder that the best minds in artificial intelligence haven't been able to build reliable and accurate models of similar complexity to those that are inherent in the human brain.
It is now possible to develop sophisticated models; deploy those models in a range of operational environments; and treat, integrate, and ingest numerous types of data to produce reliable, scalable, and accurate results. However, none of these approaches go far enough to understand why events happen and how predictions are made. The value of causal AI is that it provides a technique for helping organizations to leverage the power of AI and advanced analytics so that organizations understand why events happen and how to come up with better solutions.
Traditionally, AI has been thought of as one large category. However, the reality is that to be successful, there needs to be a more nuanced and multifaceted approach to AI. Commercial firms, regulators, and academic organizations—to name a few—are interested in and eager to work with a new class of AI. This new category of AI needs to provide for transparency in data transformation, model development, and, most importantly, the interpretation of the results produced by AI systems. This is one of the primary reasons why we see that causal AI and causal inference are moving out of research labs and the halls of academia into the commercial market.
Almost since the beginning of time, humans have been searching for the reasons why things happen. Once statistics became associated with correlation, there was an effort to disregard the power of causality. However, with the requirement to understand why an event has happened, what it means, and how to change outcomes—causality has emerged as a transformative trend.
As we develop a broader and deeper view of causality throughout the book, we will provide definitions and employ a business-oriented use case so that data scientists, subject-matter experts, data analysts, managers, and business leaders can have a common vocabulary to understand causal AI and how it is important to implement business solutions that will help the organization move forward.
Understanding the Core Concepts of Causal AI
As we move through the chapters in this book, we will get into more and more detail about how causal AI works and the elements that are important to understand. In this next section, we will provide an overview of the key concepts that are required to understand causal AI.
Explainability and Bias Detection
As AI has evolved over the last several decades, models have become more sophisticated, and, in some cases, multiple models are implemented in an analytics pipeline where the models interact and influence the results of the overall pipeline and the individual models. Monitoring and understanding multiple models is challenging and can be difficult to discern the influence of any one model on the overall results or outcomes.
In addition to ensembles of models running simultaneously and, in some cases, in parallel, certain analytical techniques in traditional AI rely on algorithms that are very difficult to examine, discern, and understand why certain results or outcomes are produced. Algorithms such as neural networks belong to a class of analytical technical that are referred to as black boxes. These class models have been given this label because the designs of the resulting models and processes running within the models are so complicated that it has been nearly impossible for humans to understand how the input variables are combined to reach the ultimate conclusions.
Academics, researchers, and commercial technology companies have been working to develop solutions to provide explainability solutions for neural networks. Over the past 5 years, we have seen solid progress made on providing clear explanations of how neural networks process data and why those models generate the outcomes and results that are seen. It will be a few years before we see and can use explainability technology that will be acceptable to government regulators and governance professionals.
A black-box model executes its analytical process based on how it interprets the data. The black-box model will not be able to provide business managers with an explanation about how the model makes decisions as to why a campaign did not work. As a result, data business managers cannot explain to shareholders or government regulators why certain business decisions were made. This traditional approach to AI lacks two important factors that are imperative for business success: explainability and fairness or identification of bias.
Explainability
