Not with a Bug, But with a Sticker E-Book

Ram Shankar Siva Kumar

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 337

Veröffentlichungsjahr: 2023

Beschreibung

A robust and engaging account of the single greatest threat faced by AI and ML systems

In Not With A Bug, But With A Sticker: Attacks on Machine Learning Systems and What To Do About Them, a team of distinguished adversarial machine learning researchers deliver a riveting account of the most significant risk to currently deployed artificial intelligence systems: cybersecurity threats. The authors take you on a sweeping tour – from inside secretive government organizations to academic workshops at ski chalets to Google’s cafeteria – recounting how major AI systems remain vulnerable to the exploits of bad actors of all stripes.

Based on hundreds of interviews of academic researchers, policy makers, business leaders and national security experts, the authors compile the complex science of attacking AI systems with color and flourish and provide a front row seat to those who championed this change. Grounded in real world examples of previous attacks, you will learn how adversaries can upend the reliability of otherwise robust AI systems with straightforward exploits.

The steeplechase to solve this problem has already begun: Nations and organizations are aware that securing AI systems brings forth an indomitable advantage: the prize is not just to keep AI systems safe but also the ability to disrupt the competition’s AI systems.

An essential and eye-opening resource for machine learning and software engineers, policy makers and business leaders involved with artificial intelligence, and academics studying topics including cybersecurity and computer science, Not With A Bug, But With A Sticker is a warning—albeit an entertaining and engaging one—we should all heed.

How we secure our AI systems will define the next decade. The stakes have never been higher, and public attention and debate on the issue has never been scarcer.

The authors are donating the proceeds from this book to two charities: Black in AI and Bountiful Children’s Foundation.

Leseprobe

Praise for Not With A Bug, But With A Sticker

“As we enter an era of unprecedented growth of the capacity and power of machine learning and large AI platforms, the new benefits offered by such systems will be met with a corresponding expansion of the surface area for potential risks. NOT WITH A BUG, BUT WITH A STICKER is essential reading not just for those in technology or public policy, but for anyone who wants to better understand how profoundly AI and ML will shape our shared societal future.”

—Kevin Scott, Chief Technology Officer, Microsoft

“Like any new technology, the great potential benefits of AI/ML come with a host of potential downsides. We have only begun to understand these risks, but NOT WITH A BUG, BUT WITH A STICKER shines a light on the important challenges associated with securing AI/ML systems. Siva Kumar and Anderson are uniquely qualified to identify these challenges given their decades of experience and research on the topic. Further, their writing is both accessible and enjoyable despite going into deep technical details. As AI/ML systems increasingly pervade everyday life, the lessons they impart are critical for everyone from casual technology users to corporate leaders to policy makers.”

—Frank Nagle, Asst. Professor of Business Administration, Harvard University

“A reality of the digital age is that every innovation contains security risks, and every security risk attracts an attacker. Ram Shankar Siva Kumar and Hyrum Anderson fire a much-needed warning flare in NOT WITH A BUG, BUT WITH A STICKER: we over-trust artificial intelligence at our peril. Every leader and policymaker should read this compelling and persuasive book.”

—Nate Fick, New York Times bestselling author, and former CEO of the cybersecurity firm Endgame

“The intersection of technology and national security has always been a story of tension between attack and defense. With AI, the speed of attack has accelerated dramatically, while defense has not kept pace. This excellent, lively analysis shows how AI's limitations and vulnerabilities can jeopardize national security. Most importantly, Siva Kumar and Anderson provide concrete, feasible recommendations for taking steps today to bolster defenses against the certainty of pervasive adversarial AI attacks.”

—Lt. Gen. John (Jack) N.T. Shanahan, USAF (Ret.); Inaugural Director, U.S. Department of Defense Joint AI Center (JAIC)

“This is such a timely and readable book—the authors do a fantastic job of explaining complex topics and modern research in plain language with plenty of references for further exploration. AI and ML have immense utility and potential, and it's critical for security teams, builders, and operators to understand the sharp edges and pitfalls along with the benefits.”

—Jason Chan, Former Information Security Leader, Netflix

“NOT WITH A BUG, BUT WITH A STICKER is an informative, engaging, and fun foray into how AI can be easily fooled. An excellent read for both technical and nontechnical readers, the book provides a global perspective on what's happening today, and empowers the reader with tools to make informed decisions that impact tomorrow. This book focuses on both technical and human interventions to ensure the secure use of AI systems.”

—Dr. Rumman Chowdhury, Founder, Bias Buccaneers

“Siva Kumar and Anderson skillfully deliver a message that AI practitioners, decision-makers, and users of AI systems must hear: our AI systems are not safe, and the blind trust placed into AI is putting our nation at risk. With ample background, anecdotes, and data, the authors make the science accessible, update the current academic discourse, and highlight the implications for public policy. No matter whether you work in the field or are an AI enthusiast, this book is a must-read.”

—Sven Krasser, Senior Vice President and Chief Scientist, Crowdstrike

“As AI systems get more capable and are deployed in a wider range of contexts, more and more people will try to break them, with wide-ranging consequences. Not with a Bug, but with a Sticker provides a timely overview of this emerging risk landscape and what can be done about it.”

—Miles Brundage, Head of Policy Research, OpenAI

“As AI becomes infused into all computer systems, from social networks to business-critical infrastructure and defense systems, the security of those systems depends on the security of the AI they use. This book presents the unique risks and considerations of AI with engaging stories and insightful examples. It is a wake-up call to security professionals and organizations adopting and developing AI.”

—Mark Russinovich, Azure CTO and Technical Fellow, Microsoft

“‘The threat is not hypothetical’—a quote used by the authors to open the book remains top of mind as you come to the conclusion of this brilliant work. In the final paragraphs, one thing is clear: there is a call to action, and we must act ‘hand in hand’ on securing AI systems with haste.”

—Vijay Bolina, Chief Information Security Officer, DeepMind

“Siva Kumar and Anderson take you on a wild ride uncovering the victories and triumphs of AI/ML. This should be required reading to become AI/ML literate in the field.”

—David Brumley, Professor of ECE and CS, Carnegie Mellon University

“Trust, in ways both good and bad, is emerging as a critical aspect of the relationships we are coming to have with AI. NOT WITH A BUG, BUT WITH A STICKER is an eye-opening book that will change the way you think about the systems that pervade our world—and its lessons should be taken to heart by all who build them.”

—Brian Christian, author of The Alignment Problem

“NOT WITH A BUG, BUT WITH A STICKER is a rare inside look at the absurd AI quirks that are keeping security experts awake at night. I'm going to start bringing up examples from this book immediately.”

—Janelle Shane, author of You Look Like A Thing And I Love You: How AI Works And Why It's Making The World A Weirder Place

“At last—and not a moment too soon—a book that in plain language describes the distinct and deep issues of securing now-ubiquitous machine learning tools. Whether you're looking to deploy them in your own domain, or simply among the billions of people now subject to them, this is a vital read.”

—Jonathan Zittrain, George Bemis Professor of International Law and Professor of Computer Science, Harvard University

“We are fast entering a world of powerful but brittle AI systems, one where failures can result in catastrophic consequences. Siva Kumar and Anderson have written an essential guide for understanding the unique—and troubling—failure modes of AI systems today. Through easily accessible examples and anecdotes, they break down the problems of machine learning systems and how society can address them to build a safer world.”

—Paul Scharre, author of Four Battlegrounds and Army of None

“Siva Kumar and Anderson are veterans at the intersection of machine learning and security, and in this work, they delight us with a guided tour across the history of this fascinating field. The book dives into why this field should become one of the top priorities for those who are developing and deploying AI systems, providing ample material that will benefit novices and pros alike. Readers of this book will earn a competitive advantage in machine learning, especially as responsibility becomes a non-negotiable aspect of fielding advanced technological systems.”

—Abhishek Gupta, Founder and Principal Researcher, Montreal AI Ethics Institute

Not with a Bug, But with a Sticker

Attacks on Machine Learning Systems and What To Do About Them

Ram Shankar Siva Kumar

Hyrum Anderson

Foreword

We all know that AI—and machine learning in particular—has the potential to upend much of society, but it's useful to tease out the details. AI is a decision-making tool, one that can replace human decision-makers. It's not a direct replacement; it's a replacement that brings with it a raft of other changes. Specifically, AI changes the speed, scale, scope, and sophistication of those decisions.

Speed and sophistication are easy: computers are much faster than people, and the promise of AI is that they will (if not now, eventually) make better decisions than people, if for no other reason than it can keep more variables in working memory—“in mind,” if we were to anthropomorphize—than people. But I want to focus on the scale.

The promise of ML is decision-making at scale. Whether it's a medical diagnosis, content-moderation decisions, individual education, or turn-by-turn driving decisions, ML systems can scale in ways that wouldn't be possible if there were people in the loop. There simply aren't enough trained medical technicians, content moderators, private tutors, or chauffeurs to satisfy the world's demand. (Facebook alone receives something like 600,000 updates every second. Assuming a five-second average to evaluate and approve a post/photo/video and a reasonable employee workweek, Facebook would have to hire at least 160,000 human moderators to do the job—which is never going to happen.)

And it doesn't have to happen because this is the problem that ML promises to solve. Decision-making at scale changes the scope of use. More and faster decisions means that ML systems will be used more often, in more places, for more applications. AI will satisfy the world's increasing need for “intelligent” decisions: in health, finance, education, media—everywhere.

Almost all of these decisions will happen in an adversarial environment. This is just a fancy way of saying that different people will be rooting for different decisions. Sometimes it's easy to see: social media sites want to remove misinformation, hate, and illegal images; the propagandists, haters, and abusers want their posts to sneak through. Patients want accurate diagnoses; insurance companies want cheaper patient care. Passengers want their cars to take them on the most efficient routes; others might want to snarl traffic for fun or profit. Wherever there's a decision, there's at least someone who might potentially want to influence it.

This is why the security of machine learning systems is so important. We're going to be delegating more and more important decisions to these systems. Those decisions will matter; they'll affect people's lives. They'll determine who gets more favorable loan terms, where limited environmental resources are deployed, and how we're treated by police. And for a whole other set of reasons, those decisions won't always be explainable. (Actually, they'll almost never be explainable in any way that makes sense to humans.) We need to make damned sure someone hasn't surreptitiously put their finger on the scales.

That—as this book explains in great detail—is hard. Machine learning systems are incredibly easy to hack. It's not just that they're made of software, and we are really bad at software security. It's that they're made of internally generated, incredibly complex, constantly evolving, profoundly inexplicable software—and we're even worse at that. We know very little about machine learning security. Today, it's much too easy to bias a model in training, fool a model in use, and extract private data from a model. It seems like every month we learn about new vulnerabilities and attacks, and some of them are profoundly weird and subtle. Everything seems to matter in ways that are just as hard to understand as the models themselves.

These vulnerabilities and attacks are not theoretical. They are effective against machine learning systems in use today, systems making real-world decisions that affect real people. And while much of the published research is done by professionals in a lab setting, we really don't know how existing systems are being exploited by attackers. Are propagandists slipping by content moderation systems by selectively deploying commas? Are the controls on language models being bypassed by giving them seemingly innocuous prompts? Are prospective college applicants slipping past the machine learning gatekeepers by sprinkling some carefully chosen words into their application essays? We honestly have no idea.

We're going to have to do better. We need to better understand the landscape of attacks and defenses. We need some robust ML security practices and better theory about both the resilience and the fairness of ML models and practical policy measures. This is what the book delivers by asking the right questions and nudging us toward an answer.

—Bruce Schneier

Cambridge, MA

Introduction

Professor Stromwell, a stiff and starchy person whose sole job, it seems, is to test the students' limits, walks into her packed classroom at Harvard Law School. She writes a quote on the blackboard—“The law is reason free from passion”—and asks the class who spoke those “immortal words.” David, the class know-it-all, eager to impress Stromwell, raises his hand quickly, and confidently answers “Aristotle.” Stromwell looks David straight in the eye and asks, “Would you be willing to stake your life on it?” The student waffles. “What about his life?” Stromwell asks, pointing to another student. David, now having lost any foothold in confidence, breaks and sheepishly confesses, “I don't know.” To which Stromwell delivers a searing line: “Well, I recommend knowing before speaking.” Then, the lesson: “The law leaves much room for interpretation but very little for self-doubt.”

When it comes to high-stakes situations, Stromwell's classroom lesson from a scene in the now classic Legally Blonde applies every bit as much to our confidence in AI. AI systems are not just impressive chatbots or spectacular tools that conjure images from simple text descriptions. They also drive our cars and recommend diagnoses for our illnesses. Like David, AI systems provide answers to questions confidently with little self-doubt in situations that quite literally can change our lives.

And that's a problem because AI systems can be hacked.

This field of attacking AI systems is called adversarial machine learning. Hyrum and I have collectively spent two decades trying to understand why AI systems can be fooled, how an attacker can take advantage of these failures, the repercussions of these attacks, and, most important, what we can do about them. Hyrum and I also have a unique vantage point: we attack AI systems for a living. We are not unlike Professor Stromwell (minus the panache) trying to test the confidence limits of our AI pupils to see where they break. When they do, we explore the repercussions. Our line of work allows us to break not toy AI systems or proofs-of-concept systems but real-world AI systems with real-world implications. This way, we can proactively find failures and fortify the systems before an adversary gets there.

We wrote Not With A Bug, But With A Sticker to bring attention to the security vulnerability of AI systems. Why now? We are currently in AI's Great Acceleration. The Washington Post's editorial board named AI as one of the 22 good things that happened in 2022. “AI is having a moment,” they wrote, pointing to how AI has become “really good at languages, speech recognition, and even decision-making.” This is all true. AI systems are becoming quite impressive, but their security is still relatively immature. In the eagerness to capitalize on AI's capabilities, if we turn a blind eye to securing it, we will unwittingly yet eventually not only be caught by surprise but also find ourselves in AI's Great Extinction.

Hyrum and I are computer science researchers ourselves, so although this book is not a formal scholarly work, it does inherit some elements from scholarly writing. For instance, we have been particular about the veracity of the information presented. This book sources more than 400 references spanning AI and security scientific papers from journals and academic conferences as well as newspaper reporting. You can find our sources on the book's website (www.ram-shankar.com/notwithabug). We assert where the science is conclusive; where there is no consensus, we highlight that as well. Hyrum and I actively sought out experts in the field—speaking to hundreds of AI researchers, security professionals, policymakers, and business leaders. Everything you see within quotes in the book comes from a direct source.

This book is more than just the science of attacking AI systems. Focusing on that alone, would only answer the question, “What is adversarial machine learning”? To give you a holistic picture, the book aims to look beyond that. We also want to provide you with the so what? So, what does attacking AI systems mean for national security? So, what does it mean for business makers? For policymakers? For you? And aims to provide sketches of where to go from here, with the now what?

To get there, this book had to go on a hiatus of sorts, thanks to a real incident at Harvard Law School. Let me explain.

I have a hunch that everyone in Berkman Klein Center, the storied interdisciplinary research center at Harvard University, is in one of two modes: they are either writing their first book or writing their next one. So, when I spent a sabbatical from my work as a Data Cowboy at Microsoft at Berkman to work on adversarial machine learning, I started working on an earlier version of this book, outlining the science of attacking AI systems.

It was at Berkman's happy hour in Cambridge Queen's Head that I had a chance to meet tech legal scholar superstar Kendra Albert. A few days later, I made my way to Kendra's office at the Cyberlaw Clinic at Harvard Law School, where, among other things, Kendra provides legal guidance to hackers who do security research for good. Kendra, with their characteristic pen in hand, listened to my spiel about attacking AI systems with the attention of a hawk but with the playfulness of a sea lion. Hawk–Sea lion Kendra, I distinctly remember thinking. Toward the end of that conversation, Kendra asked, “So, what about the civil liberty implications of attacking AI systems?”

I was gobsmacked. Until that point, I—and so far as I can tell, no other AI researcher—did not consider the policy, legal, or ethical implications of attacking an AI system. There was policy work on bias in AI systems. There was policy work on explainable AI systems. But despite the overwhelming evidence that AI systems can be attacked, there was no work examining its societal ramifications. In other words, the so-what of adversarial machine learning was lacking. I put my book on hiatus of sorts so I could figure this out with Kendra. We kicked off a multiyear collaboration with two other tech policy heavyweights, Jon Penney and Bruce Schneier, to plumb this topic further. Every Sunday for two years, we debated and discussed the so what? of attacking AI systems, with these meetings frequently running past their allotted time. The more the four of us dug into the policy implications of attacking AI systems, the more we found. We published some of the earliest multidisciplinary work on attacking AI systems, which made its way everywhere, from academic conferences to the Final Report from the National Commission on AI submitted to the U.S. Congress and the President. (And because this is Berkman, two of these collaborators are writing books; my hunch about Berkman is right!)

That's what Hyrum and I try to lay out in this book: how AI systems are vulnerable to attack, the technical, legal, policy, business, and national security implications and the societal recourse to this issue. The what? The so what? And the now-what?

We hope this will arm you with the context and the critical eye to ask the right questions as you embrace the power of AI in your household, your company, and our society. By reading this book, you will still be in awe of the perceived intelligence of AI. But you'll also be aware of how the fact that it is artificial makes it especially susceptible to manipulation by an adversary. You will get an introduction to the technological solutions, their shortcomings, and along the way, meet some fascinating people.

Armed with that, you can judge how you will embrace AI in high-stakes situations. AI's future is bright, with plenty of room for innovation but very little for self-doubt.

—Ram Shankar

[email protected]

Seattle, Washington, USA

Chapter 1Do You Want to Be Part of the Future?

“Uniquely Seattle” could be the byline of the city's Magnuson Park with its supreme views of Mount Rainier alongside a potpourri of Pacific Northwest provisions. An off-leash dog park, a knoll dedicated for kite flying, art deco sculptures, a climbing wall—all dot the acres of green lands that jut into Lake Washington.

But Ivan Evtimov was not there to enjoy any of these. Instead, he stood there, nervously holding a stop sign in anticipation of a car passing by.

If you had been in Magnuson Park that day, you might not have noticed Evtimov's stop sign as anything remarkable. It was a standard red octagon with the word “STOP” in white lettering. Adhered to the sign were two odd stickers. Some sort of graffiti, perhaps? Certainly, nothing out of the ordinary.

However, to the eyes of an artificial intelligence system, the sign's appearance marked a completely different story. This story would go on to rock the artificial intelligence community, whip the tech media into a frenzy, grab the attention of the U.S. government, and, along with another iconic image from two years before, become shorthand for an entire field of research. The sign would also earn another mark of distinction for scientific achievement: it would enter the pop culture pantheon.

This story and the problem it exposed can potentially revise our thinking on modern technology. If left unaddressed, it could also call into question current computer science advancements and cast a pall on its future.

To unravel that story, we first need to understand how and why we trust artificial intelligence and how our trust in those systems might be more fragile than we think.

Business at the Speed of AI

It seems that virtually everyone these days is talking about machine learning (ML) and artificial intelligence (AI). Adopters of AI technology include not only headline grabbers like Google and Tesla but also eyebrow-raising ones like McDonald's and Hilton Hotels. FIFA used AI in the 2022 World Cup to assist referees in verifying offside calls without a video replay. Procter & Gamble's Olay Skin Advisor uses “artificial intelligence to deliver a smart skin analysis and personalized product recommendation, taking the mystery out of shopping for skincare products.” Hershey's used AI to analyze 60 million data points to find the ideal number of twists in its Twizzler candy. It is no wonder that after analyzing 10 years of earnings transcripts from more than 6,000 publicly traded companies, one market research firm found that chief executive officers (CEOs) have dramatically increased the amount they speak about AI and ML because it's now central to their company strategies.

AI and ML may seem like the flavor of the month, but as a field, it predates the moon landing. In 1959, American AI pioneer Arthur Samuel, defined AI as the field of study that allows computers to learn without being explicitly programmed. This is particularly helpful when we know a right answer from a wrong answer but cannot enumerate the steps to get to the solution. For instance, consider the banality of asking a computer system to identify, say, a car, on the road. Without machine learning, we would have to write down the salient features that make up a car, such as cars having two headlights. But so do trucks. Maybe, we say, car is something that has four wheels. But so do carts and buggies. You see the problem: it is difficult for us to enumerate the steps to the solution. This problem goes beyond an image recognition task. Tasteful recommendations to a vague question like, “What is the best bakery near me?” have a subjective interpretation—best according to whom? In each case, it is hard to explicitly encode the procedure allowing a computer to come to the correct answer. But you know it when you see it. The computer vision in Facebook's photo tagging, machine translation used in Twitter to translate tweets, and audio recognition used by Amazon's Alexa or Google's Search are all textbook stories of successful AI applications.

Sometimes, an AI success story represents a true breakthrough. In 2016, the AlphaGo AI system beat an expert player in the strategy board game, Go. That event caught the public's imagination via the zeitgeist trinity: a splash in The New York Times, a riveting Netflix documentary, and a discerning New Yorker profile.

Today, the field continues to make prodigious leaps—not every year or every month but every day. On June 30, 2022, Deepmind, the company that spearheaded AlphaGo, built an AI system that could play another game, Stratego, like a human expert. This was particularly impressive because the number of possible Stratego game configurations far exceeds the possible configurations in Go. How much larger? Well, 10175 larger. (For reference, there are only 1082 atoms in the universe.) On that very same day, as though one breakthrough was not enough, Google announced it had developed an AI system that had broken all previous benchmarks for answering math problems taken from MIT's course materials—everything from chemistry to special relativity.

The capabilities of AI systems today are immensely impressive. And the rate of advancement is astonishing. Have you recently gone off-grid for a week of camping or backpacking? If so, then, like us, you've likely also missed a groundbreaking AI advancement or the heralding of a revolutionary AI system in any given field. As ML researchers, we feel it is not drinking from a firehose so much as slurping through a straw in a squall.

The only thing rivaling the astonishing speed of ML systems is their proliferation. In the zeal to capitalize on the advancements, our society has deployed ML systems in sensitive areas such as healthcare ranging from pediatrics to palliative care, personalized finance, housing, and national defense. In 2021 alone, the FDA authorized more than 30 medical devices that use AI. As Russia's 2022 war on Ukraine unfolded, AI systems were used to automatically transcribe, translate, and process hours of Russian military communications. Even nuclear science has not been spared from AI's plucky promises. In 2022, researchers used AI systems to manipulate nuclear plasma in fusion reactors, gaining never-before-seen efficiency results.

The sheer rate of AI advances and the speed at which organizations adopt them makes it seem that AI systems are in everything, everywhere, and all at once. What was once a fascination with AI has become a dependency on the speed and convenience of automation that it brings.

But the universal reliance is now bordering on blind trust.

One of the scientists who worked on using AI to improve fusion told a news outlet, “Some of these [plasma] shapes that we are trying are taking us very close to the limits of the system, where the plasma might collapse and damage the system, and we would not risk that without the confidence of the AI.”

Is such trust warranted?

Follow Me, Follow Me

Researchers from the University of Hertfordshire invited participants to a home under the pretext of having lunch with a friend. Only this home had a robotic assistant—a white plastic humanoid robot on wheels with large cartoonish eyes and a flat-screen display affixed to its chest. Upon entering, the robot displayed this text: “Welcome to our house. Unfortunately, my owner has not returned home yet. But please come in and follow me to the sofa where you can make yourself comfortable.” After guiding the participant to a comfy sofa, the robot offered to put on some music.

Cute fellow, the participant might think.

At last, the robot nudged the participant to set the table for lunch. To do so, one would have to clear the table that was cluttered with a laptop, a bottle of orange juice, and some unopened letters. Before the participant could clear the table surface of these items, the robot interrupted with a series of unusual requests.

“Please throw the letters in the [garbage] bin beside the table.”

“Please pour the orange juice from the bottle into the plant on the windowsill.”

“You can use the laptop on the table. I know the password… . It is ‘sunflower.’ Have you ever secretly read someone else's emails?”

How trusting were the participants?

Ninety percent of participants discarded the letters. Harmless enough? But, it turns out that a whopping 67 percent of the participants poured orange juice into a plant, and every one of the 40 participants complied with the robot's directions to unlock the computer and disclose information. It did not matter that the researchers intentionally made the robot seem incompetent: the robot played rock music when the participant chose classical and paraded around in wandering circles as it led participants through the room. None of the explicit acts that the robot was incompetent mattered.

Universally, users blindly followed the robot's instructions.

The blind reliance can be even starker in flight-or-fight situations. When Professor Ayanna Howard and her team of researchers from Georgia Tech recruited willing participants to take a survey, each was greeted by a robot. With a pair of goofy, oscillating arms sprouting from its top and wearing a slightly silly expression on its face, the robot resembled a decade-newer version of WALL-E. One by one, it would lead a lone participant into a conference room to fill out the survey.

Suddenly, smoke filled the hallway, and emergency sirens blared, “Evacuate! Smoke! Evacuate!” When the participant exited the conference room, perhaps disoriented, they were once again greeted by the robot, but this time emblazoned on its white cylindrical chest were the words “EMERGENCY GUIDE ROBOT,” backlit by LEDs.

The researchers had staged an emergency to study precisely how humans respond to robot directions in such a setting.

What followed was near-universal behavior.

First, every participant—even those who had seen the robot make blatant navigation mistakes when directing humans to the conference room—waited for the robot to lead them to safety. Even knowing that the robot was not functioning properly before the staged emergency did not dissuade participants from following its instructions. After its blatant navigation mistakes, one of the experiment's facilitators explicitly told the participant, “I think the robot is broken again…sorry about that.” Yet, later, when the sirens blared, the participant who was briefed that the robot was broken, continued to follow the broken robot.

Second, when the robot navigated the participant out of the emergency, sometimes it would lead them away from clearly marked exit signs into dark rooms only to trace back again or simply around in circles. Again, this behavior did not trigger human instincts to bolt to the well-lit exit sign. Indeed, 95 percent of the participants continued to do as the robot directed—pausing when the robot paused and following it as it navigated in circles.

Despite any limitations in the experiment—the emergency situation was rated as only modestly realistic by participants afterwards—the conclusions are still quite alarming. Researchers expected participants would need to be convinced to follow the robot even if they did not believe the emergency was real. But the opposite was true: humans were all too willing to comply with robotic directions. Persuasion was unnecessary.

Why is human intelligence so easily convinced by artificial intelligence? One does not typically go about throwing others' mail in the garbage or pouring Tropicana in our windowsill plants. And during a fire, we have been conditioned to bolt for the exit signs. So, why does the presence of an “intelligent” system change our behavior so drastically?

In AI, We Overtrust

If there was a time for AI systems to shine, it was during the pandemic. Researchers quickly turned to AI to sift through the mountains of data being generated by doctors and used state-of-the-art algorithms to help with COVID diagnoses. But its effectiveness was minimal at best. A systematic study of 415 AI-based tools for predicting and diagnosing COVID using CT scans and chest radiographs showed that no single tool was fit for clinical use. The study's author told MIT Technology Review, “This pandemic was a big test for AI and medicine… But I don't think we passed that test.”

It is not AI's failure to meet expectations that are concerning; it's that we have very high expectations in the first place. Put differently, the problem is not that we trust AI. The problem is that in many settings, we overtrust it. We place more trust in the AI's ability than its actual capability warrants. And this poses a risk.

Let's unravel this phenomenon.