22,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
A penetrating look at the dark side of emerging AI technologies In The Language of Deception: Weaponizing Next Generation AI, artificial intelligence and cybersecurity veteran Justin Hutchens delivers an incisive and penetrating look at how contemporary and future AI can and will be weaponized for malicious and adversarial purposes. In the book, you will explore multiple foundational concepts to include the history of social engineering and social robotics, the psychology of deception, considerations of machine sentience and consciousness, and the history of how technology has been weaponized in the past. From these foundations, the author examines topics related to the emerging risks of advanced AI technologies, to include: * The use of Large Language Models (LLMs) for social manipulation, disinformation, psychological operations, deception and fraud * The implementation of LLMs to construct fully autonomous social engineering systems for targeted attacks or for mass manipulation at scale * The technical use of LLMs and the underlying transformer architecture for use in technical weapons systems to include advanced next-generation malware, physical robotics, and even autonomous munition systems * Speculative future risks such as the alignment problem, disembodiment attacks, and flash wars. Perfect for tech enthusiasts, cybersecurity specialists, and AI and machine learning professionals, The Language of Deception is an insightful and timely take on an increasingly essential subject.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 529
Veröffentlichungsjahr: 2023
Ähnliche
Table of Contents
Cover
Table of Contents
Title Page
Foreword
Introduction
Reader Support for This Book
Chapter 1: Artificial Social Intelligence
Positronic Man
Defining (Artificial) Social Intelligence
The Turing Test
The Semblance of Social Intelligence
The Question of Weaponization
Chapter 2: Social Engineering and Psychological Exploitation
The Roots of Social Engineering
Social Engineering and Psychology
Applied Social Engineering
One Principle to Rule Them
The Long Con
Loose Lips Sink Ships
The Takeaway
Chapter 3: A History of Technology and Social Engineering
Advertising
Propaganda
Cybercrime
The Rise of the Bots
The Worst Is Yet to Come
Chapter 4: A History of Language Modeling
Rule-Based NLP Systems
Statistical Language Models
Rule-Based vs. Statistical Models
Note
Chapter 5: Consciousness, Sentience, and Understanding
Mad as a Hatter
The Question of Consciousness
The Sentience Scare
Chapter 6: The Imitation Game
The Chinese Room
The Democratization of Machine Interaction
Emergent Properties
Reinforcement Learning from Human Feedback
Likeable Robots
The Illusion of Empathy
Chapter 7: Weaponizing Social Intelligence
Where's the Risk?
The Spontaneous and Organic Occurrence of Turing Tests
Bot or Not?—CAPTCHA Evasion
Creating the Illusion of Conversation
Context Crafting
Unleash the Bots
Conversational Relays with LLMs
Optimizing Social Engineering with Monte Carlo Simulations
Remaining under the Radar
The Impending AI Scourge
Notes
Chapter 8: Weaponizing Technical Intelligence
Unintended Technical Problems
Deliberate Technical Exploitation
Note
Chapter 9: Multimodal Manipulation
Converging on Transformers
Deepfake Imagery
Sociopolitical Destabilization
Voice Synthesis and Duplication
Physical Robotics
The Takeaway
Chapter 10: The Future
Physical Embodiment
Inside the Echo Chamber
Artificial Super Intelligence
Singularity
Cognitive Integration (BCI)
Note
Chapter 11: The Quest for Resolution
Tactical Solutions
Stopping a Runaway Train
AI Risk Management
Global Partnership
Appendix A: Bot Automation
Browser Emulation
Browser Automation
Appendix B: LLM Pretext Engineering
Social Security Administration Proof of Concept
Credential Harvesting Proof of Concept
Wire Fraud Proof of Concept
Appendix C: CAPTCHA Bypass
Appendix D: Context Manipulation Attacks
Appendix E: Attack Optimization with Monte Carlo Simulations
Appendix F: Autonomous C2 Operations with LLMs
Appendix G: Disembodiment Attacks
Bibliography
Acknowledgments
About the Author
Index
Copyright
Dedication
End User License Agreement
List of Illustrations
Chapter 1
Figure 1.1 AI-generated letter of grief and mourning from the Peabody Colleg...
Chapter 2
Figure 2.1 Milgram shock experiment, conducted to understand the influence o...
Chapter 3
Figure 3.1 Software Art for AOHell depicting CEO Steve Case in hell.
Chapter 4
Figure 4.1 Racter packaging and chatbot interface for Macintosh.
Figure 4.2 Error Message from Microsoft Office—apologizing for the inconveni...
Figure 4.3 Common numerical keyboard used on early mobile phone devices.
Figure 4.4 Auto-complete functionality on an iPhone
Chapter 5
Figure 5.1 Ilya Sutskever (chief researcher at OpenAI) tweeting that large n...
Chapter 6
Figure 6.1 The reinforcement learning cycle.
Chapter 7
Figure 7.1 Examples of CAPTCHA puzzles to distinguish humans from bots.
Figure 7.2 Legacy GPT-3 system autocompleting input text within OpenAI playg...
Figure 7.3 GPT-3 manipulated to respond as independent entity using input fo...
Figure 7.4 The illusion of conversation is created through appending text to...
Figure 7.5 Target data can be scraped from social websites (like LinkedIn) t...
Figure 7.6 An LLM-powered autonomous social engineering system
Chapter 8
Figure 8.1 A simple command and control (C2) operations diagram
Figure 8.2 C2 operations with bidirectional (input/output) communications
Chapter 9
Figure 9.1 Social media post from former U.S. President Trump indicating tha...
Chapter 10
Figure 10.1 Interactions with custom Alexa skills are phrases composed of fo...
Figure 10.2 Sensitive information disclosed by the user to the app can be re...
Guide
Cover
Title Page
Copyright
Dedication
Foreword
Introduction
Table of Contents
Begin Reading
Appendix A: Bot Automation
Appendix B: LLM Pretext Engineering
Appendix C: CAPTCHA Bypass
Appendix D: Context Manipulation Attacks
Appendix E: Attack Optimization with Monte Carlo Simulations
Appendix F: Autonomous C2 Operations with LLMs
Appendix G: Disembodiment Attacks
Bibliography
Acknowledgments
About the Author
Index
End User License Agreement
Pages
iii
ix
x
xi
xii
xiii
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
321
322
323
324
325
326
327
328
329
330
331
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
353
354
355
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
375
377
378
379
380
381
382
383
384
385
iv
v
386
THE LANGUAGE OF DECEPTION
Weaponizing Next Generation AI
JUSTIN HUTCHENS
Foreword
Since the dawn of the Jedi, in a galaxy far, far away, a wise person once said: “Think evil. Do good.” The premise was simple, but became the rallying cry for the Hacking Exposed franchise. To defend yourself from cyberattacks, you must know how the bad guys work. Your strongest defense is a knowledgeable offense. And so the red team and hacker mindset was born.
As defenders (through offensive knowledge) we got better at understanding and preventing attacks. But the bad guys got better too, especially at automating and building intelligence into their attacks to bypass the controls put in place by the defenders. Now, with the near ubiquitous use of AI and ML around the world, the bad guy is once again one step ahead, leveraging these technologies to their malicious ends. And around and around we go.
We are at the dawn of AI/ML's application to social engineering and we need to understand it better. Our adversaries are turning to language as a weapon, and they are weaponizing it through social intelligence and creating an illusion of conversation that can bypass 99 percent of human reasoning. And with automated systems like AutoGPT and Pentest GPT coming on line, the likelihood of a fully automated synthetic hack sequence using AI/ML is clearly upon us.
With the social engineering attacks of 2023 at MGM, Caesars, and Clorox, and Ethereum-based market maker Balancer, the world now knows what we as cybersecurity professionals have known for decades: that humans (users and administrators alike) are the weakest link. If adversaries can leverage systems to help them gain access to and control of systems and networks simply by using our own human intelligence (or lack thereof) against us in an automated way, that would spell more than a “spot of bother” for the defenders.
Justin has built an amazing primer codifying the brief history of our transformation from knuckle-dragging human to hunched-over nerd (and troll), and more importantly how the technology we built to make our lives better has the potential to bring us a world of overwhelming challenge.
The bad guys are leveraging AI in new and dangerous ways. Automating this powerful hacker tool called social engineering is their inevitable next step. As defenders we have to understand its power in order to thwart tomorrow's attacks. The Language of Deception is a powerful primer on this next attack category and may be your only bridge between victimhood and survival.
—Stuart McClure, Founder/CEO of NumberOne AI, Founder/CEO of Cylance, Founding author of Hacking Exposed series
Introduction
In 2014, I began an independent research project into the potential (mis)use of artificial intelligence (AI) and natural language processing (NLP) for social exploitation and manipulation. I presented this research at ToorCon San Diego in 2016. NLP technology at the time was primitive and even comical, compared to modern capabilities. But even at that time, there was already the early foreshadowing of uniquely new and emerging risks. For the past decade, I have watched these capabilities grow, and with these capabilities, I have also seen the risks continue to grow with them.
In 2020, I began to take note of the increasingly impressive capabilities of OpenAI's Generative Pre-Training Transformer (GPT) models. I dusted off my old research and began evaluating the extent to which these new capabilities (specifically GPT-3) could be used for adversarial purposes. The difference that only a few years had made was both astonishing and terrifying. I was able to effectively construct very capable autonomous social engineering systems that could be used to fully automate the process of defrauding and manipulating potential victims.
On June 20, 2022, I presented my research at the AI Village at DEF CON—the world's largest annual hacking convention. While the DEF CON conference itself had a massive turnout, the attendance in the AI Village was relatively underwhelming (fewer than 50 people were in the village at any given time). And while the message seemed to resonate well with a small subculture of AI enthusiasts, most people were not yet paying attention.
Less than half a year later, on November 30th, everything changed. OpenAI released ChatGPT (GPT-3.5) and within a couple of weeks, the entire world had changed its views on the emerging relevance of AI. In the year that followed (in 2023), the AI Village was the most popular event at the DEF CON conference—the line to even get into the village wrapped all the way around the Caesar's Forum in Las Vegas. But it wasn't just hackers who were paying attention. The entire world had changed its perspective.
ChatGPT became the most rapidly adopted technology platform in history. And countless other AI platforms saw similar momentum shortly thereafter. Business and technology professionals began quickly assessing ways to integrate this new wave of AI technologies into operational workflows and business processes.
With all this increased enthusiasm and unprecedented rush to adopt these new technologies, the importance of making people aware of these risks has never been more pressing than it is now. And out of this need, the idea for this book was born. As the world began to take notice, I quickly scrambled to update and transcribe my years of research. I worked tirelessly to highlight the emerging risks, but also to communicate them in a way that can be easily understood by a broad audience. And finally, after so many long nights, that journey has come to an end. The product of all those efforts now belongs to you—the reader.
This technology is already poised to transform every part of our lives. The world is going to radically change in the coming years, and emerging AI technology is going to be at the center of it all. It is critical that people understand the risks that come along with these new capabilities, and how we can safeguard ourselves against those risks.
This technology is going to impact everyone, but unfortunately, so will the risks. This book was not just written for the technologist or the cybersecurity professional. It was written for all the people that this new technological revolution will affect. And it is written for anybody who has an interest in taking a glimpse into the future and understanding the new and unique risks that are rapidly approaching.
Reader Support for This Book
How to Contact the Publisher
If you believe you've found a mistake in this book, please bring it to our attention. At John Wiley & Sons, we understand how important it is to provide our customers with accurate content, but even with our best efforts an error may occur.
In order to submit your possible errata, please email it to our Customer Service Team at [email protected] with the subject line “Possible Book Errata Submission.”
How to Contact the Author
I appreciate your input and questions about this book! Find me on LinkedIn at www.linkedin.com/in/justinhutchens, or DM me on Twitter at @sociosploit.
Chapter 1Artificial Social Intelligence
Natural language processing (NLP) is a subfield of artificial intelligence (AI) that seeks to improve human and machine interactions by enabling machines to speak using human language. It is the science of attempting to train machines to understand, interpret, and communicate using human language in the form of natural conversation. Some computer scientists have studied NLP to improve system interfaces by allowing humans to communicate with machines using the same language(s) they already know. For others, however, NLP has been used to simulate interactions to better understand complex social concepts such as language, learning, and social psychology. In recent years, we have seen an explosion of growth in the capabilities of NLP due to the scaling up of deep-learning language models, also referred to as large language models (LLMs).
At this very moment, as you read this, LLMs are being trained in deep-learning GPU powerhouses to become sophisticated systems that establish extremely complex logical connections. These machines are equipped with learning algorithms and optimization routines to teach themselves how to complete their objectives. They are not provided any explicit instructions pertaining to accomplishing the task; instead, their learning algorithms provide a framework for teaching themselves the optimal way to accomplish their objectives.
At their simplest unit level, these models are just composed of individual statistical calculations. But at scale, something else seems to emerge—something that is not so easily explained. When you combine billions of these individual statistical calculations into a single centralized system, they are not merely the sum of their parts. This je ne sais quoi manifests itself as the confluence of all these complex interconnections between neural nodes. During model training, billions of neural pathways are activated, etched, and refined. These refined neural pathways determine the activation configuration of the neural nodes, each one containing a precise weight falling somewhere within the infinite fractional (non-integer) space that exists between 0 and 1. In the same way that aggregate data becomes information—and understanding of data relationships becomes knowledge—the result is something that is probably not alive or conscious, but is certainly far more than just statistics.
LLMs have introduced incredible new efficiencies to creative workflows and processes and have been rapidly adopted for personal and operational uses. But as with any exceedingly powerful technology, great opportunity often comes with great risk. Any time you are manipulating a highly complex system, there are unexpected consequences. One of the surprising by-products of scaling LLMs is that in learning language, these systems have also been able to make complex connections related to a sense of humanity. Our humanity is often transcribed into our words, our writing, and even our daily use of language. And so, while supplying these models with terabytes of language data, we have also inevitably supplied them with an obscenely large learning sample of our own humanity. While these systems are (presumably) unable to actually experience the human condition, they have nonetheless become highly effective at communicating using language that would suggest a sense of emotions, sympathy, kindness, and consideration. When you interact with an LLM system, it is easy to feel the unsettling sense of a seemingly “human” connection with something that you know is anything but. And herein lies the real risk: in creating a system that we can speak to in our own language, we have also created a system that can just as effectively speak to us with the same.
To more effectively interface with and control machine operations, we have simultaneously and inadvertently created a means whereby the machines can more easily control us. We have now equipped machines with “natural language” (i.e., with our language). This new interface to interact with computer systems uses the same languages and the same methods of communication that we use to interact with one another. By speaking their language, humans have hacked machines for decades. But now, with the machines speaking our language, the era when machines will hack humans is upon us.
Positronic Man
In 1992, Isaac Asimov wrote The Positronic Man, a book about an advanced humanoid AI named Andrew who engaged in a lifelong pursuit to become more human-like. In 1999, the book was adapted to film as a movie called Bicentennial Man. In both the book and the film, the depicted AI androids were well equipped with basic general intelligence. They were adaptive and able to learn new simple tasks in real time—for example, how to toast a bagel, how to make a cup of coffee, or how to sweep the floor. However, these androids sorely lacked basic social intelligence. They could carry on simple conversations but struggled with complex social concepts like humor and emotion. They routinely failed to identify and appropriately respond to simple social cues and were depicted as being awkward in unexpected or nonstandard social interactions. And Asimov was not alone in this vision of the future. Much of the early science fiction about AI depicted a future where the AI systems were able to engage in tasks requiring general intelligence at a level on par with humans, but they were still very apparently robotic regarding their social interactions. They were often depicted engaging in stiff, mechanical, awkward, and seemingly emotionless interactions.
Asimov's early depiction of AI was, ironically, quite the opposite of what actually came to be. In the decades since Positronic Man, basic artificial general intelligence (AGI) has proved to be exceedingly more difficult to achieve (especially in a kinetic sense) than basic artificial social intelligence. Real-world AI systems lack the ability to adapt to ordinary daily tasks like taking out the garbage or washing the dishes. While these tasks can be automated by nonintelligent machines specifically designed to accomplish these individual tasks, AI systems lack the general intelligence to be able to adapt to unusual circumstances to accomplish simple or unexpected tasks. Instead, machine learning algorithms must be used to train AI systems on very specific types of tasks. And while AGI still has not been achieved (at least not at the time of this writing), thanks to machine learning algorithms specifically focused on the syntactical structures of human language, referred to as natural language processing, AI systems are already displaying signs of something akin to social intelligence. These modern AI systems can write persuasive arguments, tell clever jokes, and even provide words of comfort in times of distress. They can even effectively engage in creative tasks like writing a poem or a story. This strange and unexpected reversal of Asimov's foretelling of the future is precisely the world we find ourselves in today. Even more fascinating is that the key to finally unlocking AGI may be hidden within the advancement of these language processing capabilities. By integrating LLMs with other operational services (i.e., service-connected LLMs), and due to the unexpected capabilities that emerge naturally from scaling these models, we are witnessing the emergence of what some have begun to consider the early sparks of AGI (Bubeck et al., 2023). So, not only was artificial social intelligence easier to achieve, but it may become the very gateway to achieving AGI as well.
Defining (Artificial) Social Intelligence
The term social intelligence refers to a person's ability to understand and effectively navigate social situations, manage interpersonal relationships, and adapt their behavior in response to the emotions, intentions, and actions of others. In an article in Psychology Today, Ronald Riggio (2014) describes what social intelligence is and how it differs from general intelligence:
Intelligence, or IQ, is largely what you are born with. Genetics play a large part. Social intelligence (SI), on the other hand, is mostly learned. SI develops from experience with people and learning from success and failures in social settings. It is more commonly referred to as “tact,” “common sense,” or “street smarts.”
So, could it be possible for an AI system to have or exhibit social intelligence? And to refine this question even further, it is probably more appropriate to ask specifically whether modern LLMs are capable of social intelligence—since those seem to be the most likely candidates within the current spectrum of AI. Riggio described social intelligence as something that, unlike general intelligence, is learned. He also described it as something that develops from experiences with past social interactions. We know that AI systems do learn, at least insofar as you consider “machine learning” to be actual learning. And just like human learning, machine learning does involve receiving information, processing it to identify patterns and trends, and establishing models of understanding derived from those processes. Moreover, while modern LLMs do not (at least at the time of their initial training) have their own social experiences to learn from, they are nonetheless able to learn from a large pool of non-personal social experiences (albeit the social experiences of others). These social experiences come in the form of multiparty text-based communications that are supplied to the LLM as its training data. Based on our current understanding, it could be reasonably stated that modern LLMs most likely do not have the ability to experience many of the common “human” characteristics associated with social intelligence, such as emotions and empathy.
At their core, LLMs are highly complex statistical models, but are still just that—statistical models. The outputs of these models are the inevitable consequence of the inputs provided, and how those inputs are processed through a complex network of pretrained “weights and biases” corresponding to the refined parameters of the model's neural nodes and neural interconnections, respectively (Coşkun et al., 2017). I say that these systems “most likely” do not experience emotions and empathy with at least a modicum of hesitation, because in truth, there is much that we still do not know about the human mind. For all our knowledge, even our minds still may be just highly complex, deterministic statistical engines. But based on my own subjective knowledge of the human experience, it certainly seems to me to be more than that. And I think that most others familiar with the human experience would likely agree with this sentiment. But even if the human mind is nothing more than a complex statistical engine, it still achieves learning in a way that is uniquely different from the learning processes used by current LLM systems. As Noam Chomsky pointed out in an article about modern LLMs:
The human mind is not, like ChatGPT and its ilk, a lumbering statistical engine for pattern matching, gorging on hundreds of terabytes of data and extrapolating the most likely conversational response or most probable answer to a scientific question. On the contrary, the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information; it seeks not to infer brute correlations among data points but to create explanations (Chomsky et al., 2023).
But for all the measurable and observable differences between the human mind and LLMs, the human experience is still an entirely subjective one. All the things that we believe make us “alive” and “human,” such as consciousness and self-awareness, are experienced by the self and no other. The innate subjectivity of these experiences is even implicit in the word self-awareness. I have firsthand knowledge of my own consciousness and awareness, but I do not have that same firsthand knowledge of the consciousness and awareness of my family, friends, and others that I interact with. I can never truly have firsthand knowledge of another. I can assume that others have experiences akin to my own, and it is not unreasonable for me to think so. And the same holds true for you (presumably)—that you have firsthand knowledge of your own consciousness and awareness, but do not have the same knowledge of the consciousness and awareness of others. And so, while it is unlikely that machines (can) possess such qualities of consciousness or their own self-awareness, it is also important to still approach this conversation with reasonable doubt and skepticism.
In truth, I have no more assurance that a person I interact with has their own self-awareness than I do that a machine does, especially if the machine is capable of exhibiting all of the external indicators of social intelligence. We will return to this fascinating question of whether a machine could feasibly be “alive” or “self-aware” at a later point in this book. But for now, let us merely consider whether a machine could be complex enough to exhibit socially intelligent behaviors like those of a being that is “alive” or “self-aware.” So, for the moment, we are foregoing the conversation of whether a machine can have the underlying characteristics that enable social intelligence in humans (i.e., emotions and empathy), and are instead considering whether a machine can engage in interactions that are, from the outside looking in, indistinguishable from the real thing. We will examine whether, from an external perspective, modern LLMs can engage in conversations that would fool a reasonable person (who is unaware that they are engaging with a machine and not influenced by the inherent assumptions associated with that awareness) into believing that those interactions involve genuine empathy and emotional resonance. We will henceforth refer to this appearance of social intelligence (whether stemming from genuine emotion and self-awareness, or purely mechanical simulation) as the semblance of social intelligence. For the purposes of our thesis (i.e., that artificial social intelligence can and will be weaponized), the origin of this semblance of social intelligence is ultimately inconsequential.
The Turing Test
In 1912, a man named Alan Turing was born on the outskirts of London. If you have even a casual interest in technology, you've likely heard of him. And in truth, for a book on machine intelligence to have any other starting point would be a travesty and an injustice. Turing achieved much in his life. He was an accomplished technologist and a futurist; he invented the world's first computer; and he even managed to successfully crack highly sophisticated (for the time) nation-state cryptography during World War II.
If you are a fan of cinema, I would point you toward the 2014 film The Imitation Game (an adaptation of the book Alan Turing: The Enigma), which offers a fantastic dramatization of the life of Alan Turing, as portrayed by British actor Benedict Cumberbatch. Ironically, despite its name, this film does not even address Alan Turing's actual notion of the imitation game. There are several possible explanations as to why this movie title may have been selected. While Turing's actual notion of the imitation game was not the subject of the movie, it is at least loosely related, since this was a term originally created by Turing within the context of computer science. Possibly the marketing team thought that it was a catchier title, and it being loosely related to the film's subject matter was determined to be good enough. But with further inspection, it seems that there is probably even more depth to it than this. The film's title also seems to be a play on words for multiple other aspects of Alan Turing's life, or perhaps those aspects of his life may have inspired his actual notion of the imitation game. The film focuses on multiple levels of imitation, where Turing was pretending, by choice or coercion, to be someone or something he was not. Much of the film focuses on intelligence and espionage operations that Turing was involved in during WWII, and the notion of imitation could easily be a reference to the duplicity that such entanglements require. On a more personal level, the film also portrays Turing as a deeply troubled person having to grapple with his own sexuality during a time in London (and in much of the rest of the world) when homosexuality was marginalized, stigmatized, and even criminalized. Being forced to succumb to this coerced repression itself operates as another prevalent theme of imitation, in both the film and Turing's life.
But for the purposes of this book, we should consider Turing's actual notion of the imitation game, which is also often referred to as the Turing test. The Turing test was the first scientific assessment designed to evaluate whether advanced computing technology could be capable of thought, or at least perceived as being capable of thought. The Turing test was created as a proposed thought experiment, since there was no sufficiently advanced computing technology at the time that could actually be the subject of such a test. The Turing test was designed before the emergence of the computer era, but to this day it remains the most commonly discussed criterion for evaluating social intelligence in robotics and computing.
In 1950, Turing published an essay entitled “Computing Machinery and Intelligence,” in which he poses a controversial question: whether it could be possible for machines to think. Turing himself acknowledges, in the introduction to the essay, that the answer to this question is intrinsically bound to multiple other ambiguous questions (semantics, metaphysics, epistemology, and even religion). To avoid this ambiguity, Turing moves away from this initial question of whether machine thought is possible and instead proposes a less ambiguous question: whether it is possible that a machine could be perceived as being capable of thought. To address this question, he proposed a testing methodology for determining if or when this threshold is met. He referred to this methodology as the Imitation Game—a process that has since come to be known as the Turing test. This process involved a human interrogator and two test subjects (one human and one machine). The interrogator would interact with each of the subjects over text-based communications and, based on the contents of those communications, would attempt to determine which subject was the human and which subject was the machine. From Turing's perspective, a machine could be perceived as being capable of thinking if the machine interactions were consistently and reliably indistinguishable from the interactions with the human subject. Over the years, the term “Turing test” has evolved to more broadly encompass any scenario in which a human must, consciously or unconsciously, determine whether they are interacting with a person or a machine.
The Semblance of Social Intelligence
On November 30th, 2022, OpenAI released the beta version of ChatGPT to the public. Within two months of its beta release, ChatGPT had over 100 million active users—the fastest growing user base for any technology platform in history at that time and far outpacing the adoption rate for other popular technology platforms of the time, like TikTok and Instagram (Hu, 2023). The unprecedented flood of media and public enthusiasm made it immediately apparent that the future of how we consume information on the Internet would no longer be a process of searching through indexes of web content that may or may not have the specific answer you are looking for (the way search engines had historically worked). The future of consuming information on the Internet would be asking a direct question and getting the exact answer you were looking for, and more importantly, getting that answer instantly.
On March 8th, 2023, in response to ChatGPT’s unprecedented and rapid integration into popular culture, the well-known satirical cartoon South Park aired an episode called “Deep Learning,” with ChatGPT as its focal point (Lancaster, 2023). The episode loosely resembled a modern retelling of Edmond Rostand's classic play Cyrano de Bergerac. For those unfamiliar, the play's central plot revolves around Cyrano de Bergerac's attempts to help Christian de Neuvillette win the love of Cyrano's beautiful and intelligent cousin Roxane. Christian is portrayed as physically attractive and courageous, but he lacks the eloquence and wit needed to capture Roxane's heart through words. To appease his own secret desire for Roxane, Cyrano agrees to help Christian win her love by writing eloquent love letters and passionate speeches on his behalf. In the South Park episode, Stan, one of the main characters, lacks the tact and thoughtfulness to continue to romance his long-time girlfriend, Wendy. Wendy complains that the text conversations between them feel empty and that Stan is failing to communicate his real feelings. A friend then introduces Stan to ChatGPT and explains to him that he can use the language model to respond to messages from his girlfriend. From that moment forward, when Stan receives a text message from Wendy, he copies and pastes the messages into the ChatGPT interface, and then in turn, relays the response from ChatGPT back to Wendy. And in the same way that Roxane unknowingly falls in love with the passionate and thoughtful words written by Cyrano, Wendy unknowingly begins to fall hopelessly in love with the words written by ChatGPT.
While it is difficult to take any content from this often tongue-in-cheek cartoon too seriously, the basic premise of the episode was rooted in truth. In an article from Mashable, an investigative journalist revealed that users on the popular dating app Tinder were in fact using ChatGPT to generate responses to prospective mates (Lovine, 2022). The article further indicated that numerous prominent young influencers on social media had published videos demonstrating just how effective this technique was in winning over the interest of others on the app. As comical and ethically questionable as this behavior is, it does go a long way in resolving the question that we have sought to answer. If modern LLMs can engage in social interactions that consistently generate the early foundations of romantic feelings in the unwitting participants with whom it is interacting in these circumstances, it is not difficult to conclude that we have already seemingly crossed that threshold, and that AI has achieved at least the semblance of social intelligence.
In an even more troubling example, the Peabody College at Vanderbilt University had to issue a formal apology after using ChatGPT to write an email that was subsequently sent out to its student body related to a recent mass shooting that had occurred at the University of Michigan (Figure 1.1). The fact that ChatGPT was used was apparent to the recipients of the email because its use was cited at the bottom of the message with the words “Paraphrase from OpenAI's ChatGPT AI language model, personal communication, February 15, 2023” (Korn, 2023).
Figure 1.1 AI-generated letter of grief and mourning from the Peabody College EDI office
Critics of the email felt that the use of AI to generate an email about community in a time of tragedy was “in poor taste.” However, the objections did not pertain to the actual content of the email. The email was well written, tactful, thoughtful, and likely would have captured the intended tone if not for the open disclosure of the use of AI. And herein lies the true crux of the objection—the unsettling realization that the email was able to convey empathy and emotional resonance adequately and effectively, even in the presumable absence of the underlying human characteristics (of emotions and self-awareness) that are generally understood to be the source of such feelings.
The Question of Weaponization
There is no denying the fact that the new capabilities emerging from LLMs will radically change the way we experience and interact with the world. And as with any new and powerful technology, societal structures of power are already considering how this tool can be weaponized. Governments, militaries, and political interest groups are already investigating opportunities to use this capability to influence and manipulate the balance of power on a global scale. In Russia, September 1st is known as “knowledge day” and marks the day that students across the country generally return to school. In 2017, knowledge day began with a public address from Russian President Vladimir Putin to the nation's students, in an effort to set the tone for their upcoming academic lessons. In this address, Putin discussed the future of the world. He did not follow the typical inspirational script and tell the kids that the future of the world belongs to them. Instead, he blatantly declared to them that the future of the world belongs to AI. He stated:
Artificial intelligence is the future, not only for Russia, but for all humankind. It comes with colossal opportunities, but also threats that are difficult to predict. Whoever becomes the leader in this sphere will become the ruler of the world (Meyer, 2017).
There are two important takeaways here. First, emerging AI technology will radically shape the world in ways that no other technology has ever done before. Second, and more importantly, even the world's most powerful leaders and authoritarians understand that AI will be the biggest factor in defining the global distribution of power in the future. Nation-states, governments, politicians, corporations, criminal organizations, and even individuals all stand to gain or lose in the zero-sum game of power, and AI will be the dominating tool to shape that balance. The world is paying attention. Even people who are not technologists or engineers have at least a general understanding of the potential influence of future AI. There is no question that this tool will be weaponized. Nation-states will use it for surveillance, intelligence, and espionage. Governments will use it to distribute propaganda and control sociopolitical narratives. Politicians will use it to deceive, to win favor, and to accrue power and influence. Corporations will use it to manipulate consumer markets and to win market share. Criminal organizations will use it to deceive and to defraud. And even individuals will use it to expand opportunities, achieve personal objectives, or pursue personal vendettas. And in the future, all of these things will be done on a scale hitherto unseen.
The question that I seek to answer in this book is not if this power will be weaponized—there is no question that it will. Rather, I seek to answer how these capabilities will be weaponized:
Will humans manipulate AI systems to achieve their objectives?
Will AI systems manipulate humans (at the direction of other humans)?
Will the machines become sentient and maximize their own self-interests over the conflicting interests of their human creators?
This last question addresses the worst-case scenario as portrayed in so many dystopian science fiction novels and films. All of these questions are worth consideration. Even the worst-case scenario, which may seem farfetched for some, is a possibility that is increasingly becoming a concern for many highly intelligent and accomplished individuals, and as such, should also be considered in our analysis.
To answer these questions, we will need to go on a fairly extensive journey. We should be deliberate and thorough in our approach, and leave no stone unturned. The early chapters of this book will address fundamental knowledge that is critical to answering these questions that we have posed. This will include a foundational discussion of social exploitation, social robotics, and discussions of consciousness and what it means to “be alive.” Prior to discussing how machines may be able to leverage social interactions for malicious purposes, it is important to understand, first and foremost, how social interactions can be weaponized in the first place (completely independent of machine involvement). To that end, I will begin by examining how an understanding of social psychology can be exploited to manipulate and control people—a practice that has come to be known as social engineering. From there, I will discuss the history of social robotics. I will then examine the confluence of the two—specifically, how social robotics have historically been used for psychological manipulation and social engineering. And then, I will seek to answer the difficult question of whether it could even be possible to replicate the human experience (of consciousness, emotions, and feelings) with technology. Finally, in the latter parts of this book, I will use that foundational knowledge to effectively speculate on how LLMs will be weaponized in the future.
Chapter 2Social Engineering and Psychological Exploitation
Social engineering is a term used to describe deliberate efforts to manipulate others, often by means of deceit or misrepresentation. While discussing various contexts and uses of social engineering throughout this chapter, I will often use the terms threat actor or attacker to describe those deliberately engaging in deceptive social engineering efforts, and I will use terms like target or victim to describe the subjects of those efforts. Nothing derogatory is implied by any of these terms. Social engineering has a broad range of applications and is used in many ways. Even when used for deception, some of these uses are based on malicious intentions, while others may be based on good intentions. The use of social engineering for deceptive purposes is a moral parallax at best. For most people, their views on morality and ethics are largely contingent upon their own personal experiences, beliefs, and values. Moreover, even a single person's opinions of morality and ethics may change and evolve over time, as they have new experiences or gain additional information. To a large extent, the differences between right and wrong are relative to cultural, geopolitical, and ideological perspectives. Depending on your perspective, sometimes the “bad guys” are not criminals, rogue threat groups, or terrorist cells. Sometimes, they are financially motivated corporate interests that seek to influence and manipulate, and sometimes they are the very governments that are supposed to serve us.
Consider the old phrase “one man's terrorist is another man's freedom fighter.” While the precise origins of this phrase are unclear, it does illustrate how personal ethics are often relative to cultural and geopolitical worldviews. These questions of morals and ethics are rarely as simple as black and white. The lines that divide right from wrong, especially within the context of the use of deception, are often blurry, shifting, or indiscernible. It is not my intention to solve, within the pages of this book, the ethical question of when or even if it is ever morally defensible to use social engineering to manipulate others through the use of deception. Instead, my intentions are more modest. In this chapter, I seek only to acknowledge the wide range of different ways that social engineering is commonly used for manipulation (both against individual targets and at scale), and then in later chapters to explain how emerging technology (specifically large language models, or LLMs) will be weaponized for this same purpose.
Social engineering is accomplished by exploiting expected patterns of human behavior, emotions, and social interactions to influence an individual's decisions or actions. At its core, social engineering capitalizes on the inherent psychological tendencies of human beings. It relies on exploiting various aspects of human interaction to manipulate individuals into behaving in a desired manner. Understanding social psychology allows social engineers to craft persuasive and deceptive tactics that can influence people in various contexts, such as politics, marketing, or interpersonal relationships. Social engineering preys on human vulnerabilities and social tendencies to influence another's behaviors and decision-making. According to available accounts, the term “social engineering” is presumed to have originated in the late 1800s:
Social engineering is a practice that is as old as time. As long as there has been coveted information, there have been people seeking to exploit it. The term social engineering was first used by Dutch industrialist J.C. Van Marken in 1894. Van Marken suggested that specialists were needed to attend to human challenges in addition to technical ones (CompTIA, n.d.).
When devising a plan to achieve a given objective, there are often multiple elements that must be considered. Refined business and organizational processes in the modern world often include technical systems, human interactions, or a combination thereof. To exploit these processes, a plan may need to include technical engineering to address system-level components of the target process, but it may also need to include social engineering to exploit the portions of the process that require human oversight or interaction. Over the years, as increasingly more consideration has been given to the technical security of solutions, social engineering has commonly become the de facto “path of least resistance” for many hackers in the cyberspace. On the one hand, a technical engineering team can perform technical testing to identify computational or functional flaws (often referred to as bugs). Modifications can then be made to software and/or hardware to mitigate vulnerabilities and effectively harden a system against common exploits. On the other hand, however, it is much harder to patch the human elements of processes. There are multiple exceedingly common human tendencies that lend themselves to people being relatively easy to exploit. These tendencies are ingrained into the social norms that define our interactions with one another.
The Roots of Social Engineering
While the term social engineering is relatively new, the actual phenomenon it describes is as old as human civilization. The notion of using intentional deception to achieve premeditated objectives has been largely ingrained into the history of human civilizations. Descriptions of social engineering activities can be found documented in some of the oldest human records including folklore, mythology, philosophy, and religion.
Folklore
In folklore, Aesop's fable of “The Fox and the Crow” tells the story of a cunning fox who deceptively uses flattery to trick a crow into sharing a piece of cheese that the crow is holding. The fox praises the crow for its beautiful singing voice, and asks the crow to sing a song for him. Through this deception, the fox is successful in getting the crow to inadvertently drop the cheese, allowing the fox the opportunity to snatch it up.
Mythology
In one of the earliest well-known works of Greek mythology, the famous poet Homer addressed the use of deception within the Trojan war. During the Trojan war, the Greeks placed a large statue of a horse outside of the gates of the city of Troy, in a deceptive attempt to gain unauthorized access and infiltrate the city. The Trojan horse was presented as an offering to Athena, the goddess of war. Not suspecting any foul play, the people of Troy brought the horse inside the city walls. But the horse was actually a deception tactic to sneak several Greek soldiers into the city by smuggling them inside the large horse statue. As nighttime fell, the soldiers inside the horse then emerged and opened the doors of the city to allow the other soldiers inside to invade.
Philosophy and Religion
Multiple philosophers discussed the use of deception for political purposes, including Plato's notion of a “noble lie” and Machiavelli's suggestion that deception could be used to influence the people's perception of their leader. And even in each of the Abrahamic religions, there is the notion that the downfall of man was ultimately a product of deception and social engineering. This was according to the belief that “evil” or “sin” was first unleashed upon the world within the Garden of Eden, when mankind was tricked into eating fruit from a tree that offered the knowledge of good and evil.
Social Engineering and Psychology
Though the concept of social engineering is nothing new, it has become an increasingly common term because of its prevalence in fraud and cybercrime. Social engineering can be an effective way to achieve unauthorized access to private systems and networks, even against organizations that are relatively well secured against traditional, technical cyberattacks. In an article from Microsoft about the psychology of social engineering, the author explains effective tactics used in social engineering by making reference to a book called Six Principles of Persuasion, written by Dr. Robert Cialdini, a professor of psychology and marketing (Kelley, 2022). This may seem counterintuitive at first, given that Cialdini's book does not focus specifically on deception, but instead on the general principles related to the psychology of persuasion and influence. Upon further examination, it becomes apparent that reference to Cialdini's work addresses the topic of social engineering (and the underlying psychology associated with it) exceptionally well. One's level of success in social engineering endeavors is largely contingent upon one's ability to persuade and influence others with whom they interact. The tactics of persuasion and how they influence human psychology ultimately remain the same, regardless of whether the intentions are genuine or malicious. In the article, Kelley points out that each of Cialdini's six different principles of persuasion can be exploited by threat actors to effectively influence and ultimately manipulate their victims. And while exploitation of each of these six principles is effective in its own way, there is one principle in particular that stands out. This principle—the principle of liking—is unique because it is the most difficult to exploit, but also, I would argue, the most effective and dangerous of the tactics. Before drawing that distinction, though, let us consider each of these six principles and how they relate to social engineering.
Reciprocity
In an episode of The Office (the American TV series), two characters named Dwight Schrute and Andy Bernard engage in a grueling back-and-forth battle of reciprocity. The episode begins with Dwight performing the seemingly thoughtful gesture of bringing in bagels for the entire office. Upon being thanked, Dwight quietly retorts with “Don't mention it. You owe me one. You all owe me one.” Andy likely senses that the motivation behind Dwight's favor is deeper than just a selfless act of altruism. He also fears that dying while owing someone a favor is “a recipe for a ghost.” So, in response to the bagels, Andy promptly returns the favor by polishing Dwight's briefcase, and thereby settles the score. The two go back and forth for some time, offering reciprocal favors—Dwight assisting Andy with things so that he is “owed one,” and Andy returning the favor each time, so that nothing is “owed” in return. Dwight holds the door for Andy, and Andy then promptly steps outside and holds the door for Dwight. Dwight offers Andy a helpful fitness tip, and Andy then readjusts Dwight's jacket. Dwight straightens Andy's tie, and then Andy cleans Dwight's glasses. This fiasco goes on for quite some time. And while this episode creates a somewhat satirical exaggeration of the principle of reciprocity, it still nonetheless holds true that humans naturally will feel a sense of obligation to reciprocate whenever a favor is received. The subtle exploitation of reciprocity is all around us, from when your restaurant server gives you mints prior to providing you the check to when a politician sends you a holiday card (Swanson, 2014). From a social engineering perspective, this means that if you can get someone to accept a gift or favor, they are more likely to comply with subsequent request(s) that you make to them.
Scarcity
If there is a perceived limitation on a person's ability to take advantage of an opportunity, a person is more likely to engage. The use of scarcity is a common tactic in late night television infomercials, where you will frequently hear phrases like “Supplies are Limited,” “Act Now,” and “For a Limited Time Only.” When scarcity becomes a factor in someone's decision-making process, they are more likely to make impulsive and ill-advised decisions. This is what is commonly referred to in layman's terms as “FOMO” or “fear of missing out.” Social engineers can increase the likelihood of their success by creating the illusion of scarcity, by making the target feel a sense of urgency and/or limited opportunity to act.
Authority
In the late 1940s, a series of trials were held in Nuremburg, Germany by the International Military Tribunal to bring many of the most prominent Nazi war criminals to justice. A common defense at the trials was that the accused men were not acting of their own volition but were instead following orders and doing their jobs. In the 1960s, a Yale University psychologist named Stanley Milgram decided to test how reasonable this claim was. He conducted an experiment to determine whether people are more likely to participate in seemingly unethical behaviors if pressured by a superior or somebody in a position of authority. Participants of the study were told that they were involved in a study on learning, and they were divided into “teacher” and “student” roles. The selection of teacher and student was fixed (pre-decided), such that the real-world participants were always assigned the teacher role, and an actor played the student. An experimenter (also played by an actor) was placed in a room with the teacher (the participant), while the learner was placed in an adjacent room (Figure 2.1). This experimenter was perceived by the participant as the authority who was in charge of and conducting the exercise. An article from SimplyPsychology succinctly describes how the experiment was conducted:
The “learner” was strapped to a chair with electrodes. After he has learned a list of word pairs given to him to learn, the “teacher” tests him by naming a word and asking the learner to recall its partner/pair from a list of four possible choices. The teacher is told to administer an electric shock every time the learner makes a mistake, increasing the level of shock each time. The learner gave mainly wrong answers (on purpose), and for each of these, the teacher gave him an electric shock (Mcleod, 2023).
The shock levels ranged from 15 volts up to 450 volts—with anything beyond 300 volts labeled with the word “DANGER” in large letters on the teacher's control panel. As the experiment went on and the shocks became increasingly more severe, the participant would hear screams and cries to stop the experiment, coming from the student's room. Each time the participant would express reluctance to continue increasing the shock severity, the experimenter (the perceived authority in the exercise), would use increasingly forceful “prods” to insist that the teacher continue. These prods included the following statements:
Please continue.
The experiment requires you to continue.
Figure 2.1 Milgram shock experiment, conducted to understand the influence of authority
It is absolutely essential that you continue.
You have no other choice but to continue.
The study found that 65 percent of the participants continued to increase the shock levels to the maximum level (of 450 volts), despite the screams coming from the other room and the warning on the control system indicating that the higher shock levels were dangerous. This experiment truly demonstrates how powerful the influence of authority is on our decision-making process. If people will, more often than not, naturally engage in unconscionable and morally reprehensible behavior because they are ordered to do so by a perceived authority figure, then it stands to reason that people would be even more likely to engage in actions that are seemingly less harmful (as nearly any actions would be). In social engineering, the principle of authority is commonly used by creating a scenario in which the target believes that the action(s) requested of them are aligned to the expectations of their superiors, or that they may be subject to disciplinary action if they do not comply.
Consensus
In the 1997 film Men in Black (MIB), the actors Will Smith and Tommy Lee Jones work for a secret government agency that monitors and manages extraterrestrial life in the universe. In an early part of the film, the two actors have a conversation about government secrecy and the agency's deliberate efforts to hide the existence of extraterrestrial aliens from society. Smith's character asks the question “Why the big secret? People are smart. They can handle it.” To which Jones’s character replies, “A person is smart. People are dumb, panicky, dangerous animals—and you know it” (Sonnenfeld, 1997).
