Machine Learning Security Principles E-Book

John Paul Mueller

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 692

Veröffentlichungsjahr: 2022

Beschreibung

Businesses are leveraging the power of AI to make undertakings that used to be complicated and pricy much easier, faster, and cheaper. The first part of this book will explore these processes in more depth, which will help you in understanding the role security plays in machine learning.
As you progress to the second part, you’ll learn more about the environments where ML is commonly used and dive into the security threats that plague them using code, graphics, and real-world references.
The next part of the book will guide you through the process of detecting hacker behaviors in the modern computing environment, where fraud takes many forms in ML, from gaining sales through fake reviews to destroying an adversary’s reputation. Once you’ve understood hacker goals and detection techniques, you’ll learn about the ramifications of deep fakes, followed by mitigation strategies.
This book also takes you through best practices for embracing ethical data sourcing, which reduces the security risk associated with data. You’ll see how the simple act of removing personally identifiable information (PII) from a dataset lowers the risk of social engineering attacks.
By the end of this machine learning book, you'll have an increased awareness of the various attacks and the techniques to secure your ML systems effectively.

Leseprobe

Machine Learning Security Principles

Keep data, networks, users, and applications safe from prying eyes

John Paul Mueller

BIRMINGHAM—MUMBAI

Machine Learning Security Principles

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Ali Abidi

Publishing Product Manager: Ali Abidi

Senior Editor: David Sugarman

Technical Editor: Sweety Pagaria

Copy Editor: Safis Editing

Project Coordinator: Farheen Fathima

Proofreader: Safis Editing

Indexer: Manju Arasan

Production Designer: Ponraj Dhandapani

First published: December 2022

Production reference: 3280923

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB.

ISBN 978-1-80461-885-1

www.packtpub.com

This book is dedicated to Eva Beattie, a friend and faithful beta reader for 25 years. Books are never the result of one person’s efforts, but of the influence of many people working together to help an author produce something wonderful.

Foreword

I first e-met John in the 1990s when we both wrote for the now-defunct magazine Visual Basic Developer. In those days, Sonic the Hedgehog was brand new, CompuServe, AOL, and Prodigy all roamed the earth, everyone programmed by candlelight on computers powered by hamster wheels, and artificial intelligence struggled to recognize the digits 0 through 9 and the words “yes” and “no” when spoken by different people. In the thirtyish years since then, we’ve all been through Y2K (which wasn’t as bad as predicted), a global pandemic (which was worse than predicted), and the Cubs winning the World Series (which no one predicted).

More relevant to this book, AI has become so powerful that it understands speech better than some humans can, produces voices so realistic that future appearances by Darth Vader will be “voiced” by an AI, and generates deepfake videos so lifelike it’s brought Salvador Dalí back to life. Amazon’s algorithms seem to ship products before I order them and I’ve seen chatbots more likely to pass the Turing test than some of my friends.

AI in general and machine learning in particular have become powerful tools for both good and bad. In this book, John explains some of the ways that machine learning can be used to perpetrate and prevent security nightmares, and ways that machine learning can accidentally wreak havoc. He describes data bias (a hiring AI for Amazon penalizing female job applicants as most of their employees are male), badly selected data (IBM’s Watson learning to swear by reading the Urban Dictionary), and intentional sabotage (Twitter turning Microsoft’s chatbot Tay into a racist, misogynistic troll in less than 24 hours). Possibly even more importantly, John covers AI used to commit fraud (one AI faked a CEO’s voice to request a €220,000 funds transfer) and to detect and counter fraud (that’s why my credit card was declined the last time I had an airline connection in Las Vegas).

John covers all of these topics and more – though not these specific examples; I just think they’re interesting and/or amusing!

As I mentioned earlier, I’ve known John for a long time. During those years, I’ve been the technical editor on several of his many books (and he’s tech edited a few of mine), and one thing I’ve learned is that John knows what he’s talking about. He’s been working in AI for years and doesn’t say something unless he’s researched it, tried it out, included example programs demonstrating key techniques, and mentioned links to back up what he says and for you to follow for more information.

Hackers using AI is a relatively new concept, and so far, their success has been somewhat limited, but you can bet that their success will increase over time. One thing that AI in general and machine learning specifically are good at is learning over time. As more AIs bring bad data to the marketplace and hackers fine-tune their attacks, the consequences will become unavoidable and you need to be prepared. In the arms race between AI-empowered hackers and AI-enabled cybersecurity professionals, you can’t afford to be uninformed.

Rod Stephens

—Author and former Microsoft MVP

Contributors

About the author

John Paul Mueller is a seasoned author and technical editor. He has writing in his blood, having produced 123 books and more than 600 articles to date. Topics he has written about range from networking to artificial intelligence and from database management to heads-down programming. Some of his recent books include discussions of data science, machine learning, and algorithms, along with Android programming and functional programming techniques. He also writes about computer languages such as C++, C#, Python, and Kotlin. His technical editing skills have helped more than 70 authors refine the content of their manuscripts. John has provided technical editing services to a variety of magazines and performed various kinds of consulting, and he writes certification exams.

Acknowledgements

Thanks to my wife, Rebecca. Even though she is gone now, her spirit is in every book I write and in every word that appears on the page. She believed in me when no one else would.

Matt Wagner, my agent, deserves credit for helping me get the contract in the first place and taking care of all the details that most authors don’t really consider. I always appreciate his assistance. It’s good to know that someone wants to help.

A number of people read all or part of this book to help me refine the approach, test the coding examples, and generally provide input that all readers wish they could have. These unpaid volunteers helped in ways too numerous to mention here. I especially appreciate the efforts of Eva Beattie, who provided general input, read the entire book, and selflessly devoted herself to this project. Claudia Smith provided me with some significant insights into the accessibility and behavioral aspects of the book. Luca Massaron helped me with the design and orientation of some of the coding examples. Quite a few people also provided me with resource materials, and this particular book required a lot more research than many of my other books.

I especially appreciated Rod Stephens’s help in maintaining a sense of humor. He also wrote a fantastic foreword, which is much appreciated by me.

The efforts of the technical reviewers are appreciated because they keep mistakes out of the book that would otherwise require reworking later. It can be quite hard to provide tactful and constructive input and I received both. You can see their names in the About the reviewers section.

Finally, I would like to thank David Sugarman, Farheen Fathima, Ali Abidi, and the rest of the editorial and production staff at Packt for their tireless efforts in helping me put this book together.

About the reviewers

Luca Massaron joined Kaggle over 10 years ago and is now a Kaggle Grandmaster in discussions and a Kaggle Master in competitions and notebooks. In Kaggle competitions, he reached number 7 in the worldwide rankings. On the professional side, Luca is a data scientist with more than a decade of experience in transforming data into smarter artifacts, solving real-world problems, and generating value for businesses and stakeholders. He is a Google Developer Expert (GDE) in machine learning and the author of best-selling books on AI, machine learning, and algorithms.

Akshay Kumar Prasad is a data scientist who builds machine learning algorithms on big and fast data platforms in cyber security. He has his P.G. diploma degree in data science from the renowned Manipal University and a BTech in biotechnology from Dr. Rajendra Prasad Central Agricultural University. Before switching fully to the data science domain, he also worked in the FMCG manufacturing industry as a quality analyst. He brings immense experience in data engineering, data analysis, and machine learning roles in manufacturing, as well as the cyber security domain. He also writes about data science on his blog and contributes to open source projects in his free time.

Deepayan Chanda is a seasoned cybersecurity professional, architect, strategist, and advisor, with a strong intent to solve cybersecurity problems for enterprises by creating a balance between security and business goals, driven by 25 years of diverse cybersecurity domain experience. He is an Ex-Armed Forces Veteran (Indian Air Force), and has experience working with various enterprises like National Australia Bank, Standard Chartered Bank, Microsoft (Singapore), Cisco Systems, McAfee, and Symantec. He serves as a Board of Advisor and a mentor to a few cybersecurity start-ups worldwide and had the privilege of sharing his broad knowledge with the wider security community by authoring two books on cybersecurity, with multiple publications in the past many years.

Table of Contents

Preface

Part 1 – Securing a Machine Learning System

1

Defining Machine Learning Security

Building a picture of ML

Why is ML important?

Identifying the ML security domain

Distinguishing between supervised and unsupervised

Using ML from development to production

Adding security to ML

Defining the human element

Compromising the integrity and availability of ML models

Describing the types of attacks against ML

Considering what ML security can achieve

Setting up for the book

What do you need to know?

Considering the programming setup

Summary

2

Mitigating Risk at Training by Validating and Maintaining Datasets

Technical requirements

Defining dataset threats

Learning about the kinds of database threats

Considering dataset threat sources

Delving into data change

Delving into data corruption

Uncovering feature manipulation

Examining source modification

Thwarting privacy attacks

Detecting dataset modification

An example of relying on traditional methods

Working with hashes and larger files

Using a data version control system example

Mitigating dataset corruption

The human factor in missingness

An example of recreating the dataset

Using an imputer

Handling missing or corrupted data

Summary

3

Mitigating Inference Risk by Avoiding Adversarial Machine Learning Attacks

Defining adversarial ML

Categorizing the attack vectors

Examining the hacker mindset

Considering security issues in ML algorithms

Defining attacker motivations

Employing CAPTCHA bypass techniques

Considering common hacker goals

Relying on trial and error

Avoiding helping the hacker

Integrating new research quickly

Understanding the Black Swan Theory

Describing the most common attack techniques

Evasion attacks

Model poisoning

Understanding membership inference attacks

Understanding Trojan attacks

Understanding backdoor (neural) attacks

Seeing adversarial attacks in action

Mitigating threats to the algorithm

Developing principles that help protect against every threat

Detecting and mitigating an evasion attack

Detecting and mitigating a model poisoning attack

Detecting and mitigating a membership inference attack

Detecting and mitigating a Trojan attack

Detecting and mitigating backdoor (neural) attacks

Summary

Further reading

Part 2 – Creating a Secure System Using ML

4

Considering the Threat Environment

Technical requirements

Defining an environment

Understanding business threats

Protecting consumer sites

Understanding malware

Understanding network attacks

Eyeing the small stuff

Dealing with web APIs

Dealing with the hype cycle

Considering social threats

Spam

Identity theft

Unwanted tracking

Remote storage data loss or corruption

Account takeover

Employing ML in security in the real world

Understanding the kinds of application security

Considering the realities of the machine

Adding human intervention

Developing a simple authentication example

Developing a simple spam filter example

Summary

Further reading

5

Keeping Your Network Clean

Technical requirements

Defining current network threats

Developing a sense of control over chaos

Implementing access control

Ensuring authentication

Detecting intrusions

Defining localized attacks

Understanding botnets

Considering traditional protections

Working with honeypots

Using data-centric security

Locating subtle intrusion indicators

Using alternative identity strategies

Obtaining data for network traffic testing

Adding ML to the mix

Developing an updated security plan

Determining which features to track

Creating real-time defenses

Using supervised learning example

Using a subprocess in Python example

Working with Flask example

Asking for human intervention

Developing predictive defenses

Defining what is available today

Downsides of predicting the future

Creating a realistic network model

Summary

6

Detecting and Analyzing Anomalies

Technical requirements

Defining anomalies

Specifying the causes and effects of anomaly detection

Considering anomaly sources

Understanding when anomalies occur

Using and combining anomaly detection and signature detection

Detecting data anomalies

Checking data validity

Forecasting potential anomalies example

Using anomaly detection effectively in ML

Considering other mitigation techniques

Summary

Further reading

7

Dealing with Malware

Technical requirements

Defining malware

Specifying the malware types

Understanding the subtleties of malware

Determining malware goals

Generating malware detection features

Getting the required disassembler

Collecting data about any application

Extracting strings from an executable

Extracting images from an executable

Generating a list of application features

Selecting the most important features

Considering speed of detection

Building a malware detection toolbox

Classifying malware

Obtaining malware samples and labels

Development of a simple malware detection scenario

Summary

Further reading

8

Locating Potential Fraud

Technical requirements

Understanding the types of fraud

Defining fraud sources

Considering fraudsters

Considering hackers

Considering other organizations

Considering company insiders

Considering customers (or fraudsters posing as customers)

Obtaining fraud datasets

Considering fraud that occurs in the background

Detecting fraud that occurs when you’re not looking

Building a background fraud detection application

Considering fraud that occurs in real time

Considering the types of real-time fraud

Detecting real-time fraud

Building a fraud detection example

Getting the data

Setting the example up

Splitting the data into train and test sets

Building the model

Performing the analysis

Checking another model

Creating a ROC curve and calculating AUC

Summary

Further reading

9

Defending against Hackers

Technical requirements

Considering hacker targets

Hosted systems

Networks

Mobile devices

Customers

Public venues and social media

Defining hacker goals

Data stealing

Data modification

Algorithm modification

System damage

Monitoring and alerting

Considering the importance of lag

An example of detecting behavior

Building and testing an XGBoost regressor

Putting the data in perspective

Predicting new behavior based on the past

Locating other behavioral datasets

Improving security and reliability

Summary

Further reading

Part 3 – Protecting against ML-Driven Attacks

10

Considering the Ramifications of Deepfakes

Technical requirements

Defining a deepfake

Modifying media

Common deepfake types

The history of deepfakes

Creating a deepfake computer setup

Installing TensorFlow on a desktop system

Checking for a GPU

Understanding autoencoders

Defining the autoencoder

Working with an autoencoder example

Understanding CNNs and implementing GANs

An overview of a Pix2Pix GAN

Obtaining and viewing the images

Manipulating the images

Developing datasets from the modified images

Creating the generator

Creating the discriminator

Performing optimization of both generator and discriminator

Monitoring the training process

Training the model

Summary

Further reading

11

Leveraging Machine Learning for Hacking

Making attacks automatic and personalized

Gaining unauthorized access bypassing CAPTCHA

Automatically harvesting information

Enhancing existing capabilities

Rendering malware less effective using GANs

Putting artificial intelligence in spear-phishing

Generating smart bots for fake news and reviews

Summary

Further reading

Part 4 – Performing ML Tasks in an Ethical Manner

12

Embracing and Incorporating Ethical Behavior

Technical requirements

Sanitizing data correctly

Obtaining benefits from data sanitization

Considering the current dataset

Removing PII

Adding traits together to make them less identifiable

Eliminating unnecessary features

Defining data source awareness

Validating user permissions

Using recognizable datasets

Verifying third-party datasets

Obtaining required third-party permissions

Understanding ML fairness

Determining what fairness means

Understanding Simpson’s paradox

Removing personal bias

Defining algorithmic bias

Addressing fairness concerns

Computing fairness indicators with TensorFlow

Solving fairness problems with TensorFlow-constrained optimization

Mitigating privacy risks using federated learning and differential privacy

Distributing data and privacy risks using federated learning

Relying on differential privacy

Summary

Further reading

Index

Other Books You May Enjoy

Share Your Thoughts

Once you’ve read Machine Learning Security Principles, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781804618851

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1 – Securing a Machine Learning System

In this part, you will discover why security is important and the various kinds of security that you will need to consider. You will look at the threats against machine learning applications, including those from data manipulation and other machine learning applications.

This section includes the following chapters:

Chapter 1, Defining Machine Learning SecurityChapter 2, Mitigating Risk at Training by Validating and Maintaining DatasetsChapter 3, Mitigating Inference Risk by Avoiding Adversarial Machine Learning Attack

1

Defining Machine Learning Security

Organizations trust machine learning (ML) to perform a wide variety of tasks today because it has proven to be relatively fast, inexpensive, and effective. Unfortunately, many people really aren’t sure what ML is because television, movies, and other media tend to provide an unrealistic view of the technology. In addition, some users engage in wishful thinking or feel the technology should be able to do more. Making matters worse, even the companies who should know what ML is about hype its abilities and make the processes used to perform ML tasks opaque. Before making ML secure, it’s important to understand what ML is all about. Otherwise, the process is akin to installing home security without actually knowing what the inside of the home contains or even what the exterior of the home looks like.

Adding security to an ML application involves understanding the data analyzed by the underlying algorithm and considering the goals of the application in interacting with that data. It also means looking at security as something other than restricting access to the data and the application (although, restricting access is a part of the picture).

The remainder of this chapter talks about the requirements for working with the coding examples. It’s helpful to have the right setup on your machine so that you can be sure that the examples will run as written.

Get in touch

Obviously, I want you to be able to work with the examples, so if you run into coding issues, please be sure to contact me at [email protected].

Using the downloadable source code will also save you time and effort. With these issues in mind, this chapter discusses these topics:

Obtaining an overview of MLDefining a need for security and choosing a typeMaking the most of this book

Building a picture of ML

People anthropomorphize computers today, giving them human characteristics, such as the ability to think. At its lowest level, a computer processes commands to manipulate data, perform comparisons, and move data around. There is no thought process involved—just electrical energy cleverly manipulated to produce a mathematical result from a given input. So, the term “machine learning” is a bit of a misnomer because the machine is learning nothing and it doesn’t understand anything. A better way to view ML is as a process of algorithm manipulation such that added weighting produces a result that better matches the data input. Once someone trains a model (the combination of algorithm and weighting added to the algorithm), it’s possible to use the model to process data that the algorithm hasn’t seen in the past and still obtain a desirable result. The result is the simulation of human thought processes so that it appears that the application is thinking when it isn’t really thinking at all.

The feature that distinguishes ML most significantly is that the computer can perform mundane tasks fast and consistently. It can’t provide original thought. A human must create the required process but, once created, the machine can outperform the human because it doesn’t require rest and doesn’t get bored. Consequently, if the data is clean, the model correct, and the anticipated result correctly defined, a machine can outshine a human. However, it’s essential to consider everything that is required to obtain a desirable result before employing ML for a particular task, and this part of the process is often lacking today. People often think that machines are much more capable than they really are and then exhibit disappointment when the machine fails to work as expected.

Why is ML important?

Despite what you may have heard from various sources, ML is more important for mundane tasks than for something earth-shattering in its significance. ML won’t enable Terminators to take over the planet, nor will this technology suddenly make it possible for humans to stop working entirely in a utopian version of the future. What ML can do is reduce the boredom and frustration that humans feel when forced to perform repetitive factory work or other tasks of the sort. In the future, at the lowest level, humans will supervise machines performing mundane tasks and be there when things go wrong.

However, the ability to simply supervise machines is still somewhat far into the future, and letting them work unmonitored is further into the future still. There are success stories, of course, but then there are also failures of the worst sort. For example, trusting the AI in a car to drive by itself without human intervention can lead to all sorts of problems. Sleeping while driving will still garner a ticket and put others at risk, as described at https://www.theguardian.com/world/2020/sep/17/canada-tesla-driver-alberta-highway-speeding. In this case, the driver was sleeping peacefully with a passenger in the front seat of the car when the police stopped him. Fortunately, the car didn’t cause an accident in this case, but there are documented instances where self-driving cars did precisely that (see https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html for an example).

Besides performing tasks, ML can perform various kinds of analysis at a speed that humans can’t match, and with greater efficiency. A doctor can rely on ML to assist in finding cancer because the ML application can recognize patterns in an MRI that the doctor can’t even see. Consequently, the ML application can help guide the doctor in the right direction. However, the doctor must still make the final determination as to whether a group of cells really is cancerous, because the ML application lacks experience and the senses that a doctor has. Likewise, ML can make a doctor’s hands steadier during surgery, but the doctor must still perform the actual task. In summary, ML is currently assistive in nature, but it can produce reliable results in that role.

Pattern recognition is a strong reason to use ML. However, the ability to recognize patterns only works when the following applies:

The source data is untaintedThe training and testing data are unbiasedThe correct algorithms are selectedThe model is created correctlyAny goals are clearly defined and verified against the training and test data

Classification uses of ML rely on patterns to distinguish between types of objects or data. For example, a classification system can detect the difference between a car, a pedestrian, and a street sign (at least, a lot of the time). Unfortunately, the current state of the art clearly shows that ML has a long way to go in this regard because it’s easy to fool an application in many cases (see https://arxiv.org/pdf/1710.08864.pdf?ref=hackernoon.com for examples). There are a lot of articles now that demonstrate all of the ways in which an adversarial attack on a deep learning or ML application will render it nearly useless. So, ML works best in an environment where nothing unexpected happens, but in that environment, it works extremely well.

Recommender systems are another form of ML that try to predict something based on past data. For example, recommender systems play a huge role in online stores where they suggest some items to go with other items a person has purchased. If you’re fond of online buying, you know from experience that the recommender systems are wrong about the additional items more often than not. A recommender setup attached to a word processor for suggesting the next work you plan to type often does a better job over time. However, even in this case, you must exercise care because the recommendation is often not what you want (sometimes with hilarious results when the recipient receives the errant text).

As everything becomes more automated, ML will play an ever-increasing role in performing the mundane and repeatable elements of that automation. However, humans will also need to play an increasingly supervisory role. In the short term, it may actually appear that ML is replacing humans and putting them out of work, but in the long term, humans will simply perform different work. The current state of ML is akin to the disruption that occurred during the Industrial Revolution, where machines replaced humans in performing many manual tasks. Because of that particular disruption in the ways that things were done, a single farmer today can tend to hundreds of acres of land, and factory work is considerably safer. ML is important because it’s the next step toward making life better for people.

Identifying the ML security domain

Security doesn’t just entail the physical protection of data, which might actually be impossible for online sources such as websites where the data scientist obtains the data using screen-scraping techniques. To ensure that data remains secure, an organization must monitor and validate it prior to use for issues such as data corruption, bias, errors, and the like. When securing an ML application, it’s also essential to review issues such as these:

Data bias: The data somehow favors a particular group or is skewed in a manner that produces an inaccurate analysis. Model errors give hackers a wedge into gaining access to the application, its model, or underlying data.Data corruption: The data may be complete, but some values are incorrect in a way that shows damage, poor formatting, or in a different form. For example, even in adding the correct state name to a dataset, such as Wisconsin, it could appear as WI, Wis, Wisc, Wisconsin, or some other legitimate, but different form.Missing critical data: Some data is simply absent from the dataset or could be replaced with a random value, or a placeholder such as N/A or Null for numeric entries.Errors in the data: The data is apparently present, but is incorrect in a manner that could cause the application to perform badly and cause the user to make bad decisions. Data errors are often the result of human data entry problems, rather than corruption caused by other sources, such as network errors. Hackers often introduce data errors that have a purpose, such as entering scripts in the place of values.Algorithm correctness: Using the incorrect algorithm will create output that doesn’t meet analysis goals, even when the underlying data is correct in every possible manner.Algorithm bias: The algorithm is designed in such a manner that it performs analysis incorrectly. This problem can also appear when weighting values are incorrect or the algorithm handles feedback values inappropriately. The bottom line is that the algorithm produces a result, but the result favors a particular group or outputs values that are skewed in some way.Repeatable and verifiable results: ML applications aren’t useful unless they can produce the same results on different systems and it’s possible to verify those results in some way (even if verification requires the use of manual methods).

ML applications are also vulnerable to various kinds of software attacks, some of which are quite subtle. All of these attacks are covered in detail starting in Chapter 3 of the book. However, here is an overview of the various attack types and a quick definition of each that you can use for now:

Evasion: Bypassing the information security functionality built into a system.Poisoning: Injecting false information into the application’s data stream.Inference: Using data mining and analysis techniques to gain knowledge about the underlying dataset, and then using that knowledge to infer vulnerabilities in the associated application.Trojans: Employing various techniques to create code or data that looks legitimate, but is really designed to take over the application or manipulate specific components of it.Backdoors: Using system, application, or data stream vulnerabilities to gain access to the underlying system or application without providing the required security credentials.Espionage: Stealing classified, sensitive data or intellectual property to gain an advantage over a person, group, or organization to perform a personnel attack.Sabotage: Performing deliberate and malicious actions to disrupt normal processes, so that even if the data isn’t corrupted, biased, or damaged in some way, the underlying processes don’t interact with it correctly.Fraud: Relying on various techniques, such as phishing or communications from unknown sources, to undermine the system, application, or data security in a secretive manner. This level of access can allow for unauthorized or unpaid use of the application and influence ways in which the results are used, such as providing false election projections.

The target of such an attack may not even know that the attack compromised the ML application until the results demonstrate it (the Seeing the effect of bad data section of Chapter 10, Considering the Ramifications of Deepfakes, shows a visual example of how this can happen). In fact, issues such as bias triggered by external dataset corruption can prove so subtle that the ML application continues to function in a compromised state without anyone noticing at all. Many attacks, such as privacy attacks (see the article entitled Privacy Attacks on Machine Learning Models, at https://www.infoq.com/articles/privacy-attacks-machine-learning-models/), have a direct monetary motive, rather than simple disruption.

It’s also possible to use ML applications as the attack vector. Hackers employ the latest techniques, such as relying on ML applications to attack you to obtain better results, just as you do. The article entitled 7 Ways in Which Cybercriminals Use Machine Learning to Hack Your Business, at https://gatefy.com/blog/cybercriminals-use-machine-learning-hack-business/, describes just seven of the ways in which hackers use ML in their nefarious trade. You can bet that hackers use ML in several other ways, some of them unexpected and likely unknown for now.

Distinguishing between supervised and unsupervised

ML relies on a large number of algorithms, used in a variety of ways, to produce a useful result. However, it’s possible to categorize these approaches in three (or possibly four) different ways:

Supervised learningUnsupervised learningReinforcement learning

Some people add the fourth approach of semi-supervised learning, which is a combination of supervised and unsupervised learning. This section will only discuss the first three because they’re the most important in understanding ML.

Understanding supervised learning

Supervised learning is the most popular and easiest-to-use ML paradigm. In this case, data takes the form of an example and label pair. The algorithm builds a mapping function between the example and its label so that when it sees other examples, it can identify them based on this function. Figure 1.1 provides you with an overview of how this process works:

Figure 1.1 – Supervised learning relies on labeled examples to train the model

Supervised learning is often used for certain types of classification, such as facial recognition, and prediction, or how well an advertisement will perform based on past examples. This paradigm is susceptible to many attack vectors including someone sending data with the wrong labels or supplying data that is outside the model’s usage.

Understanding unsupervised learning

When working with unsupervised learning, the algorithm is fed a large amount of data (usually more than is required for supervised learning) and the algorithm uses various techniques to organize, group, or cluster the data. An advantage of unsupervised learning is that it doesn’t require labels: the majority of the data in the world is unlabeled. Most people consider unsupervised learning as data-driven, contrasted with supervised learning, which is task-driven. The underlying strategy is to look for patterns, as shown in Figure 1.2:

Figure 1.2 – Unsupervised learning groups or clusters like data together to train the model

Unsupervised learning is often used for recommender systems because such systems receive a constant stream of unlabeled data. You also find it used for tracking buying habits and grouping users into various categories. This paradigm is susceptible to a broad range of attack vectors, but data bias, data corruption, data errors, and missing data would be at the top of the list.

Understanding reinforcement learning

Reinforcement learning is essentially different from either supervised or unsupervised learning because it has a feedback loop element built into it. The best way to view reinforcement learning is as a methodology where ML can learn from mistakes. To produce this effect, an agent, the algorithm performing the task, has a specific list of actions that it can take to affect an environment. The environment, in turn, can produce one of two signals as a result of the action. The first signals successful task completion, which reinforces a behavior in the agent. The second provides an environment state so that the agent can detect where errors have occurred. Figure 1.3 shows how this kind of relationship works:

Figure 1.3 – Reinforcement learning is based on a system of rewards and an updated state

You often see reinforcement learning used for video games, simulations, and industrial processes. Because you’re linking two algorithms together, algorithm choice is a significant priority and anything that affects the relationship between the two algorithms has the potential to provide an attack vector. Feeding the agent incorrect state information will also cause this paradigm to fail.

Using ML from development to production

It’s essential to understand that ML does have an important role to fulfill today in performing specific kinds of tasks. Figure 1.4 contains a list of typical tasks that ML applications perform today, along with the learning type used to perform the task and observations of security and other issues associated with this kind of task. In no instance will you find that ML performs any task perfectly, especially without human assistance.

Figure 1.4 – ML tasks and their types

Figure 1.4 doesn’t contain some of the more exotic uses for ML. For example, some people use ML to generate art (see https://www.bbc.com/news/uk-england-oxfordshire-61600523 for one of the newest examples). However, the ML application isn’t creating art. What happens instead is that the ML application learns a particular art style from examples, and then transforms another graphic, such as a family picture, into a representation using the art examples. The results can be interesting, even beautiful, but they aren’t creative. The creativity resides in the original artist and the human guiding the generation (see https://aiartists.org/ai-generated-art-tools for details). The same technique applies to ML-generated music and even videos. Many of these alternative uses for ML are interesting, but the book doesn’t cover them heavily, except for the perspective of ethical treatment of data. So, why is security so important for ML projects? The next section begins to answer that question.

Adding security to ML

Security is a necessary component of ML to ensure that results received from an analysis reflect reality. Otherwise, decisions made based on the analysis will be flawed. If the mistake made based on such analysis merely affected the analyst, then the consequences might not be catastrophic. However, ML affects people – sometimes large groups of people. When the effects are large enough, businesses fold, lawsuits ensue, and people lose faith in the ability of ML applications to produce reliable results. Adding security ensures the following:

ReliabilityVerifiabilityRepeatabilityTransparencyConfidenceConsistency

Let’s examine how security can impact ML in more detail.

Defining the human element

At this point, it’s important to take a slight detour from the technical information presented so far to discuss the human element. Even if the processes are clear, the data is clean, the algorithms are chosen correctly, and the code is error-free, humans still provide the input and interpret the result. Humans are an indirect source of security issues in all ML scenarios. When working with humans, it’s essential to consider the five mistruths that creep into every part of the ML environment and cause security issues that are difficult or sometimes impossible to find:

Commission: Performing specific and overt engagement in a mistruth and supplying incorrect data. However, a mistruth of commission need not always imply an intent to mislead. Sometimes these mistruths are the result of a lack of information, incorrect information, or a need to please others. In some cases, it’s possible to detect mistruths of commission as outliers in plotted results created during analysis. Mistruths of commission create security issues by damaging the data used for analysis and therefore corrupting the model.Omission: Leaving out essential details that would make the resulting conclusions different. In many cases, the person involved simply forgets to provide the information or is unaware of it. However, this mistruth also makes an appearance when the facts are inconvenient. In some cases, it’s possible to detect this sort of mistruth during missingness checks of the data or in considering the unexpected output of an algorithm. Mistruths of omission create security issues by creating holes in the data or by skewing the model.Bias: Seeing the data or results in an unrealistic or counterintuitive manner due to personal concerns, environmental pressures, or traditions. Human biases often keep the person involved from seeing the patterns and outcomes that are obvious when the bias isn’t present. Environmental pressures, including issues such as tiredness, are hard to overcome and spot. The same checks that work for other kinds of bias can help root out human biases in data. Mistruths of bias create security issues by skewing the model and possibly causing the model to overfit or underfit the data.Perspective: Viewing the data based on experience, environmental conditions, and available information. In reviewing the statements of witnesses to any event, it’s possible to obtain different stories from each witness, even when the witnesses are being truthful from their perspective. The same is true of ML data, algorithms, and output. Different people will see the data in different ways and it’s nearly impossible to say that one perspective is correct and another incorrect. In many cases, the only way to handle this issue is to create a consensus opinion, much as interviewers do when speaking to witnesses to an event. Mistruths of perspective cause security issues by limiting the effectiveness of the model in providing a correct solution due to the inability of computers to understand anything.Frame of reference: Conveying information to another party incorrectly because the other party lacks the required experience. This kind of soft knowledge is precisely why humans are needed to interpret the analysis provided through ML. A human who has had a particular experience understands the experience and recognizes the particulars of it, but is unable to articulate the experience in a concrete manner. Mistruths of frame of reference create security issues by causing the model to misinterpret situational data and render incorrect results.

Now that you have a better idea of how humans play into the data picture, it’s time to look more at the technical issues, keeping the human element in mind.

Compromising the integrity and availability of ML models

In many respects, the ML model is a kind of black box where data goes in and results come out. Many countries now have laws mandating that models become more transparent, but still, unless you want to spend a great deal of time reviewing the inner workings of a model (assuming you have the knowledge required to understand how they work at all), it still amounts to a black box. The model is the weakest point of an ML application. It’s possible to verify and validate data, and understanding the algorithms used need not prove impossible. However, the model is a different story because the only practical ways to test it are to use test data and perform some level of verification and validation. What happens, however, if hackers or others have compromised the integrity of the model in subtle ways that don’t affect all results, just some specific results?

The integrity of a model doesn’t just involve training it with correct data but also involves keeping it trained properly. Microsoft’s Tay (see https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation) is an example of just how wrong training can go when the integrity of the model is compromised. In Tay’s case, unregulated Twitter messages did all the damage in about 16 hours. Of course, it took a lot longer than that to initially create the model, so the loss to Microsoft was immense. To the question of why internet trolls damaged the ML application, the answer of because they can seems trite, but ends up being on the mark. Microsoft created a new bot named Zo that fared better but was purposely limited, which serves to demonstrate there are some limits to ML.

The problem of discerning whether someone has compromised a model becomes greater for pre-trained models (see https://towardsdatascience.com/4-pre-trained-cnn-models-to-use-for-computer-vision-with-transfer-learning-885cb1b2dfc for examples). Pre-trained models are popular because training a model is a time-consuming and sometimes difficult process (pretrained models are used in a process called transfer learning where knowledge gained solving one problem is used to solve another, similar problem. For example, a model trained to recognize cars can be modified to recognize trucks as well). If you can simply plug a model into your application that someone else has trained, the entire process of creating the application is shorter and easier. However, pre-trained models also aren’t under your direct control and you have no idea of precisely how they were created. There is no way to completely validate that the model isn’t corrupted in some way. The problem is that datasets are immense and contain varied information. Creating a test harness to measure every possible data permutation and validate it is impossible.

In addition to integrity issues, ML models can also suffer performance and availability issues. For example, greedy algorithms can get stuck in local minima. Crafting data such that it optimizes the use of this condition to cause availability problems could be a form of attack. Because the data would appear correct in every way, data checks are unlikely to locate these sorts of problems. You’d need to use some sort of tuning or optimization to reduce the risk of such an attack. Algorithm choice is important when considering this issue. The easiest way to perpetrate such attacks is to modify the data at the source. However, making the attack successful would require some knowledge of the model, a level of knowledge known as white box access.

A major issue that allows for integrity and availability attacks is the assumption on the part of humans (even designers) that ML applications think in the same way as we do, which couldn’t be further from the truth. As this book progresses, you will discover that ML isn’t anything like a thought process—it’s a math process, which means treating adversarial attacks as a math or data problem. Some researchers have suggested including adversarial data in the training data for algorithms so that the algorithm can learn to spot them (learn, in this case, is simply a shortcut method of saying that the model has weights and variables adjusted to process the data in a manner that allows for a correct output result). Of course, researchers are looking into this and many other solutions for dealing with adversarial attacks that cause integrity - and performance-type problems. There is currently no silver bullet solution.

Describing the types of attacks against ML

The introduction to this chapter lists a number of attack types on data (such as data bias) and the application (evasion). The previous section lists some types of attacks perpetrated against the underlying model. As a collection, all of these attacks are listed as adversarial attacks, wherein the ML application as a whole tends not to perform as intended and often does something unexpected. This isn’t a new phenomenon—some people experimented with adversarial attacks as early as 2004 (see https://dl.acm.org/doi/10.1145/1014052.1014066 for an article on the issue). However, it has become a problem because ML and deep learning are now deeply embedded within society in such a way that even small problems can lead to major consequences. In fact, sites such as The Daily Swig (https://portswigger.net/daily-swig/vulnerabilities) follow these vulnerabilities because there are too many for any single individual to track.

Underlying the success of these attacks is that ML essentially relies on statistics. The transformation of input values to the desired output, such as the categorization of a particular sign as a stop sign, relies on the pixels in a stop sign image relating statistically well enough to the model’s trained values to make it into the stop sign category. By adding patches to a stop sign, it no longer matches the learned pattern well enough for the model to classify it as a stop sign. Because of the misclassification, a self-driving car may not stop as required but run right through the stop sign, causing an accident (often to the hacker’s delight).

Several elements come into play in this case. A human can look at the sign and see that it’s octangular, red, and says Stop, even if someone adds little patches to it. In addition, humans understand the concept of a sign. An ML application receives a picture consisting of pixels. It doesn’t understand signs, octangular or otherwise, the color red, or necessarily read the word Stop. All the ML application is able to do is match the object in a picture created with pixels to a particular pattern it has been trained to statistically match. As mentioned earlier in the chapter, machines don’t think or feel anything—they perform computations.

Modifying a street sign is an example of an overt attack. ML is even more susceptible to overt attacks. For example, the article at https://arxiv.org/pdf/1801.01944.pdf explains how to modify a sound file such that it embeds a command in the sound file that the ML application will recognize, but a human can’t even hear. The commands could do something innocuous, such as turn the speaker volume up to maximum, but they could perform nefarious tasks as well. Just how terrible the attack becomes depends on the hacker’s knowledge of the target and the goal of the attack. Someone’s smart speaker could send commands to a voice-activated security system to turn the system off when the owner isn’t at home, or perhaps it could trigger an alarm, depending on what the hacker wants (read Attackers can force Amazon Echos to hack themselves with self-issued commands at https://arstechnica.com/information-technology/2022/03/attackers-can-force-amazon-echos-to-hack-themselves-with-self-issued-commands/ to get a better understanding of how any voice-activated device can be hacked).

Attacks can affect any form of ML application. Simply changing the order of words in a text document can cause an ML application to misclassify the text (see the article at https://arxiv.org/abs/1812.00151). This sort of attack commonly thwarts the activities of spam and sentiment detectors but could be applied to any sort of textual documentation. Most experts classify this kind of attack as a paraphrasing attack. (See the Developing a simple spam filter example section of Chapter 4, Considering the Threat Environment, for details on working with text.) When you consider how much automated text processing occurs because there is simply too much being generated for humans to handle alone, this kind of attack can take on monumental proportions.

Considering what ML security can achieve

The essential goal of ML security is to obtain more consistent, reliable, trustworthy, and unbiased results from ML algorithms. Security focuses on creating an environment where the data, algorithm, responses, and