Deep Learning and XAI Techniques for Anomaly Detection E-Book

Cher Simon

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 185

Veröffentlichungsjahr: 2023

Beschreibung

Despite promising advances, the opaque nature of deep learning models makes it difficult to interpret them, which is a drawback in terms of their practical deployment and regulatory compliance.
Deep Learning and XAI Techniques for Anomaly Detection shows you state-of-the-art methods that’ll help you to understand and address these challenges. By leveraging the Explainable AI (XAI) and deep learning techniques described in this book, you’ll discover how to successfully extract business-critical insights while ensuring fair and ethical analysis.
This practical guide will provide you with tools and best practices to achieve transparency and interpretability with deep learning models, ultimately establishing trust in your anomaly detection applications. Throughout the chapters, you’ll get equipped with XAI and anomaly detection knowledge that’ll enable you to embark on a series of real-world projects. Whether you are building computer vision, natural language processing, or time series models, you’ll learn how to quantify and assess their explainability.
By the end of this deep learning book, you’ll be able to build a variety of deep learning XAI models and perform validation to assess their explainability.

Leseprobe

Deep Learning and XAI Techniques for Anomaly Detection

Integrate the theory and practice of deep anomaly explainability

Cher Simon

BIRMINGHAM—MUMBAI

Deep Learning and XAI Techniques for Anomaly Detection

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Dhruv Jagdish Kataria

Senior Editor: Tazeen Shaikh

Technical Editor: Devanshi Ayare

Copy Editor: Safis Editing

Project Coordinator: Farheen Fathima

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Arunkumar Govinda Bhat

Marketing Coordinators: Shifa Ansari and Vinishka Kalra

First published: January 2023

Production reference: 1310123

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80461-775-5

www.packtpub.com

To my family, for your unconditional love and support. To my readers, thank you for the inspiration and for taking the time to read this book.

– Cher Simon

Foreword

I was delighted to be asked to write the foreword for this interesting and helpful book. In my role as Chief Evangelist for Amazon Web Services (AWS), I can see that our customers are eager to use machine learning to solve many different types of real-world problems at scale. With access to vast amounts of storage and compute power in the cloud, they are able to take on challenges today that would have been impractical or even impossible just a decade ago.

This book addresses one of those challenges in detail: using all of that storage and compute power to create machine learning (ML) models that can accurately and efficiently detect anomalies hidden in vast amounts of data.

When I think about anomaly detection, I immediately think of my own intuition, my Spidey-sense that something is not quite right. You know that feeling, that early warning deep inside, where your subconscious seemingly knows something before you do, right? That’s your internal anomaly detector at work, using rules and patterns that you might not even realize you have set up.

To me, ML-powered anomaly detection is the scientific, mathematically sound, and scalable version of my inexplicable Spidey-sense. This book will show you how to build anomaly detectors that you can apply to all sorts of mission-critical use cases – fraud detection, predictive machine maintenance, and much more. While it will give you a strong grounding in the theory, it will also show you how to put the theory to work right away. In fact, you will learn how to set up Amazon SageMaker Studio Lab before the end of the fourth page!

Early ML models were black boxes and provided no insights into how they made their predictions. As ML moved from the lab to production, forward-thinking people raised very valid concerns and asked hard questions – how did the model make decisions, is the model biased, and why should we trust it to make mission-critical decisions?

To that end, this book will also bring you up to speed on the important topic of explainable artificial intelligence, or XAI for short. With millions of parameters and millions of weights, sophisticated models might seem to be beyond human comprehension at first glance. Fortunately, that is not actually the case, and as you work through the examples in each chapter, you will learn how you can use explanations to earn trust with your users, ensure compliance with your objectives, and ensure that your models accurately reflect your business needs.

I hope that you will learn a lot from this book (I know that I did!) and that you will be able to put those lessons to use within days.

Good luck, and let me know how it goes!

Jeff Barr

VP and Chief Evangelist, Amazon Web Services

Contributors

About the author

Cher Simon is a principal solutions architect specializing in artificial intelligence, machine learning, and data analytics at AWS. Cher has 20 years of experience in architecting enterprise-scale, data-driven, and AI-powered industry solutions. Besides building cloud-native solutions in her day-to-day role with customers, Cher is also an avid writer and a frequent speaker at AWS conferences.

About the reviewers

Austen Groener is an applied scientist at AWS living in the greater Boston area. His research and work are focused on developing computer vision and artificial intelligence capabilities for the Amazon Dash Cart. Previously, he was a research scientist at Lockheed Martin Space Systems where he worked within the field of remote sensing and anomaly detection. Austen holds a Ph.D. in physics from Drexel University.

Samet Akcay is an AI research engineer/scientist. His primary research interests are real-time image classification, detection, anomaly detection, and unsupervised feature learning via deep or machine learning algorithms. He recently co-authored and open sourced anomalib, one of the largest anomaly detection libraries in the field. Samet holds a Ph.D. degree from the department of computer science at Durham University, UK, and received his M.Sc. degree from the Robust Machine Intelligence Lab at the department of electrical engineering at Penn State University, USA. He has over 30 academic papers published in top-tier computer vision and machine/deep learning conferences and journals.

Aditya Jain is a trained computational scientist from the University of Texas at Austin and IIT Roorkee. He has experience in deploying large-scale machine learning models at organizations such as Meta Platforms, SparkCognition, and Ola Cabs for billions of users. He loves demystifying the workings of machine learning models and answering the question, “What is model learning?” thus making models more interpretable. Aditya Jain is a technologist at heart with an avid interest in solving society’s biggest problems using deep learning and artificial intelligence. When he is not researching the latest advances in machine learning, he can be found running a marathon, dancing, or scuba diving. Follow his work at https://adityajain.in/.

Table of Contents

Preface

Part 1 – Introduction to Explainable Deep Learning Anomaly Detection

1

Understanding Deep Learning Anomaly Detection

Technical requirements

Exploring types of anomalies

Discovering real-world use cases

Detecting fraud

Predicting industrial maintenance

Diagnosing medical conditions

Monitoring cybersecurity threats

Reducing environmental impact

Recommending financial strategies

Considering when to use deep learning and what for

Understanding challenges and opportunities

Summary

2

Understanding Explainable AI

Understanding the basics of XAI

Differentiating explainability versus interpretability

Contextualizing stakeholder needs

Implementing XAI

Reviewing XAI significance

Considering the right to explanation

Driving inclusion with XAI

Mitigating business risks

Choosing XAI techniques

Summary

Part 2 – Building an Explainable Deep Learning Anomaly Detector

3

Natural Language Processing Anomaly Explainability

Technical requirements

Understanding natural language processing

Reviewing AutoGluon

Problem

Solution walk-through

Exercise

4

Time Series Anomaly Explainability

Understanding time series

Understanding explainable deep anomaly detection for time series

Technical requirements

The problem

Solution walkthrough

Exercise

Summary

5

Computer Vision Anomaly Explainability

Reviewing visual anomaly detection

Reviewing image-level visual anomaly detection

Reviewing pixel-level visual anomaly detection

Integrating deep visual anomaly detection with XAI

Technical requirements

Problem

Solution walkthrough

Exercise

Summary

Part 3 – Evaluating an Explainable Deep Learning Anomaly Detector

6

Differentiating Intrinsic and Post Hoc Explainability

Technical requirements

Understanding intrinsic explainability

Intrinsic global explainability

Intrinsic local explainability

Understanding post hoc explainability

Post hoc global explainability

Post hoc local explainability

Considering intrinsic versus post hoc explainability

Summary

7

Backpropagation versus Perturbation Explainability

Reviewing backpropagation explainability

Saliency maps

Reviewing perturbation explainability

LIME

Comparing backpropagation and perturbation XAI

Summary

8

Model-Agnostic versusModel-Specific Explainability

Technical requirements

Reviewing model-agnostic explainability

Explaining AutoGluon with Kernel SHAP

Reviewing model-specific explainability

Interpreting saliency with Guided IG

Choosing an XAI method

Summary

9

Explainability Evaluation Schemes

Reviewing the System Causability Scale (SCS)

Exploring Benchmarking Attribution Methods (BAM)

Understanding faithfulness and monotonicity

Human-grounded evaluation framework

Summary

Index

Other Books You May Enjoy

Part 1 – Introduction to Explainable Deep Learning Anomaly Detection

Before embarking upon an AI journey to drive transformational business opportunities, it is essential to understand the compelling rationale for embracing and incorporating explainability throughout the process.

Part 1 introduces the usage of deep learning and the significant role of XAI in anomaly detection. By the end of Part 1, you will have gained a conceptual understanding with hands-on experience of where XAI fits in the bigger picture of the machine learning (ML) life cycle. Besides the awareness of executive accountability and increased regulatory pressure in AI adoption, you will learn how XAI can bring significant business benefits and turn your AI strategy into a competitive differentiator.

This part comprises the following chapters:

Chapter 1, Understanding Deep Learning Anomaly DetectionChapter 2, Understanding Explainable AI

1

Understanding Deep Learning Anomaly Detection

Anomaly detection is an active research field widely applied to many commercial and mission-critical applications, including healthcare, fraud detection, industrial predictive maintenance, and cybersecurity. It is a process of discovering outliers, abnormal patterns, and unusual observations that deviate from established normal behaviors and expected characteristics in a system, dataset, or environment.

Many anomaly detection applications require domain-specific knowledge to extract actionable insights in a timely manner for informed decision-making and risk mitigation. For example, early detection of equipment performance degradation prevents unplanned downtime, whereas early discovery of disease threats prevents a pandemic outbreak.

The advent of cloud technologies, unlimited digital storage capacity, and a plethora of data have motivated deep learning research for anomaly detection. Detecting outliers requires an enormous dataset because anomalies are rare by nature in the presence of abundance. For example, detecting abnormal machinery vibrations and unusual power consumption or temperature increases allows companies to plan for predictive maintenance and avoid expensive downtime.

Deep learning anomaly detection has shown promising results in addressing challenges with the rare nature of anomalies, complex modeling of high-dimensional data, and identifying novel anomalous classes. The primary interest in anomaly detection is often focused on isolating undesirable data instances, such as product defects and safety risks, from the targeted domain. Other interests include improving model performance by removing noisy data or irrelevant outliers and identifying emerging trends from the dataset for a competitive advantage.

This chapter covers an overview of anomaly detection with the following topics:

Exploring types of anomaliesDiscovering real-world use casesConsidering when to use deep learning and what forUnderstanding challenges and opportunities

By the end of this chapter, you will have an understanding of the basics of anomaly detection, including real-world use cases, and the role of deep learning in accelerating outlier discovery. You will also have gained a sense of existing challenges and growth potential in leveraging deep learning techniques for anomaly detection.

Technical requirements

For this chapter, you will need the following components for the example walkthrough:

PyOD – An open-source Python library for outlier detection on multivariate dataMatplotlib – A plotting library for creating data visualizationsNumPy – An open-source library that provides mathematical functions when working with arraysPandas – A library that offers data analysis and manipulation toolsSeaborn – A Matplotlib-based data visualization libraryTensorFlow – An open-source framework for building deep learning applications

Sample Jupyter notebooks and requirements files for package dependencies discussed in this chapter are available at https://github.com/PacktPublishing/Deep-Learning-and-XAI-Techniques-for-Anomaly-Detection/tree/main/Chapter1.

You can experiment with this example on Amazon SageMaker Studio Lab, https://aws.amazon.com/sagemaker/studio-lab/, a free ML development environment that provides up to 12 hours of CPU or 4 hours of GPU per user session and 15 GiB storage at no cost. Alternatively, you can try this on your preferred Integrated Development Environment (IDE).

Before exploring the sample notebooks, let’s cover the types of anomalies in the following section.

Exploring types of anomalies

Before choosing appropriate algorithms, a fundamental understanding of what constitutes an anomaly is essential to enhance explainability. Anomalies manifest in many shapes and sizes, including objects, vectors, events, patterns, and observations. They can exist in static entities or temporal contexts. Here is a comparison of different types of anomalies:

A point anomaly exists in any dataset where an individual data point is out of the boundary of normal distribution. For example, an out-of-norm expensive credit card purchase is a point anomaly.A collective anomaly only occurs when a group of related data records or sequences of observations appear collectively and significantly differ from the remaining dataset. A spike of errors from multiple systems is a collective anomaly that might indicate problems with downstream e-commerce systems.A contextual anomaly occurs when viewed against contextual attributes such as day and time. An example of a temporal contextual anomaly is a sudden increase in online orders outside of expected peak shopping hours.

An anomaly has at least one (univariate) or multiple attributes (multivariate) in numerical, binary, continuous, or categorical data types. These attributes describe the characteristics, features, and dimensions of an anomaly. Figure 1.1 shows examples of common anomaly types:

Figure 1.1 – Types of anomalies

Defining an anomaly is