Deep Learning from the Basics E-Book

Koki Saitoh

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 330

Veröffentlichungsjahr: 2021

Beschreibung

Deep learning is rapidly becoming the most preferred way of solving data problems. This is thanks, in part, to its huge variety of mathematical algorithms and their ability to find patterns that are otherwise invisible to us.
Deep Learning from the Basics begins with a fast-paced introduction to deep learning with Python, its definition, characteristics, and applications. You’ll learn how to use the Python interpreter and the script files in your applications, and utilize NumPy and Matplotlib in your deep learning models. As you progress through the book, you’ll discover backpropagation—an efficient way to calculate the gradients of weight parameters—and study multilayer perceptrons and their limitations, before, finally, implementing a three-layer neural network and calculating multidimensional arrays.
By the end of the book, you’ll have the knowledge to apply the relevant technologies in deep learning.

Leseprobe

Deep Learning from the Basics

Python and Deep Learning: Theory and Implementation

Koki Saitoh

Deep Learning from the Basics

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Author: Koki Saitoh

Managing Editor: Ashish James

Acquisitions Editor: Bridget Neale

Production Editor: Salma Patel

Editorial Board: Megan Carlisle, Mahesh Dhyani, Heather Gopsill, Manasa Kumar, Alex Mazonowicz, Monesh Mirpuri, Bridget Neale, Abhishek Rane, Brendan Rodrigues, Ankita Thakur, Nitesh Thakur, and Jonathan Wray

First Published: March 2021

Production Reference: 1040321

ISBN: 978-1-80020-613-7

Published by Packt Publishing Ltd.

Livery Place, 35 Livery Street

Birmingham B3 2PB, UK

Table of Contents

Preface

Introduction

1. Introduction to Python

What is Python?

Installing Python

Python Versions

External Libraries That We Use

Anaconda Distribution

Python Interpreter

Mathematical Operations

Data Types

Variables

Lists

Dictionaries

Boolean

if Statements

for Statements

Functions

Python Script Files

Saving in a File

Classes

NumPy

Importing NumPy

Creating a NumPy Array

Mathematical Operations in NumPy

N-Dimensional NumPy Arrays

Broadcasting

Accessing Elements

Matplotlib

Drawing a Simple Graph

Features of pyplot

Displaying Images

Summary

2. Perceptrons

What Is a Perceptron?

Simple Logic Circuits

AND Gate

NAND and OR gates

Implementing Perceptrons

Easy Implementation

Introducing Weights and Bias

Implementation with Weights and Bias

Limitations of Perceptrons

XOR Gate

Linear and Nonlinear

Multilayer Perceptrons

Combining the Existing Gates

Implementing an XOR Gate

From NAND to a Computer

Summary

3. Neural Networks

From Perceptrons to Neural Networks

Neural Network Example

Reviewing the Perceptron

Introducing an Activation Function

Activation Function

Sigmoid Function

Implementing a Step Function

Step Function Graph

Implementing a Sigmoid Function

Comparing the Sigmoid Function and the Step Function

Nonlinear Function

ReLU Function

Calculating Multidimensional Arrays

Multidimensional Arrays

Matrix Multiplication

Matrix Multiplication in a Neural Network

Implementing a Three-Layer Neural Network

Examining the Symbols

Implementing Signal Transmission in Each Layer

Implementation Summary

Designing the Output Layer

Identity Function and Softmax Function

Issues when Implementing the Softmax Function

Characteristics of the Softmax Function

Number of Neurons in the Output Layer

Handwritten Digit Recognition

MNIST Dataset

Inference for Neural Network

Batch Processing

Summary

4. Neural Network Training

Learning from Data

Data-Driven

Training Data and Test Data

Loss Function

Sum of Squared Errors

Cross-Entropy Error

Mini-Batch Learning

Implementing Cross-Entropy Error (Using Batches)

Why Do We Configure a Loss Function?

Numerical Differentiation

Derivative

Examples of Numerical Differentiation

Partial Derivative

Gradient

Gradient Method

Gradients for a Neural Network

Implementing a Training Algorithm

A Two-Layer Neural Network as a Class

Implementing Mini-Batch Training

Using Test Data for Evaluation

Summary

5. Backpropagation

Computational Graphs

Using Computational Graphs to Solve Problems

Local Calculation

Why Do We Use Computational Graphs?

Chain Rule

Backward Propagation in a Computational Graph

What Is the Chain Rule?

The Chain Rule and Computational Graphs

Backward Propagation

Backward Propagation in an Addition Node

Backward Propagation in a Multiplication Node

Apples Example

Implementing a Simple Layer

Implementing a Multiplication Layer

Implementing an Addition Layer

Implementing the Activation Function Layer

ReLU Layer

Sigmoid Layer

Implementing the Affine and Softmax Layers

Affine Layer

Batch-Based Affine Layer

Softmax-with-Loss Layer

Implementing Backpropagation

Overall View of Neural Network Training

Presupposition

Implementing a Neural Network That Supports Backpropagation

Gradient Check

Training Using Backpropagation

Summary

6. Training Techniques

Updating Parameters

Story of an Adventurer

SGD

Disadvantage of SGD

Momentum

AdaGrad

Adam

Which Update Technique Should We Use?

Using the MNIST Dataset to Compare the Update Techniques

Initial Weight Values

How About Setting the Initial Weight Values to 0?

Distribution of Activations in the Hidden Layers

Initial Weight Values for ReLU

Using the MNIST Dataset to Compare the Weight Initializers

Batch Normalization

Batch Normalization Algorithm

Evaluating Batch Normalization

Regularization

Overfitting

Weight Decay

Dropout

Validating Hyperparameters

Validation Data

Optimizing Hyperparameters

Implementing Hyperparameter Optimization

Summary

7. Convolutional Neural Networks

Overall Architecture

The Convolution Layer

Issues with the Fully Connected Layer

Convolution Operations

Padding

Stride

Performing a Convolution Operation on Three-Dimensional Data

Thinking in Blocks

Batch Processing

The Pooling Layer

Characteristics of a Pooling Layer

Implementing the Convolution and Pooling Layers

Four-Dimensional Arrays

Expansion by im2col

Implementing a Convolution Layer

Implementing a Pooling Layer

Implementing a CNN

Visualizing a CNN

Visualizing the Weight of the First Layer

Using a Hierarchical Structure to Extract Information

Typical CNNs

LeNet

AlexNet

Summary

8. Deep Learning

Making a Network Deeper

Deeper Networks

Improving Recognition Accuracy

Motivation for a Deeper Network

A Brief History of Deep Learning

ImageNet

VGG

GoogLeNet

ResNet

Accelerating Deep Learning

Challenges to Overcome

Using GPUs for Acceleration

Distributed Training

Reducing the Bit Number for Arithmetic Precision

Practical Uses of Deep Learning

Object Detection

Segmentation

Generating Image Captions

The Future of Deep Learning

Converting Image Styles

Generating Images

Automated Driving

Deep Q-Networks (Reinforcement Learning)

Summary

Appendix A

Preface

About

This section briefly introduces the author and the material covered in this book.

About the Book

Deep learning is rapidly becoming the most preferred way of solving data problems. This is thanks, in part, to its huge variety of mathematical algorithms and their ability to find patterns that are otherwise invisible to us.

Deep Learning from the Basics begins with a fast-paced introduction to deep learning with Python, its definition, characteristics, and applications. You'll learn how to use the Python interpreter for the script files. You'll also learn how to utilize NumPy and Matplotlib in your deep learning applications. As you progress through the book, you'll discover backpropagation—an efficient way to calculate the gradients of weight parameters—and study multilayer perceptrons and their limitations before finally implementing a three-layer neural network and calculating multidimensional arrays.

By the end of the book, you'll have the knowledge to apply the relevant technologies in deep learning.

About the Authors

Koki Saitoh was born in Nagasaki, Japan in 1984. He graduated from the engineering department of the Tokyo Institute of Technology and completed a master's course at the Graduate School of Interdisciplinary Information Studies at the University of Tokyo. Currently, he conducts research and development in computer vision and machine learning. He has authored Deep Learning from the Basics series (Vol.1-3), which are published by O'Reilly, Japan.

Learning Objectives

Use Python with minimum external libraries to implement deep learning programsStudy the various deep learning and neural network theoriesLearn how to set initial values of weightsImplement techniques such as batch normalization, dropout, and AdamExplore applications like automatic driving, image generation, and reinforcement learning

Audience

Deep Learning from the Basics is designed for data scientists, data analysts, and developers who want to use deep learning techniques to develop efficient solutions. This book is ideal for those who want a deeper understanding as well as an overview of the technologies. Some working knowledge of Python is a must. Knowledge of NumPy and pandas will be beneficial but not essential.

Approach

This book takes a practical approach to deep learning. Exploring the concepts through guided practice, you will write code and implement mathematical algorithms from a programmer's point of view.

Introduction

As technology advances, what was once termed science fiction begins to look increasingly plausible—even, at times, realistic. Artificial intelligence has beaten shogi, chess, and even igo champions. Smartphones can understand human speech, and video calls can provide real-time machine interpretation. "No-crash cars" with built-in cameras protect human lives, and automatic vehicles are getting closer to practical use. When we look around, we find that artificial intelligence flawlessly performs what we once thought exclusively human tasks—even, at times, surpassing us. The development of artificial intelligence is changing our world, making it anew.

"Deep learning" technology plays an important role in such remarkable developments. Researchers around the world praise deep learning as an innovative technology, even calling it a once-in-decade breakthrough. The term is now as familiar to the general public as it is to researchers and engineers.

This book focuses on deep learning, which has been receiving a lot of attention. The purpose is for readers to understand deep learning technologies as comprehensively as possible.

Approaching the essence of deep learning through a process of implementing deep learning programs, this book attempts to exhaustively detail required technologies and functional, practical examples with which readers can experiment.

Deep learning is already used all around the world—in smartphones, in self-driving cars, and in servers that empower Web services. Deep learning is operating today in places where many people do not notice it. In the future, deep learning will more clearly step out into the light. This book will help you understand and be fascinated by its associated technologies.

Concept of this book

This book is about deep learning and covers the knowledge required to understand it step by step from the basics, including what it is, what it entails, and how it works as simply as possible to give readers a deeper understanding of the relevant technologies.

Then what should we do to better understand deep learning? Well, one of the best ways is by making something—for example, performing practical tasks to create a program that runs from scratch that promotes critical thinking while reading a source code. Now, "from scratch" in this context means using as little external already-made items (such as libraries and tools) as possible. The goal of this book is to use as little as possible of these "black boxes", whose contents are unknown, meaning that you begin with minimal basic knowledge, upon which you will build, analyze, and implement to understand and make state-of-the-art deep learning programs. If you were to compare this book to a car manual, it is not a manual that shows how to drive a car but rather one that focuses on understanding the principle of a car. It directs you to open the hood of the car, remove and examine each individual piece for its form, function, and placement before putting it back together and building your model to its exact dimensions and operations. The goal of this book is to make you feel that you can make a car and be familiar with the technologies around it through the process of making it.

In this book, we will use a programming language called Python to carry out deep learning. Python is a very popular programming language that is easy for beginners to use. It is especially suitable for making prototypes. You can try your ideas immediately and conduct various experiments while checking the results. This book describes the theoretical aspects of deep learning while implementing Python programs and conducting experiments.

What you cannot understand only with a mathematical expression or theoretical description, you can often discern by reading and running a source code, getting a feel for the flow of technologies. This book puts emphasis on "engineering"—understanding deep learning through making something by writing codes. You will see many mathematical expressions and also many source codes from a programmer's point of view.

For whom is this book written?

This book provides activities for implementation so that you can better understand deep learning. The following lists the lesson objectives for this course:

Use Python and the minimum possible external libraries to implement deep learning programs from scratch.Describe how to use Python for Python beginners.Provide Python source codes that you can run and a learning environment where you can experiment.Implement a system that recognizes images with high precision, starting from simple machine learning problemsClearly explain deep learning and neural network theories.Explain technologies that look complicated, such as backpropagation and convolution, so that you can understand them on an implementation level.Describe useful and practical techniques for deep learning, such as how to determine learning coefficients and the initial values of weights.Describe and implement trends such as Batch Normalization, Dropout, and Adam.Determine the reasons why deep learning is excellent, why a deeper layer improves recognition precision, and why a hidden layer is important.Introduce applications of deep learning, such as automatic driving, image generation, and reinforcement learning.

For whom is this book not written?

It is also important to show for whom this book is not written. The following lists what we will not do in this book.

This book…

does not describe or introduce the latest research about deep learning in detail.does not describe how to use deep learning frameworks, such as Caffe, TensorFlow, and PyTorch.does not provide a detailed theoretical descriptions about deep learning, particularly neural networks.does not provide detailed description about tuning for improving recognition precision in deep learning.does not deal with GPU implementation to accelerate deep learning.focuses on image recognition. It does not cover natural language processing or speech recognition.

Thus, this book does not cover the latest research or theoretical details. However, when you are nearing the end of this book, you can then proceed with the latest research papers and theoretical books about neural networks.

This book focuses on image recognition. You can mainly learn the technologies required for image recognition using deep learning. It does not cover natural language processing or speech recognition.

How to read this book

When we learn something new, we often fail to understand it or may soon forget it by only listening to explanations. As an ancient Chinese philosopher said, You forget what you heard. You learn what you saw. You can understand what you did. Practice is the most important when you learn something new. This book often provides practical examples (source codes that you can run) after explaining a concept.

This book provides Python code that you can run on your computer for practical implementation to reinforce and apply theory, as well as experimentation through trial and error.

This book will advance on the pair of wheels of "theoretical explanation" and "implementation by Python." It is recommended that you have an environment for programming. You can use a Windows, Mac, or Linux computer with this book. Chapter 1Introduction to Python describes how to install and use Python. You can download the programs used in this book from the following GitHub repository.

https://github.com/koki0702/deep-learning-from-the-basics

This ends the introduction and should have given you an overview of what you will be doing in this book and, hopefully, the desire to continue reading further.

In these next chapters, you will conduct various experiments during the process. Sometimes we are at a loss and stop to think why something has happened. Such time-consuming activities give us important knowledge in understanding technology deeply. The knowledge obtained after such a long time is surely useful also in using existing libraries, reading cutting-edge papers, and making an original system. Above all, making something is fun. Now, we are ready. Let's start on our journey of deep learning!

Acknowledgments

First, I would like to thank the researchers and engineers who have conducted research of technologies about deep learning: machine learning and computer science. It is thanks to them that I can write this book. I also thank those who have published useful information in books and on the Web. Above all, I learned a lot from the spirit of generously providing useful technologies and information in the open class CS231n (Convolutional Neural Networks for Visual Recognition (http://cs231n.github.io/) at Stanford University.

The following people have contributed to my writing. Tetsuro Kato, Shinya Kita, Yuka Tobinaga, Kota Nakano, Masatatu Nakamura, Akihiro Hayashi, and Ryo Yamamoto at teamLab, Inc., Kenshi Muto and Moe Masuko at Top Studio Co., Kenji Nomura at Flickfit, and Hidetaka Tanno, JSPS oversea research fellow in The University of Texas at Austin. These people read the manuscript of this book and provided much advice. I would like to thank them here. I say clearly that the author is responsible for any defects and errors in this book.

Finally, I thank Naoki Miyagawa at O'Reilly Japan for his continuous support for about one year and half from the conception to completion of this book. Thank you.

September 1, 2016

Koki Saitoh

1. Introduction to Python