Mastering PyTorch E-Book

Ashish Ranjan Jha

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 441

Veröffentlichungsjahr: 2021

Beschreibung

Deep learning is driving the AI revolution, and PyTorch is making it easier than ever before for anyone to build deep learning applications. This PyTorch book will help you uncover expert techniques to get the most out of your data and build complex neural network models.
The book starts with a quick overview of PyTorch and explores using convolutional neural network (CNN) architectures for image classification. You'll then work with recurrent neural network (RNN) architectures and transformers for sentiment analysis. As you advance, you'll apply deep learning across different domains, such as music, text, and image generation using generative models and explore the world of generative adversarial networks (GANs). You'll not only build and train your own deep reinforcement learning models in PyTorch but also deploy PyTorch models to production using expert tips and techniques. Finally, you'll get to grips with training large models efficiently in a distributed manner, searching neural architectures effectively with AutoML, and rapidly prototyping models using PyTorch and fast.ai.
By the end of this PyTorch book, you'll be able to perform complex deep learning tasks using PyTorch to build smart artificial intelligence models.

Leseprobe

Mastering PyTorch

Build powerful neural network architectures using advanced PyTorch 1.x features

Ashish Ranjan Jha

BIRMINGHAM—MUMBAI

Mastering PyTorch

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Parikh

Publishing Product Manager: Devika Battike

Senior Editor: Roshan Kumar

Content Development Editor: Tazeen Shaikh

Technical Editor: Sonam Pandey

Copy Editor: Safis Editing

Project Coordinator: Aishwarya Mohan

Proofreader: Safis Editing

Indexer: Priyanka Dhadke

Production Designer: Nilesh Mohite

First published: February 2021

Production reference: 2090221

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78961-438-1

www.packt.com

To my mother and best-friend - Rani Jha, my father and idol - Bibhuti Bhushan Jha, for their sacrifices and constant support and for being the driving forces of my life and career. Without their love, none of this would matter. To my sisters, Sushmita, Nivedita, and Shalini, for teaching me what and what not to do in life.

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and videos from over 4,000 industry professionalsImprove your learning with Skill Plans built especially for youGet a free eBook or video every monthFully searchable for easy access to vital informationCopy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and, as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Foreword

I am happy to know that Ashish, who was my student on the artificial neural networks course 8 years ago at IIT Roorkee, has now authored this hands-on book that covers a range of deep learning topics in reasonable depth.

Learning by coding is something every deep learning enthusiast wants to undertake, but tends to leave half way through. The effort needed to go through documentation and extract useful information to run deep learning projects is cumbersome. I have seen far too many students become frustrated during the process. There are tons of resources available for any beginner to become an expert. However, it is easy for any beginner to lose sight of the learning task while trying to strike a balance between concept-oriented courses and the coding-savvy approach of many academic programs.

PyTorch is uniquely placed as being pythonic and very flexible. It is appealing both to beginners who have just started coding machine learning models and to experts who like to meddle in the finer parameters of model designing and training. PyTorch is one library I am happy to recommend to any enthusiast, regardless of their level of expertise.

The best way to learn machine learning and deep learning models is by practicing coding in PyTorch. This book navigates the world of deep learning through PyTorch in a very engaging way. It starts from the basic building blocks of deep learning. The visual appeal of learning the data pipeline is one of its strong points. The PyTorch modules used for model building and training are introduced in the simplest of ways. Any student will appreciate the hands-on approach of this book. Every concept is explained through codes, and every step of the code is well documented. It should not be assumed that this book is just for beginners. Instead, any beginner can become an expert by following this book.

Starting from basic model building, such as the popular VGG16 or ResNet, to advanced topics, such as AutoML and distributed learning, all these aspects are covered here. The book further encompasses concepts such as AI explainability, deep reinforcement learning, and GANs. The exercises in this book range from building an image captioning model to music generation and neural style transfer models, as well as building PyTorch model servers in production systems. This helps you to prepare for any niche deep learning ventures.

I recommend this book to anyone who wants to master PyTorch for deploying deep learning models with the latest libraries.

Dr. Gopinath Pillai Head Of Department, Electrical Engineering, IIT Roorkee

Contributors

About the author

Ashish Ranjan Jha received his bachelor's degree in electrical engineering from IIT Roorkee (India), his master's degree in computer science from EPFL (Switzerland), and an MBA degree from the Quantic School of Business (Washington). He received distinctions in all of his degrees. He has worked for a variety of tech companies, including Oracle and Sony, and tech start-ups, such as Revolut, as a machine learning engineer.

Aside from his years of work experience, Ashish is a freelance ML consultant, an author, and a blogger (datashines). He has worked on products/projects ranging from using sensor data for predicting vehicle types to detecting fraud in insurance claims. In his spare time, Ashish works on open source ML projects and is active on StackOverflow and kaggle (arj7192).

About the reviewer

Javier Abascal Carrasco has a master's degree in telecommunication engineering from the University of Seville (Spain). He also studied abroad at TU Dresden (Germany) and Thomas College (ME, USA), where he obtained his MBA. Since his career started, Javier has been passionate about the world of data and analytics. He has had the chance to work with and help all manner of companies, ranging from small start-ups to big corporations, including the consulting firm EY and Facebook. In addition, for the last 3 years, he has been a part-time lecturer on the data science space. He truly believes that PyTorch is bringing a new, fresh style to programming and work involving deep learning, generating a friendly competitor landscape in relation to TensorFlow.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Preface

Section 1: PyTorch Overview

Chapter 1: Overview of Deep Learning using PyTorch

Technical requirements

A refresher on deep learning

Activation functions

Optimization schedule

Exploring the PyTorch library

PyTorch modules

Tensor modules

Training a neural network using PyTorch

Summary

Chapter 2: Combining CNNs and LSTMs

Technical requirements

Building a neural network with CNNs and LSTMs

Text encoding demo

Building an image caption generator using PyTorch

Downloading the image captioning datasets

Preprocessing caption (text) data

Preprocessing image data

Defining the image captioning data loader

Defining the CNN-LSTM model

Training the CNN-LSTM model

Generating image captions using the trained model

Summary

Section 2: Working with Advanced Neural Network Architectures

Chapter 3: Deep CNN Architectures

Technical requirements

Why are CNNs so powerful?

Evolution of CNN architectures

Developing LeNet from scratch

Using PyTorch to build LeNet

Training LeNet

Testing LeNet

Fine-tuning the AlexNet model

Using PyTorch to fine-tune AlexNet

Running a pre-trained VGG model

Exploring GoogLeNet and Inception v3

Inception modules

1x1 convolutions

Global average pooling

Auxiliary classifiers

Inception v3

Discussing ResNet and DenseNet architectures

DenseNet

Understanding EfficientNets and the future of CNN architectures

Summary

Chapter 4: Deep Recurrent Model Architectures

Technical requirements

Exploring the evolution of recurrent networks

Types of recurrent neural networks

RNNs

Bidirectional RNNs

LSTMs

Extended and bidirectional LSTMs

Multi-dimensional RNNs

Stacked LSTMs

GRUs

Grid LSTMs

Gated orthogonal recurrent units

Training RNNs for sentiment analysis

Loading and preprocessing the text dataset

Instantiating and training the model

Building a bidirectional LSTM

Loading and preprocessing text dataset

Instantiating and training the LSTM model

Discussing GRUs and attention-based models

GRUs and PyTorch

Attention-based models

Summary

Chapter 5: Hybrid Advanced Models

Technical requirements

Building a transformer model for language modeling

Reviewing language modeling

Understanding the transformer model architecture

Developing a RandWireNN model from scratch

Understanding RandWireNNs

Developing RandWireNNs using PyTorch

Summary

Section 3: Generative Models and Deep Reinforcement Learning

Chapter 6: Music and Text Generation with PyTorch

Technical requirements

Building a transformer-based text generator with PyTorch

Training the transformer-based language model

Saving and loading the language model

Using the language model to generate text

Using a pre-trained GPT-2 model as a text generator

Out-of-the-box text generation with GPT-2

Text generation strategies using PyTorch

Generating MIDI music with LSTMs using PyTorch

Loading the MIDI music data

Defining the LSTM model and training routine

Training and testing the music generation model

Summary

Chapter 7: Neural Style Transfer

Technical requirements

Understanding how to transfer style between images

Implementing neural style transfer using PyTorch

Loading the content and style images

Loading and trimming the pre-trained VGG19 model

Building the neural style transfer model

Training the style transfer model

Experimenting with the style transfer system

Summary

Chapter 8: Deep Convolutional GANs

Technical requirements

Defining the generator and discriminator networks

Understanding the DCGAN generator and discriminator

Training a DCGAN using PyTorch

Defining the generator

Defining the discriminator

Loading the image dataset

Training loops for DCGANs

Using GANs for style transfer

Understanding the pix2pix architecture

Summary

Chapter 9: Deep Reinforcement Learning

Technical requirements

Reviewing reinforcement learning concepts

Types of reinforcement learning algorithms

Discussing Q-learning

Understanding deep Q-learning

Using two separate DNNs

Experience replay buffer

Building a DQN model in PyTorch

Initializing the main and target CNN models

Defining the experience replay buffer

Setting up the environment

Defining the CNN optimization function

Managing and running episodes

Training the DQN model to learn Pong

Summary

Section 4: PyTorch in Production Systems

Chapter 10: Operationalizing PyTorch Models into Production

Technical requirements

Model serving in PyTorch

Creating a PyTorch model inference pipeline

Building a basic model server

Creating a model microservice

Serving a PyTorch model using TorchServe

Installing TorchServe

Launching and using a TorchServe server

Exporting universal PyTorch models using TorchScript and ONNX

Understanding the utility of TorchScript

Model tracing with TorchScript

Model scripting with TorchScript

Running a PyTorch model in C++

Using ONNX to export PyTorch models

Serving PyTorch models in the cloud

Using PyTorch with AWS

Serving PyTorch model on Google Cloud

Serving PyTorch models with Azure

Summary

References

Chapter 11: Distributed Training

Technical requirements

Distributed training with PyTorch

Training the MNIST model in a regular fashion

Training the MNIST model in a distributed fashion

Distributed training on GPUs with CUDA

Summary

Chapter 12: PyTorch and AutoML

Technical requirements

Finding the best neural architectures with AutoML

Using Auto-PyTorch for optimal MNIST model search

Using Optuna for hyperparameter search

Defining the model architecture and loading dataset

Defining the model training routine and optimization schedule

Running Optuna's hyperparameter search

Summary

Chapter 13: PyTorch and Explainable AI

Technical requirements

Model interpretability in PyTorch

Training the handwritten digits classifier – a recap

Visualizing the convolutional filters of the model

Visualizing the feature maps of the model

Using Captum to interpret models

Setting up Captum

Exploring Captum's interpretability tools

Summary

Chapter 14: Rapid Prototyping with PyTorch

Technical requirements

Using fast.ai to set up model training in a few minutes

Setting up fast.ai and loading data

Training a MNIST model using fast.ai

Evaluating and interpreting the model using fast.ai

Training models on any hardware using PyTorch Lightning

Defining the model components in PyTorch Lightning

Training and evaluating the model using PyTorch Lightning

Summary

Other Books You May Enjoy

Section 1: PyTorch Overview

This section includes a refresher on deep learning concepts, as well as PyTorch essentials. Upon completing this section, you will be able to identify how to train your own PyTorch models, as well as how to build a neural network model that generates text/captions as output when given images as input using PyTorch.

This section comprises the following chapters:

Chapter 1, Overview of Deep Learning Using PyTorchChapter 2, Combining CNNs and LSTMs