40,81 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Apply modern deep learning techniques to build and train deep neural networks using Gorgonia
Key Features
- Gain a practical understanding of deep learning using Golang
- Build complex neural network models using Go libraries and Gorgonia
- Take your deep learning model from design to deployment with this handy guide
Book Description
Go is an open source programming language designed by Google for handling large-scale projects efficiently. The Go ecosystem comprises some really powerful deep learning tools such as DQN and CUDA. With this book, you'll be able to use these tools to train and deploy scalable deep learning models from scratch.
This deep learning book begins by introducing you to a variety of tools and libraries available in Go. It then takes you through building neural networks, including activation functions and the learning algorithms that make neural networks tick. In addition to this, you'll learn how to build advanced architectures such as autoencoders, restricted Boltzmann machines (RBMs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and more. You'll also understand how you can scale model deployments on the AWS cloud infrastructure for training and inference.
By the end of this book, you'll have mastered the art of building, training, and deploying deep learning models in Go to solve real-world problems.
What you will learn
- Explore the Go ecosystem of libraries and communities for deep learning
- Get to grips with Neural Networks, their history, and how they work
- Design and implement Deep Neural Networks in Go
- Get a strong foundation of concepts such as Backpropagation and Momentum
- Build Variational Autoencoders and Restricted Boltzmann Machines using Go
- Build models with CUDA and benchmark CPU and GPU models
Who this book is for
This book is for data scientists, machine learning engineers, and AI developers who want to build state-of-the-art deep learning models using Go. Familiarity with basic machine learning concepts and Go programming is required to get the best out of this book.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 239
Veröffentlichungsjahr: 2019
Ähnliche
Hands-On Deep Learning with Go
Copyright © 2019 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Pravin DhandreAcquisition Editor: Joshua NadarContent Development Editor: Roshan KumarSenior Editor: Jack CummingsTechnical Editor: Dinesh ChaudharyCopy Editor: Safis EditingProject Coordinator: Namrata SwettaProofreader: Safis EditingIndexer:Manju ArasanProduction Designer:Jayalaxmi Raja
First published: August 2019
Production reference: 1060819
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78934-099-0
www.packtpub.com
Packt.com
Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Fully searchable for easy access to vital information
Copy and paste, print, and bookmark content
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the authors
Gareth Seneque is a machine learning engineer with 11 years' experience of building and deploying systems at scale in the finance and media industries. He became interested in deep learning in 2014 and is currently building a search platform within his organization, using neuro-linguistic programming and other machine learning techniques to generate content metadata and drive recommendations. He has contributed to a number of open source projects, including CoREBench and Gorgonia. He also has extensive experience with modern DevOps practices, using AWS, Docker, and Kubernetes to effectively distribute the processing of machine learning workloads.
Darrell Chua is a senior data scientist with more than 10 years' experience. He has developed models of varying complexity, from building credit scorecards with logistic regression to creating image classification models for trading cards. He has spent the majority of his time working with in fintech companies, trying to bring machine learning technologies into the world of finance. He has been programming in Go for several years and has been working on deep learning models for even longer. Among his achievements is the creation of numerous business intelligence and data science pipelines that enable the delivery of a top-of-the-line automated underwriting system, producing near-instant approval decisions.
About the reviewer
Xuanyi Chew is the primary author of Gorgonia. In his day job, he is the chief data scientist of a rapidly growing local start-up in Sydney. At night, he works on his hobbies of building deep learning AI (using Gorgonia), furthering his hopes of one day building an AGI. He wants to make Go the primary ecosystem for machine learning work and would love your help.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Hands-On Deep Learning with Go
About Packt
Why subscribe?
Contributors
About the authors
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Section 1: Deep Learning in Go, Neural Networks, and How to Train Them
Introduction to Deep Learning in Go
Introducing DL
Why DL?
DL – a history 
DL – hype or breakthrough?
Defining deep learning
Overview of ML in Go 
ML libraries
Word-embeddings in Go 
Naive Bayesian classification and genetic algorithms for Go or Golang
ML for Go 
Machine learning libraries for Golang 
GoBrain
A set of numeric libraries for the Go programming language
Using Gorgonia
The basics of Gorgonia
Simple example – addition
Vectors and matrices
Visualizing the graph
Building more complex expressions
Summary
What Is a Neural Network and How Do I Train One?
A basic neural network
The structure of a neural network
Your first neural network
Activation functions
Step functions
Linear functions
Rectified Linear Units
Leaky ReLU
Sigmoid functions
Tanh
But which one should we use?
Gradient descent and backpropagation
Gradient descent
Backpropagation
Stochastic gradient descent
Advanced gradient descent algorithms
Momentum
Nesterov momentum
RMSprop
Summary
Beyond Basic Neural Networks - Autoencoders and RBMs
Loading data – MNIST
What is MNIST?
Loading MNIST
Building a neural network for handwriting recognition
Introduction to the model structure
Layers
Training
Loss functions
Epochs, iterations, and batch sizes
Testing and validation
Taking a closer look
Exercises
Building an autoencoder – generating MNIST digits
Layers
Training
Loss function
Input and output
Epochs, iterations, and batch sizes
Test and validation
Building an RBM for Netflix-style collaborative filtering
Introduction to RBMs
RBMs for collaborative filtering
Preparing our data – GroupLens movie ratings
Building an RBM in Gorgonia
Summary
Further reading
CUDA - GPU-Accelerated Training
CPUs versus GPUs
Computational workloads and chip design
Memory access in GPUs
Real-world performance
Intel Xeon Phi CPU
NVIDIA Maxwell GPU
Understanding Gorgonia and CUDA
CUDA
Basic Linear Algebra Subprograms
CUDA in Gorgonia
Building a model in Gorgonia with CUDA support
Installing CUDA support for Gorgonia
Linux
Windows
Performance benchmarking of CPU versus GPU models for training and inference
How to use CUDA
CPU results
GPU results
Summary
Section 2: Implementing Deep Neural Network Architectures
Next Word Prediction with Recurrent Neural Networks
Vanilla RNNs
Training RNNs
Backpropagation through time 
Cost function
RNNs and vanishing gradients
Augmenting your RNN with GRU/LSTM units
Long Short-Term Memory units
Gated Recurrent Units
Bias initialization of gates
Building an LSTM in Gorgonia
Representing text data
Importing and processing input
Summary
Further reading
Object Recognition with Convolutional Neural Networks
Introduction to CNNs
What is a CNN?
Normal feedforward versus ConvNet
Layers
Convolutional layer
Pooling layer
Basic structure
Building an example CNN
CIFAR-10
Epochs and batch size
Accuracy
Constructing the layers
Loss function and solver
Test set output
Assessing the results
GPU acceleration
CNN weaknesses
Summary
Further reading
Maze Solving with Deep Q-Networks
What is a DQN?
Q-learning
Optimization and network architecture
Remember, act, and replay!
Solving a maze using a DQN in Gorgonia
Summary 
Further reading
Generative Models with Variational Autoencoders
Introduction to VAEs
Building a VAE on MNIST
Encoding
Sampling
Decoding
Loss or cost function
Assessing the results
Changing the latent dimensions
Summary
Further reading
Section 3: Pipeline, Deployment, and Beyond!
Building a Deep Learning Pipeline
Exploring Pachyderm
Installing and configuring Pachyderm
Getting data into Pachyderm
Integrating our CNN
Creating a Docker image of our CNN
Updating our CNN to save the model
Creating a data pipeline
Interchangeable models
Mapping predictions to models
Using the Pachyderm dashboard
Summary
Scaling Deployment
Lost (and found) in the cloud
Building deployment templates
High-level steps
Creating or pushing Docker images
Preparing your AWS account
Creating or deploying a Kubernetes cluster
Kubernetes
Cluster management scripts
Building and pushing Docker containers
Running a model on a K8s cluster
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
Go is an open source programming language designed by Google to handle huge projects efficiently. It makes building reliable, simple, and efficient software straightforward and easy.
This book immediately jumps into the practicalities of implementing Deep Neural Networks (DNNs) in Go. Simply put, the book's title contains its aim. This means there will be a lot of technical detail, a lot of code, and (not too much) math. By the time you finally close the book or turn off your Kindle, you'll know how (and why) to implement modern, scalable DNNs, and be able to repurpose them for your needs in whatever industry or mad science project you're involved.
Who this book is for
This book is for data scientists, machine learning engineers, and deep learning aspirants who are looking to inject deep learning into their Go applications. Familiarity with machine learning and basic Golang code is expected in order to get the most out of this book.
What this book covers
Chapter 1, Introduction to Deep Learning in Go, introduces the history and applications of deep learning. This chapter also gives an overview of ML with Go.
Chapter 2, What is a Neural Network and How Do I Train One?, covers how to build a simple neural network and how to inspect a graph, as well as many of the commonly used activation functions. This chapter also discusses some of the different options for gradient descent algorithms and optimizations for your neural network.
Chapter 3, Beyond Basic Neural Networks – Autoencoders and RBMs, shows how to build a simple multilayer neural network and an autoencoder. This chapter also explores the design and implementation of a probabilistic graphical model, an RBM, used in an unsupervised manner to create a recommendation engine for films.
Chapter 4, CUDA – GPU-Accelerated Training, looks at the hardware side of deep learning and also at exactly how CPUs and GPUs serve our computational needs.
Chapter 5, Next Word Prediction with Recurrent Neural Networks, goes into what a basic RNN is and how to train one. You will also get a clear idea of the RNN architecture, including GRU/LSTM networks.
Chapter 6, Object Recognition with Convolutional Neural Networks, shows you how to build a CNN and how to tune some of the hyperparameters (such as the number of epochs and batch sizes) in order to get the desired result and get it running smoothly on different computers.
Chapter 7, Maze Solving with Deep Q-Networks, gives an introduction to reinforcement learning and Q-learning and how to build a DQN and solve a maze.
Chapter 8, Generative Models with Variational Autoencoders, shows how to build a VAE and looks at the advantages of a VAE over a standard autoencoder. This chapter also shows how to understand the effect of varying latent space dimensions on a network.
Chapter 9, Building a Deep Learning Pipeline, looks at what data pipelines are and why we use Pachyderm to build or manage them.
Chapter 10, Scaling Deployment, looks at a number of the technologies that sit underneath Pachyderm, including Docker and Kubernetes, and also examines how we can deploy stacks to cloud infrastructure using these tools .
To get the most out of this book
This book primarily uses Go, the Gorgonia package for Go, the Cu package for Go, CUDA (plus drivers) from NVIDIA, and an NVIDIA GPU that supports CUDA. Docker is also needed for Section 3, Pipeline, Deployment, and Beyond!
Download the example code files
You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packt.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hands-On-Deep-Learning-with-Go. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/Hands-On-Deep-Learning-with-Go. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789340990_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system."
A block of code is set as follows:
type nn struct { g *ExprGraph w0, w1 *Node pred *Node}
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
intercept Ctrl+C sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
doneChan := make(chan bool, 1)
Any command-line input or output is written as follows:
sudo apt install nvidia-390 nvidia-cuda-toolkit libcupti-dev
Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel."
Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packt.com.
Section 1: Deep Learning in Go, Neural Networks, and How to Train Them
This section introduces you to deep learning (DL) and the libraries in Go that are needed to design, implement, and train deep neural networks (DNNs). We also cover the implementation of an autoencoder for unsupervised learning, and a restricted Boltzmann machine (RBM) for a Netflix-style collaborative filtering system.
The following chapters are included in this section:
Chapter 1
,
Introduction to Deep Learning in Go
Chapter 2
,
What is a Neural Network and How Do I Train One?
Chapter 3
,
Beyond Basic Neural Networks - Autoencoders and Restricted Boltzmann Machines
Chapter 4
,
CUDA - GPU-Accelerated Training
Introduction to Deep Learning in Go
This book will very quickly jump into the practicalities of implementing Deep Neural Networks (DNNs) in Go. Simply put, this book's title contains its aim. This means there will be a lot of technical detail, a lot of code, and (not too much) math. By the time you finally close this book or turn off your Kindle, you'll know how (and why) to implement modern, scalable DNNs and be able to repurpose them for your needs in whatever industry or mad science project you're involved in.
Our choice of Go reflects the maturing of the landscape of Go libraries built for the kinds of operations our DNNs perform. There is, of course, much debate about the trade-offs made when selecting languages or libraries, and we will devote a section of this chapter to our views and argue for the choices we've made.
However, what is code without context? Why do we care about this seemingly convoluted mix of linear algebra, calculus, statistics, and probability? Why use computers to recognize things in images or identify aberrant patterns in financial data? And, perhaps most importantly, what do the approaches to these tasks have in common? The initial sections of this book will try to provide some of this context.
Scientific endeavor, when broken up into the disciplines that represent their institutional and industry specialization, is governed by an idea of progress. By this, we mean a kind of momentum, a moving forward, toward some end. For example, the ideal goal of medicine is to be able to identify and cure any ailment or disease. Physicists aim to understand completely the fundamental laws of nature. Progress trends in this general direction. Science is itself an optimization method. So, what might the ultimate goal of Machine Learning (ML) be?
We'll be upfront. We think it's the creation of Artificial General Intelligence (AGI). That's the prize: a general-purpose learning computer to take care of the jobs and leave life to people. As we will see when we cover the history of Deep Learning (DL) in detail, founders of the top Artificial Intelligence (AI) labs agree that AGI represents a meta-solution to many of the complex problems in our world today, from economics to medicine to government.
This chapter will cover thefollowing topics:
Why DL?
DL—history applications
Overview of ML in Go
Using Gorgonia
Introducing DL
We will now offer a high-level view of why DL is important and how it fits into the discussion about AI. Then, we will look at the historical development of DL, as well as current and future applications.
Why DL?
So, who are you, dear reader? Why are you interested in DL? Do you have your private vision for AI? Or do you have something more modest? What is your origin story?
In our survey of colleagues, teachers, and meetup acquaintances, the origin story of someone with a more formal interest in machines has a few common features. It doesn't matter much if you grew up playing games against the computer, an invisible enemy who sometimes glitched out, or if you chased down actual bots in id Software's Quake back in the late 1990s; the idea of some combination of software and hardware thinking and acting independently had an impact on each of us early on in life.
And then, as time passed, with age, education, and exposure to pop culture, your ideas grew refined and maybe you ended up as a researcher, engineer, hacker, or hobbyist, and now you're wondering how you might participate in booting up the grand machine.
If your interests are more modest, say you are a data scientist looking to understand cutting-edge techniques, but are ambivalent about all of this talk of sentient software and science fiction, you are, in many ways, better prepared for the realities of ML in 2019 than most. Each of us, regardless of the scale of our ambition, must understand the logic of code and hard work through trial and error. Thankfully, we have very fast graphics cards.
And what is the result of these basic truths? Right now, in 2019, DL has had an impact on our lives in numerous ways. Hard problems are being solved. Some trivial, some not. Yes, Netflix has a model of your most embarrassing movie preferences, but Facebook has automatic image annotation for the visually impaired. Understanding the potential impact of DL is as simple as watching the expression of joy on the face of someone who has just seen a photo of a loved one for the first time.
DL – a history
We will now briefly cover the history of DL and the historical context from which it emerged, including the following:
The idea of
AI
The beginnings of computer science/information theory
Current academic work about the state/future of DL systems
While we are specifically interested in DL, the field didn't emerge out of nothing. It is a group of models/algorithms within ML itself, a branch of computer science. It forms one approach to AI. The other, so-called symbolic AI, revolves around hand-crafted (rather than learned) features and rules written in code, rather than a weighted model that contains patterns extracted from data algorithmically.
The idea of thinking machines, before becoming a science, was very much a fiction that began in antiquity. The Greek god of arms manufacturing, Hephaestus, built automatons out of gold and silver. They served his whims and are an early example of human imagination naturally considering what it might take to replicate an embodied form of itself.
Bringing the history forward a few thousand years, there are several key figures in 20th-century information theory and computer science that built the platform that allowed the development of AI as a distinct field, including the recent work in DL we will be covering.
The first major figure, Claude Shannon, offered us a general theory of communication. Specifically, he described, in his landmark paper, A Mathematical Theory of Computation, how to ensure against information loss when transmitting over an imperfect medium (like, say, using vacuum tubes to perform computation). This notion, particularly his noisy-channel coding theorem, proved crucial for handling arbitrarily large quantities of data and algorithms reliably, without the errors of the medium itself being introduced into the communications channel.
Alan Turing described his Turing machine in 1936, offering us a universal model of computation. With the fundamental building blocks he described, he defined the limits of what a machine might compute. He was influenced by John Von Neumann's idea of the stored-program. The key insight from Turing's work is that digital computers can simulate any process of formal reasoning (the Church-Turing hypothesis). The following diagram shows the Turing machine process:
So, you mean to tell us, Mr. Turing, that computers might be made to reason…like us?!
John Von Neumann was himself influenced by Turing's 1936 paper. Before the development of the transistor, when vacuum tubes were the only means of computation available (in systems such as ENIAC and its derivatives), John Von Neumann published his final work. It remained incomplete at his death and is entitled The Computer and the Brain. Despite remaining incomplete, it gave early consideration to how models of computation may operate in the brain as they do in machines, including observations from early neuroscience around the connections between neurons and synapses.
Since AI was first conceived as a discrete field of research in 1956, with ML coined in 1959, the field has gone through a much-discussed ebb and flow—periods where hype and funding were plentiful, and periods where private sector money was non-existent and research conferences wouldn't even accept papers that emphasized neural network approaches to building AI systems.
Within AI itself, these competing approaches cannibalized research dollars and talent. Symbolic AI met its limitations in the sheer impossibility of handcrafting rules for advanced tasks such as image classification, speech recognition, and machine translation. ML sought to radically reconfigure this process. Instead of applying a bunch of human-written rules to data and hoping to get answers, human labor was, instead, to be spent on building a machine that could infer rules from data when the answers were known. This is an example of supervised learning, where the machine learns an essential cat-ness after processing thousands of example images with an associated cat label.
Quite simply, the idea was to have a system that could generalize. After all, the goal is AGI. Take a picture of your family's newest furry feline and the computer, using its understanding of cat-ness, correctly identifies a cat! An active area of research within ML, one thought essential for building a general AI, is transfer learning, where we might take the machine that understands cat-ness and plug it into a machine that, in turn, acts when cat-ness is identified. This is the approach many AI labs around the world are taking: building systems out of systems, augmenting algorithmic weakness in one area with statistical near certainty in another, and, hopefully, building a system that better serves human (or business) needs.
The notion of serving human needs brings us to an important point regarding the ethics of AI (and the DL approaches we will be looking at). There has been much discussion in the media and academic or industry circles around the ethical implications of these systems. What does it mean for our society if we have easy, automated, widespread surveillance thanks to advances in computer vision? What about automated weapons systems or manufacturing? It is no longer a stretch to imagine vast warehouses staffed by nothing with a pulse. What then for the people who used to do those jobs?
Of course, full consideration of these important issues lies outside the scope of this book, but this is the context in which our work takes place. You will be one of the privileged few able to build these systems and move the field forward. The work of the Future of Humanity Institute at Oxford University, run by Nick Bostrom, and the Future of Life Institute, run by MIT physicist, Max Tegmark, are two examples of where the kind of academic debate around AI ethics issues is taking place. This debate is not limited to academic or non-profit circles, however; DeepMind, an Alphabet company whose goal is to be an Apollo program for AGI, launched DeepMind Ethics & Society in October 2017.
