23,99 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Transform games into environments using machine learning and Deep learning with Tensorflow, Keras, and Unity
Key Features
- Learn how to apply core machine learning concepts to your games with Unity
- Learn the Fundamentals of Reinforcement Learning and Q-Learning and apply them to your games
- Learn How to build multiple asynchronous agents and run them in a training scenario
Book Description
Unity Machine Learning agents allow researchers and developers to create games and simulations using the Unity Editor, which serves as an environment where intelligent agents can be trained with machine learning methods through a simple-to-use Python API.
This book takes you from the basics of Reinforcement and Q Learning to building Deep Recurrent Q-Network agents that cooperate or compete in a multi-agent ecosystem. You will start with the basics of Reinforcement Learning and how to apply it to problems. Then you will learn how to build self-learning advanced neural networks with Python and Keras/TensorFlow. From there you move o n to more advanced training scenarios where you will learn further innovative ways to train your network with A3C, imitation, and curriculum learning models. By the end of the book, you will have learned how to build more complex environments by building a cooperative and competitive multi-agent ecosystem.
What you will learn
- Develop Reinforcement and Deep Reinforcement Learning for games.
- Understand complex and advanced concepts of reinforcement learning and neural networks
- Explore various training strategies for cooperative and competitive agent development
- Adapt the basic script components of Academy, Agent, and Brain to be used with Q Learning.
- Enhance the Q Learning model with improved training strategies such as Greedy-Epsilon exploration
- Implement a simple NN with Keras and use it as an external brain in Unity
- Understand how to add LTSM blocks to an existing DQN
- Build multiple asynchronous agents and run them in a training scenario
Who this book is for
This book is intended for developers with an interest in using Machine learning algorithms to develop better games and simulations with Unity.
The reader will be required to have a working knowledge of C# and a basic understanding of Python.
Micheal Lanham is a proven software architect with 20 years' experience of developing a range of software, including games, mobile, graphic, web, desktop, engineering, GIS, and machine learning applications for various industries. In 2000, Micheal began working with machine learning and would later use various technologies for a broad range of apps, from geomechanics to inspecting pipelines in 3D. He was later introduced to Unity and has been an avid developer and author of multiple Unity apps and books since.Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 213
Veröffentlichungsjahr: 2018
Ähnliche
Learn Unity ML - Agents - Fundamentals of Unity Machine Learning
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Kunal ChaudhariAcquisition Editor: Reshma RamanContent Development Editor: Roshan KumarTechnical Editor: Shweta JadhavCopy Editor: Safis EditingProject Coordinator: Hardik BhindeProofreader: Safis EditingIndexer: Rekha NairGraphics:Jason MonteiroProduction Coordinator: Shraddha Falebhai
First published: June 2018
Production reference: 1290618
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78913-813-9
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and, as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the author
Micheal Lanham is a proven software architect with 20 years' experience of developing a range of software, including games, mobile, graphic, web, desktop, engineering, GIS, and machine learning applications for various industries. In 2000, Micheal began working with machine learning and would later use various technologies for a broad range of apps, from geomechanics to inspecting pipelines in 3D. He was later introduced to Unity and has been an avid developer and author of multiple Unity apps and books since.
About the reviewers
Michael Oakes has worked in the IT industry for over 18 years and is a graduate from the University of Westminster and a Unity Certified Developer.
He is currently working as an augmented/virtual reality consultant and AI developer with a Canadian mobile company, and with his own company, Canunky Solutions, developing custom augmented/virtual reality and AI applications. Originally from Grimsby in the UK (home to his beloved Mariners football team), he now lives in Calgary, Canada, with his wife, Camie, and two cats (Peanut and Sammy).
Casey Cupp is a software developer with a focus on customer-based problem solving using full-stack technologies. He has worked on sustainable development in multitier data applications with an emphasis on utilizing GIS and machine learning technologies. He is the Lead Developer for Petroweb in the oil and gas data management industry, and a senior developer for Sales Temperature, a machine learning start-up in retail forecasting.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Learn Unity ML - Agents - Fundamentals of Unity Machine Learning
Dedication
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the author
About the reviewers
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Introducing Machine Learning and ML-Agents
Machine Learning
Training models
A Machine Learning example
ML uses in gaming
ML-Agents
Running a sample
Setting the agent Brain
Creating an environment
Renaming the scripts
Academy, Agent, and Brain
Setting up the Academy
Setting up the Agent
Setting up the Brain
Exercises
Summary
The Bandit and Reinforcement Learning
Reinforcement Learning
Configuring the Agent
Contextual bandits and state
Building the contextual bandits
Creating the ContextualDecision script
Updating the Agent
Exploration and exploitation
Making decisions with SimpleDecision
MDP and the Bellman equation
Q-Learning and connected agents
Looking at the Q-Learning ConnectedDecision script
Exercises
Summary
Deep Reinforcement Learning with Python
Installing Python and tools
Installation
Mac/Linux installation
Windows installation
Docker installation
GPU installation
Testing the install
ML-Agents external brains
Running the environment
Neural network foundations
But what does it do?
Deep Q-learning
Building the deep network
Training the model
Exploring the tensor
Proximal policy optimization
Implementing PPO
Understanding training statistics with TensorBoard
Exercises
Summary
Going Deeper with Deep Learning
Agent training problems
When training goes wrong
Fixing sparse rewards
Fixing the observation of state
Convolutional neural networks
Experience replay
Building on experience
Partial observability, memory, and recurrent networks
Partial observability
Memory and recurrent networks
Asynchronous actor – critic training
Multiple asynchronous agent training
Exercises
Summary
Playing the Game
Multi-agent environments
Adversarial self-play
Using internal brains
Using trained brains internally
Decisions and On-Demand Decision Making
The Bouncing Banana
Imitation learning
Setting up a cloning behavior trainer
Curriculum Learning
Exercises
Summary
Terrarium Revisited – A Multi-Agent Ecosystem
What was/is Terrarium?
Building the Agent ecosystem
Importing Unity assets
Building the environment
Basic Terrarium – Plants and Herbivores
Herbivores to the rescue
Building the herbivore
Training the herbivore
Carnivore: the hunter
Building the carnivore
Training the carnivore
Next steps
Exercises
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
Machine learning (ML) has been described as the next technological wave to hit humankind, akin to that of electricity. While this is a big claim, we can make certain analogies between the two technologies. For one, you really don't need to understand the inner workings of electricity to use it, and in some ways that applies to ML and many of the more advanced concepts. If you wire up a light the wrong way, it won't work, or you could hurt yourself, and the same analogy applies to machine learning. You still need enough knowledge to call yourself an MLtician or ML practitioner (if you will), and it is the goal of this book to give you that depth of knowledge. Now, the area of ML is broad, so our focus in this book will be to use deep reinforcement learning (DRL) in the form of Unity ML-Agents. DRL is currently a hot topic for developing robotic and simulation agents in many areas, and it is certainly a great addition to the Unity platform.
Who this book is for
This book is for anyone who wants a good practical introduction to some specific ML technologies that work and are very fun to play with. While this book covers some very advanced topics, anyone with a high-school level of math, patience, and understanding of C# will be able to work through all the exercises. We do feature example Python code and use Python for most of the training, but only a superficial knowledge of the language is required.
What this book covers
Chapter 1, Introducing Machine Learning and ML-Agents, covers the basics of machine learning and introduces the ML-Agents framework within Unity. This is basically just a setup chapter, but it's essential to anyone new to Unity and/or ML-Agents.
Chapter 2, The Bandit and Reinforcement Learning, introduces many of the basic problems and solutions used to teach reinforcement learning, from the multiarm and contextual bandit problems to a newly-derived connected bandit problem.
Chapter 3, Deep Reinforcement Learning with Python, explores the Python toolset available for your system and explains how to install and set up those tools. Then, we will cover the basics of neural networks and deep learning before coding up a simple reinforcement learning example.
Chapter 4, Going Deeper with Deep Learning, sets up ML-Agents to use the external Python trainers to create some fun but powerful agents that learn to explore and solve problems.
Chapter 5, Playing the Game, explains that ML-Agents is all about creating games and simulation in Unity. So, in this chapter, we will focus on various play strategies for training and interacting with agents in a real game or simulation.
Chapter 6, Terrarium Revisited and a Multi-Agent Ecosystem, revisits a coding game developed previously called Terrarium as a way to build self-learning agents who live in a little ecosystem. We learn how game rules can be applied to building a game or simulation with multiple agents that interact together.
To get the most out of this book
The following is a short list of the tools and attributes that may make you more successful as you explore this book:
Computer:
A desktop computer capable of running Unity, but all the samples are basic enough that even a low-end machine should be sufficient. Check the Unity documentation for the minimum requirements to run Unity.
Patience
: You may need to train agents for several hours, so expect to wait. Just remember that
your patience will be rewarded
(Alton Brown
)
. The better your machine, the less you wait, so there's also that.
GPU
: Don't fret if your computer does not have a support GPU to run TensorFlow; you can run the samples without it. It is nice to have, though.
High-school math
: If you need to brush up, basic statistics, algebra, and geometry should be sufficient. Developing your own apps will certainly benefit from a better understanding of the mathematics.
Programming
: A basic understanding of C# is required. You will find it helpful if you also know Unity and Python, but this is not required to run the exercises.
Download the example code files
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Learn-Unity-ML-Agents-Fundamentals-of-Unity-Machine-Learning. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/LearnUnityMLAgentsFundamentalsofUnityMachineLearning_ColorImages.pdf.
Get in touch
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Introducing Machine Learning and ML-Agents
All around us, our perception of learning and intellect is being challenged daily with the advent of new and emerging technologies. From self-driving cars, playing Go and Chess, to computers being able to beat humans at classic Atari games, the advent of a group of technologies we colloquially call Machine Learning have come to dominate a new era in technological growth – a new era of growth that has been compared with the same importance as the discovery of electricity and has already been categorized as the next human technological age.
This book is intended to introduce you to a very small slice of that new era in a fun and informative way using the Machine Learning Agents platform called ML-Agents from Unity. We will first explore some basics of Machine Learning and ML-Agents. Then, we will cover training and specifically Reinforcement Learning and Q Learning. After that, we will learn how to use Keras to build a Neural Network that we will evolve into a Deep Q-Network. From there, we will look at various ways to improve the Deep Q-Network with different training strategies. This will lead us to our first example, where we train an agent to play a more complex game. Then, finally, we will finish with a look at a multi-agent example that allows agents to compete with or against each other.
In our first chapter, we will take a gradual introduction to ML and ML-Agents. Here is what we will cover in this chapter:
Machine Learning
ML-Agents
Running an example
Creating an environment
Academy, Agent, and Brain
Let's get started, and in the next section, we will introduce what Machine Learning is and the particular aspect of ML we plan to focus on in this book.
Machine Learning
Games and simulations are no stranger to AI technologies and there are numerous assets available to the Unity developer in order to provide simulated machine intelligence. These technologies include content like Behavior Trees, Finite State Machine, navigation meshes, A*, and other heuristic ways game developers use to simulate intelligence. So, why Machine Learning and why now? After all, many of the base ML techniques, like neural nets, we will use later in this book have been used in games before.
The reason, is due in large part to the OpenAI initiative, an initiative that encourages research across academia and the industry to share ideas and research on AI and ML. This has resulted in an explosion of growth in new ideas, methods, and areas for research. This means for games and simulations that we no longer have to fake or simulate intelligence. Now, we can build agents that learn from their environment and even learn to beat their human builders.
Training models
Machine Learning is so aptly named because it uses various forms of training to analyze data or state and provide that trained response. These methods are worth mentioning and we will focus on one particular method of learning that is currently showing good success. Before we get to that though, for later chapters, let's breakdown the three types of training we frequently see in ML:
Unsupervised Training
: This method of training examines a dataset on its own and performs a classification. The classification may be based on certain metrics and can be discovered by the training itself. Most people used to think that all AI or ML worked this way, but of course, it does not:
ESRI
, which is a major mapping provider of GIS software and data provides a demographic dataset called
Tapestry
. This dataset is derived from a combination of US census data and other resources. It is processed through an ML algorithm that classifies the data into 68 consumer segments using Unsupervised Training. The Tapestry data is not free but can be invaluable for anyone building ML for a consumer or retail application.
Supervised Training
: This is the typical training method most data science ML methods use to perform prediction or classification. It is a type of training that requires input and output data be labelled. As such, it requires a set of training data in order to build a model. Oftentimes, depending on the particular ML technique, it can require vast amounts of data:
Google Inception
is an image classification ML model that is freely available. It has been trained by millions of images into various trained classifications. The Inception model is small enough to fit on a mobile device in order to provide real-time image classification.
Reinforcement Learning
: This is based on control theory and provides a method of learning without any initial state or model of the environment. This is a powerful concept because it eliminates the need to model the environment or undertake the tedious data labeling often required by Supervised Training. Instead, agents are modeled in the environment and receive rewards based on their actions. Of course, that also means that this advanced method of training is not without its pitfalls and frustrations. We will start learning the details of RL in
Chapter 2
,
The Bandit and Reinforcement Learning
:
DeepMind
built the bot that was able to play classic Atari 2600 games better than a human.
Imitation Learning
: This is a technique where agents are trained by watching a demonstration of the desired actions and then imitating them. This is a powerful technique and has plenty of applications. We will explore this type of training in
Chapter 4
,
Going Deeper with Deep Learning
.
Curriculum Learning
: This is an advanced form of learning that works by breaking down a problem into levels of complexity, which allows the agent or ML to overcome each level of complexity before moving on to more advanced activities. For example, an agent waiter may first need to learn to balance a tray, then the tray with a plate of food, then walking with the tray and food, and finally delivering the food to a table. We will explore this form of training in
Chapter 5
,
Playing the Game
.
Deep Learning
: This uses various forms of internal training mechanisms to train a multi-layer neural network. We will spend more time on neural networks and Deep Learning in
Chapter 3
,
Deep Reinforcement Learning with Python
.
You may have already noticed the interchange of terms ML and agent use to denote the thing that is learning. It is helpful to think of things in these terms for now. Later in this chapter, we will start to distinguish the differences between an agent and their brain or ML. For now, though, let's get back to some basics and explore a simple ML example in the next section.
A Machine Learning example
In order to demonstrate some of these concepts in a practical manner, let's look at an example scenario where we use ML to solve a game problem. In our game, we have a cannon that shoots a projectile at a specific velocity in a physics-based world. The object of the game is to choose the velocity to hit the target at a specific distance. We have already fired the cannon ten times and recorded the results in a table and chart, as shown in the following screenshot:
Since the data is labelled already, this problem is well-suited for Supervised Training. We will use a very simple method called linear regression in order to give us a model that can predict a velocity in order to hit a target at a certain distance. Microsoft Excel provides a quick way for us to model linear regression on the chart by adding a trendline, as follows:
By using this simple feature in Excel, you can quickly analyze your data and see an equation that best fits that data. Now, this is a rudimentary example of data science, but hopefully you can appreciate how this can easily be used to predict complex environments just based on the data. While the linear regression model can provide us with an answer, it obviously is not very good and the R2
