34,79 €
Do you want to create stunning applications with HoloLens? Are you a developer who is fascinated with Microsoft HoloLens and its capabilities? If so, this is the book for you.
This book introduces and demystifies the HoloLens platform and shows you different ways of interaction with computers (mixed-reality).
You will start your mixed-reality journey by understanding different types of digital reality. You will learn to build your first holographic app. Also, you will understand holographic application integration possibilities within Line of Business Applications using Azure. Moving ahead, you will create Integrated Solutions using IoT with HoloLens. Gradually you'll learn how to create and deploy apps on a device. You will learn to publish application to the store; if you are an enterprise developer, you will also manage and distribute applications for enterprise-enabled or domain-joined HoloLens.
Finally, you will develop an end-to-end realistic holographic app, ranging from scenario identification to sketching, development, deployment, and, finally, production.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 262
Veröffentlichungsjahr: 2017
BIRMINGHAM - MUMBAI
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: June 2017
Production reference: 1160617
ISBN 978-1-78728-194-3
www.packtpub.com
Authors
Abhijit Jana
Manish Sharma
Mallikarjuna Rao
Copy Editor
Dhanya Baburaj
Reviewer
Michael Washington
Project Coordinator
Ritika Manoj
Commissioning Editor
Smeet Thakkar
Proofreader
Safis Editing
Acquisition Editor
Denim Pinto
Indexer
Tejal Daruwale Soni
Content Development Editor
Mohammed Yusuf Imaratwale
Graphics
Jason Monterio
Technical Editor
Akansha Bathija
Production Coordinator
Arvindkumar Gupta
Abhijit Jana works with Microsoft as a development consultant as part of Microsoft Services. As a consultant, his job is to help customers design, develop, and deploy enterprise-level secure solutions using Microsoft technologies. Apart from being a former Microsoft MVP (Most Valuable Professional), he is a speaker and author as well as an avid technology evangelist. He has delivered sessions at prestigious Microsoft events, such as TechED, Web Camps, Azure Camps, Community TechDays, Virtual TechDays, DevDays, and developer conferences. He loves to work with different .NET communities and help them with different opportunities.
He is a well-known author and has published many articles on various .NET community sites. You can follow him on Twitter at @abhijitjana. He has authored the book Kinect for Windows SDK Programming Guide (ISBN: 1849692386 ISBN 13: 9781849692380).
Abhijit lives in Hyderabad, India, with his wife, Ananya, and a beautiful little angel, Nilova.
Manish Sharma works with Microsoft as a solution architect as part of Microsoft Services. As, a solution architect, he is responsible for leading large enterprise-transformational engagements defining technology/solution roadmaps, conducting architecture and technical evaluations of enterprise engagements, and architecting multimillion-dollar solution developments, and maintenance of outsourcing management. He is also a technology evangelist and speaker at prestigious events, such as Microsoft TechEd on the latest and cutting edge technologies, such as HoloLens, Internet of Things, connected cars, and cloud technologies such as Azure.
Mallikarjuna Rao works with Microsoft as UX Designer/3D mixed reality artist as part of Microsoft Services. He has 15 years of experience in the animation industry, with around 12 years' international exposure.
He has worked as a senior 3D asset supervisor for feature films, television series, commercials, and game projects. Mallikarjuna has worked on pivotal projects for big names in the industry, such as Walt Disney Pictures, Disney Junior, Warner Bros Pictures, Yash Raj Films, and UTV Motion Pictures, which put him under the limelight at the international level. He has an in-depth knowledge of the 3D Asset production pipeline and UX design principles.
Writing this book would not have been possible without help of many people. We had a wonderful time while writing, which was mainly due to the skills support, dedication and motivation of people around us.
First of all, we are extremely thankful to Kundan Prakash, Ankur Agarwal and Shanmugadas Chembakassery Sivadasan, Suresh Viswanath for their continuous support and motivation from the time we started writing this book.
We are deeply thankful to the entire team at Packt Publishing, including Tushar, Parul, Narendra, Ritika, Smeet, Denim, Safis, Dhanya, Tejal, Jason and Arvind. We would like to extend our thanks to Content Development Editor, Yusuf and our Technical Editor Akansha. They have been very supportive and helpful from the beginning. Thank you all for your effort and dedication.
A sincere thanks to Michael Washington for their insightful and deep technical review. He helped us to identify and fill the gaps and improve the quality of the book.
We would like to thank Charul Pant and Akshay Dixit for their offline review of the book and sharing their feedback. It was extremely helpful to get some feedback from avid reader and technology geek like you people. Thank you very much for your time and help us improving the book over the period.
We spent time in writing when we should have sleeping, spending time with family or playing with our kids. We wouldn't have never been able to write this book without support from our family. Be it late night working or meeting over weekends, they supported us every bit and pieces.
Michael Washington is an ASP.NET, C#, and Visual Basic programmer. He has extensive knowledge of process improvement, billing systems, and student information systems. He is a Microsoft MVP. He has a son, Zachary, and resides in Los Angeles with his wife, Valerie.
< html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
The opinions in this book are purely our personal opinions and do not reflect in any way the opinion of our employers.
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at http://www.amazon.in/dp/1787281949.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
Digital Reality - Under the Hood
Virtual Reality - what it is?
Realizing Virtual Reality
Virtual Reality in the field
Augmented Reality - what it is?
Visualization techniques for Augmented Reality
Screen-based
Augmented Reality head-mounted globally displays
Projection-based
Augmented Reality in the field
Mixed Reality - what it is?
Mixed Reality in the field
Digital Reality and the current market place
Summary
HoloLens – The Most Natural Way to Interact
A closer look at HoloLens hardware
Inside the HoloLens device
Cameras and sensors
The Lenses
Speakers
Microphones
The processing units
Inertial Measurement Unit
Vent
Connectivity
Control buttons
LED indicator
Putting things together
Turning your device on/off
Cortana on HoloLens
The Mixed Reality world - HoloLens shell
Holograms in reality
HoloLens, holograms and your real world
Hologram and the Windows Holographic platform
Recording holograms - Mixed Reality capture
HoloLens interaction and application model
Interacting with holograms
Gaze
Gesture
Voice
Spatial Audio
Spatial Mapping
HoloLens app life cycles
HoloLens Clicker
Controlling your HoloLens remotely
Key differentiators
Immersive with surroundings
Independent standalone device
No screen staring
Mixed not virtual
Broad scope
Exploring the possibilities with HoloLens
Virtual collaboration
Remote expert guidance
People-centric applications
Development process and team building
Building up the team
Designers
3D Artists
Developers
Audio designers
Leads
Development process overview
Requirement Envisioning
Storyboarding
Planning, Designing and Prototyping
Planning and Design
Prototyping
Develop
Deploy and Test
Summary
Explore HoloLens as Hologram - Scenario Identification and Sketching
Envisioning
Training scenario for HoloLens
Features
Collaboration
Scenario prioritization
Scenario elaboration
Primary View
Lenses
Speakers
Cameras
Volume control buttons
Brightness control buttons
Sketching the scenarios
Sketching - interaction with speakers
Assets design and development
3D Modeling
3D Modeling tools
3D Modeling workflow
Build your 3D Model
Model sheet - base reference images
Scene creation
Importing model sheet
Import side image
Import front image
Import top image
3D Model construction with cube
Edge loop
Vertex adjustment
Mirroring
Texturing workflow
UV Unwrapping of model
Create texture in Photoshop
Export as Filmbox (FBX)
Assembling assets in Unity3D
Things to consider while using the 3D Asset Pipeline
Maintaining quads
Checking Surface Normals
Maintaining grouping
Optimizing the scene file
Setting up a development environment
Prerequisites
Installation checklist
Using HoloLens emulator
Summary
Explore HoloLens as Hologram - Developing Application and Deploying on Device
Getting started - creating a new project
The Unity3D basics
The Core interface
Scene View
Game View
Object Hierarchy
Project Assets
Inspector
The Toolbar
The Core concepts
Game Object
Components
Prefabs
Tags
Coordinate system
Setting up a project for holographic application
Setting up the HoloToolkit - Unity
Get The HoloToolkit - Unity
Preparing the package
Import the package
Applying project settings
Behind the scenes
Updating the initial scene
Importing 3D Assets
Exploring the 3D Object
Saving the scene
Build, run, and test the initial setup
Giving life to object - scripting
Highlighting the objects - enabling Gaze
The Gaze manager
The Cursor
Adding the Box Collider
See Gaze in action
Behind the scene
Adding actions to objects - applying gesture
Adding Air Tap on Lenses
Air Tap on Lenses - see it in action
Adding Air Tap on speaker
Air Tap in speaker - see it in action
Behind the scenes
Interaction using Voice Command
Enabling the Microphone capabilities
The Keyword Manager
Defining Keywords
Implementing actions for Keywords
Mapping with Keyword Manager, Keyword and actions
Voice Command - see it in action
Behind the scenes
Interaction using text to speech
Setting up text to speech Manager
Integration of text to speech
Text to speech in application launch
Text to speech in Air Tap
Text to speech - see it in action
Behind the scenes
Deploying your app
Deploy using Visual Studio
Deploy using Build Automation Window
Deploy using Device Portal
Load the package
Start the app
Deploy using Unity remoting
Configure HoloLens device for remoting
Holographic emulation - remoting from Unity
Running the app on a device
Testing your application
Lighting conditions
Space
Angle of views
View distance
Audio or visual hints
Ambient sounds
Summary
Remote Monitoring of Smart Building(s) Using HoloLens - Scenario Identification and Sketching
Envisioning
Solution scope overview
Devices and Sensors
Event Processing
Visualization
Real-time monitoring
Analyse and Act
Scenario prioritization
Scenario elaboration
Internet of Things - Remote Monitoring
Devices
Cloud Gateway (Azure IoT Hub)
Steam Processing (Azure Stream Analytics)
Data Store (Azure Storage)
Machine Learning
Real-time visualization through HoloLens
APIs
Holographic application
Sketching the scenarios
Sketching - Interaction with Rooms and Sensors
3D Model - structure and components
3D Model of building
3D Model of sensors
Summary
Remote Monitoring of Smart Building(s) Using HoloLens - Developing Application and Deploying on Device
Solution development
Logical design diagram
Setting up the backend Infrastructure
Create the Azure Event Hub
Create the Azure Cosmos DB
Create the Stream Analytics Job
Building up the Azure IoT solution
Connecting Device with Azure Event Hub - simulator
Developing the Web API
Setting up a project for a holographic application
Getting started - creating a new project
Import the package
Applying Project Settings
Updating the initial scene
Importing 3D Assets - building model
Exploring the building's 3D Objects
Save the scene
Build, run and test the initial setup
Tracking building floors from 3D Model
Adding Gaze
Adding the cursor
Adding the Box Collider
Add tag to each floor for mapping
Tracking Gaze for each floor
See Gaze in action
Verify the Gazed floor
Connecting a holographic app with Azure
Defining data models
Enabling the InternetClient capabilities
Building Azure Bridge
Getting the data
Attaching the Azure Bridge
Binding data with sensor indicators
See the live data in action
Enabling sensor binding for all floors
Adding actions to floors - applying gesture
Adding Air Tap on floors
Displaying detailed information
Designing the Panel layouts
Grouping UI elements - adding Canvas Group
Binding Data with Panel
Invoking the Panel
Closing the Panel
Air Tap on Lenses - see it in action
Deploying your app onto a device
Extending the current solution
Adding Voice Command
3D Model placement as table top mode
Additional interaction with services
Summary
Build End-to-End Retail Solution - Scenario Identification and Sketching
Envisioning
Scenario elaboration
Catalog and merchandising
Visualization
Order placement
Sketching the scenarios
Sketching - interaction for catalog and items
Sketching - interaction with 3D Model
3D Model
Summary
Build End-to-End Retail Scenario - Developing Application and Deploying on Device
Getting the 3D Assets ready
Unity Project and Asset Bundle setup
Importing 3D retail assets
Assets Bundles generation
Solution development
Logical design diagram
Building the backend Azure Solution
Creating the Azure Blob Storage
Creating the Azure Cosmos DB
Developing the Web API
Setting up the holographic project
Importing the main menu 3D Model
Exploring the main menu
Building , running and testing the initial solution
Making the main menu Intractable
Enabling Gaze with cursor
Attaching the script with menu items
Enabling Air Tap with menu items
API connector - connecting with Web API
Defining the application data model
Enabling the IntenetClient capabilities
API Connector
Getting the data
Attaching the API connector
Getting holograms on the fly
Handling the Air Tap on menu item
Downloading assets and rendering holograms on the fly
Associating toolbars with holograms
The toolbar controls
Instantiating toolbars on the fly
Showing controls based on types
Dynamic holograms in action
Adding action to the toolbar controls
Remove product
Add to Cart
Changing pictures for Wall Paints
Placing the holograms - spatial mapping
Deploying your app on a device
Extending the current solution
Dynamic menu for products
Adding Voice Command
Application and storage access security
Rotating the individual product
Summary
Possibilities
HoloLens and Microsoft Bot Framework
Solution
Client authentication
Start conversation
Send message
Receive message
HoloLens in healthcare
Solution
Search data from HealthVault
HoloLens and live streaming
Solution
Setting up media services for live streaming
Consume live streaming within holographic application
Handling secure keys with Azure Key Vault
Summary
Microsoft HoloLens in Enterprise
Microsoft HoloLens Commercial Suite
HoloLens device management using Microsoft Intune
Manual registration
Automatic registration
Installing applications on HoloLens
Device Portal for application deployment
Business Store for application deployment
Summary
Digital reality brings immersive experiences, such as transporting you to different world or places, making you interact within those immersive, mixing digital experiences with reality, and ultimately, opening new horizons to make you more productive.
The spectrum and scenarios of digital reality are huge; to understand them better, they are broken down into three different categories: Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).
Microsoft HoloLens is the most powerful mixed reality personal computing Windows device. It allows you to place your digital content in the real physical world, where it matters most to you. This is what is called an MR device, one that tries to blend the real and digital worlds seamlessly and provides new ways to visualize the world beyond the screens.
The boundaries between the real and the digital world start to blur, and you can interact with the digital world in a very similar way to how you would interact with the real, physical world. What is it about HoloLens that makes it stand out from the other devices in this genre? It is the fact that Microsoft HoloLens is the first fully untethered, self-contained computer running on Windows 10. Yes, that's right: no hassle of wires, additional cables, or connections to a computer or additional devices. It is hands-free, self-contained device with a built-in battery.
This book is mainly focused on the building mixed reality application using Microsoft HoloLens. This book doesn't require any prior knowledge about the holographic platform from the reader, and its strength is the simplicity in which the concepts have been presented using sketching, architecting, code snippets, a step-by-step process, and detailed descriptions. This book covers the following:
Understanding what mixed reality is and the possibilities of it
Understanding the process of developing enterprise holographic applications using HoloLens
Learning the application and interaction model for holographic apps
Understanding holographic application integration possibilities within Line of Business application using Azure
Learning by realistic integrated project examples
Learning end-to-end complex integrated solutions using holographic app and Azure
Chapter 1, Digital Reality - Under the Hood, begins your mixed-reality journey by explaining different types of digital realities, their differences, and usage scenarios. Also, it will explain the different market players and how they are different from each other based on their products. This chapter will also introduce you to the immersive HoloLens device and show you how to take a step further with the device.
Chapter 2, HoloLens - the Most Natural Way to Interact, introduces HoloLens as a hardware device. You will get an insight into the different components that make up HoloLens and the technology behind the device, which makes the components work together. This chapter also gives an overview of holograms and ways to interact with holograms with different interaction models. You will also become familiar with HoloLens Application Model and take a step toward developing holographic applications by understanding the team structures and roles required.
Chapter 3, Explore HoloLens as Hologram - Scenario Identification and Sketching, dives into the development process of developing your first holographic application. You will learn about identifying the scenario and the interaction models required within that scenario, such as how the user will interact with gaze, gesture, and other interaction models. You will also learn how to identify 3D Assets and take a quick overview of creating new 3D Assets.
Chapter 4, Explore HoloLens as Hologram - Developing Application and Deploying on Device, further enhances the scene developed in the preceding chapter and adds new life to it by developing different interaction models using scripting, building interaction model scripts, and testing them in Unity3D and holographic emulators. Finally, you deploy the application on a device and test your first holographic application, and visualize hologram interactions with the real world.
Chapter 5, Remote Monitoring of Smart Building(s) Using HoloLens Scenario Identification and Sketching, teaches you how to identify and design real-world integrated scenarios. You will integrate Azure IoT solutions with holographic application and visualize the data feed coming in near real time from the cloud through holograms. You will learn about integration scenario options, architecture, and the interaction models required within this scenario.
Chapter 6, Remote Monitoring of Smart Building(s) Using HoloLens - Developing Application and Deploying on Device, takes you through integration scenario identified in the preceding chapter, shows you how to implement and develop the holographic application, and explain how to integrate it with IoT system hosted on Azure. You will learn how to pull the data from the IoT system and render 3D assets on the holographic application based on the data. In this chapter, you will build interaction model scripts, test them in Unity and emulators, and finally deploy them on a device and test your first integrated application.
Chapter 7, Build End-to-End Retail Solution - Scenario Identification and Sketching, will teach you how to identify and design end-to-end retail solutions. You will sketch a solution to design your home appliances with virtual holograms. Then, you will place an order directly from the holographic application.
Chapter 8, Build End-to-End Retail Scenario - Developing Application and Deploying on Device, will take the retail scenario identified in the preceding chapter and implement and develop its holographic application. It will integrate it with backend services hosted on Azure. You will learn how to pull the data from the backend system, render 3D Assets on the fly, and place an order within holographic application. Here, you will build interaction model scripts, test them in Unity and emulators, and finally deploy them on a device and test your end-to-end retail application.
Chapter 9, Possibilities, outlines the new possibilities and scenarios with mixed reality and HoloLens.
Chapter 10, Microsoft HoloLens in Enterprise, looks at managing enterprise devices, such as adding them onto enterprise domains, managing updates, application distribution, and other device management features, such as managing devices through Microsoft Intune.
The basic requirements for this book are as follows:
Unity3D - 5.5 or higher
Visual Studio 2015 (Update 3) or higher
HoloLens emulator
Optional requirements:
Autodesk Maya
Adobe Photoshop
HoloLens device
Refer to Chapter 03, Explore HoloLens as Hologram - Scenario Identification and Sketching, for detailed information on the development environment setup.
This book is intended to help experienced developers and designers start working on mixed reality development using Microsoft HoloLens. This book is also useful for entrepreneurs who are planning to build a team of the following:
Developers/professionals
Designers
Entrepreneurs who are planning to build a team for mixed reality development
This book uses C# and Unity3D Scripting (C#) in the examples, so you need to know the basics of C# and scripting within Unity3D. You should be familiar with Visual Studio IDE and Unity Editor as well. You don't have to know anything about the HoloLens as a device.
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/HoloLens-Blueprints.We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/HoloLensBlueprints_ColorImages.pdf.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at [email protected] with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Welcome to the world of Digital Reality. The purpose of Digital Reality is to bring immersive experiences, such as taking or transporting you to different world or places, make you interact within those immersive, mix digital experiences with reality, and ultimately open new horizons to make you more productive. Applications of Digital Reality are advancing day by day; some of them are in the field of gaming, education, defense, tourism, aerospace, corporate productivity, enterprise applications, and so on.
The spectrum and scenarios of Digital Reality are huge. In order to understand them better, they are broken down into three different categories:
Virtual Reality
(
VR
): It is where you are disconnected from the real world and experience the virtual world. Devices available on the market for VR are Oculus Rift, Google VR, and so on. VR is the common abbreviation of Virtual Reality.
Augmented Reality
(
AR
): It is where digital data is overlaid over the real world. Pokemon GO, one of the very famous games, is an example of this globally. A device available on the market, which falls under this category, is Google Glass.Augmented Reality is abbreviated to AR.
Mixed Reality
(
MR