34,79 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
- Veröffentlichungsjahr: 2017
Are you a developer who is fascinated with Microsoft HoloLens and its capabilities? Do you want to learn the intricacies of working with the HoloLens SDK and create your own apps? If so, this is the book for you.
This book introduces and demystifies the HoloLens platform and introduces new ways you can interact with computers (Mixed Reality). It will teach you the important concepts, get you excited about the possibilities, and give you the tools to continue exploring and experimenting. You will go through the journey of creating four independent examples throughout the book, two using DirectX and two using Unity. You will learn to implement spatial mapping and gesture control, incorporate spatial sound, and work with different types of input and gaze. You will also learn to use the Unity 5 SDK for HoloLens and create apps with it.
Collectively, the apps explore the major concepts of HoloLens, but each app is independent, giving you the flexibility to choose where to start (and end).
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 403
Ähnliche
BIRMINGHAM - MUMBAI
Microsoft HoloLens By Example
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: August 2017
Production reference: 1290817
ISBN 978-1-78712-626-8
www.packtpub.com
Credits
Author
Joshua Newnham
Copy Editor
Shaila Kusanale
Reviewers
Vangos Pterneas
Jason M. Odom
Project Coordinator
Ritika Manoj
Commissioning Editor
Amarabha Banerjee
Proofreader
Safis Editing
Acquisition Editor
Reshma Raman
Indexer
Tejal Daruwale Soni
ContentDevelopmentEditor
Aditi Gour
Graphics
Jason Monteiro
Technical Editor
Lydia Vazhayil
Production Coordinator
Shantanu Zagade
About the Author
Joshua Newnham is a lead design technologist at Method, an experience design studio, with a focus on the intersection of design and artificial intelligence, working on a variety of projects ranging from wearables to assist medical staff during a surgery to a health assistant chatbot. Prior to this, Joshua was a technical director at Masters of Pie, a digital production studio focused on designing and developing virtual reality (VR) and augmented reality (AR) experiences, and premonitory developing tools for engineers and creatives.
Other roles before this included head of mobile for the digital agency--Razorfish--and cofounder of one of London’s first mobile production studios--ubinow--acquired by Havas Workclub, where he lead many projects for major global brands such as Universal Pictures, Diageo, and adidas.
Joshua’s interest lies in designing and building digital solutions that make our lives a little easier and more enjoyable, exploring new ways in which we can interact with computers, and designing and developing systems that are more intuitive, intelligent and social, and thus better integrated into our daily lives.
About the Reviewer
Vangos Pterneas helps innovative companies increase their revenue using motion technology and virtual reality. He is an expert in Kinect, HoloLens, Oculus Rift, and HTC Vive.
Microsoft has awarded him with the title of Most Valuable Professional for his technical contributions to the open source community. Vangos runs LightBuzz Inc, collaborating with clients from all over the world. He’s also the author of Getting Started with HTML5 WebSocket Programming and The Dark Art of Freelancing.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787126269
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
Enhancing Reality
A brief look at the past
Defining mixed reality
Designing for mixed reality
Identifying the type of experience you want to create
Understanding the environment
Thinking in terms of the real world
Interacting in mixed reality
Summary
Tagging the World Using DirectX
Creating a mixed reality app
Project setup
Setting up Azure
Creating a group
The project
Adding the Face API key and App Capabilities
Grabbing frames
Capturing frames
Rendering holographic labels
Positioning our label
Summary
Assistant Item Finder Using DirectX
Project setup
Adding the Computer Vision API service
The project
Leaving breadcrumbs
Node
Edges
Entity
Leaving a trail
Seeing the environment
FrameGrabber
Capturing frames
FrameAnalyzer
Integrating the FrameAnalyzer
Constructing a path
Getting assistance
Summary
Building Paper Trash Ball in Unity
Configuring a Unity project for HoloLens
Creating our first Mixed Reality scene
Mapping our environment
Spatial mapping API
Spatial mapping components
Placing our bin
Managing state
Implementing gaze
Finding a "good" place for the bin
Placing the bin
Nothing but net - throwing paper
Summary
Building Paper Trash Ball Using Holotoolkit in Unity
Adding HoloToolkit to your project
Keeping the user informed
Mapping our environment
Procedurally placing our bin
SurfaceMeshesToPlanes
Placing holograms
Spatial understanding
User placement versus procedural placement
Nothing but net - throwing paper
Summary
Interacting with Holograms Using Unity
Project setup
Interacting with gestures
Voice
Keyword recognizer
Grammar Recognizer using Speech Recognition Grammar Specification (SRGS)
Summary
Collaboration with HoloLens Using Unity
Project setup
Installing and using the Blender plugin
Unity starter project
The project
Connecting to BlenderLIVE
Understanding and using the real world
Shared space
Assisted discovery with spatial sound
Summary
Developing a Multiplayer Game Using Unity
Multiplayer networking
The project
Project setup
Discovery
Communication
Shared space
Coordinating
Summary
Deploying Your Apps to the HoloLens Device and Emulator
Rapid feedback with Unity's Tooling
Holographic Simulation
Holographic Remoting
Exporting from Unity to Visual Studio
HoloLens emulator
Deploying to the emulator
Deploying to the device
Windows Device Portal
Summary
Preface
This book is about developing mixed reality applications for the HoloLens platform, specifically focusing on core topics unique to mixed reality applications and diving deep into areas such as hologram placement, modes of interaction, and sharing experiences across devices; these are delivered through a series of examples, each focusing on illustrating one or more of these core topics. This focus on core topics differentiates this book from others, along with illustrating one of these through non-trivial and, hopefully, inspiring examples.
After completing this book, you will have a good grasp of the HoloLens platform and, more importantly, be inspired and motivated to continue your journey of exploring and experimenting with envisioning new ways in which we can interact with our computers with platforms such as Microsoft HoloLens.
What this book covers
Chapter 1, Enhancing Reality, introduces readers to the concepts of mixed reality, contrasting it against augmented reality (AR) and virtual reality (VR), including a discussion about the opportunities it offers in creating more natural and immersive experiences.
Chapter 2, Tagging the World Using DirectX, begins walking through setting up the environment, and then walks the user through building an augmented reality application that leverages the Microsoft Cognitive Services Computer Vision API to tag recognized content within the users' field of view, specifically, recognizing faces.
Chapter 3, Assistant Item Finder Using DirectX, that explains how to build on top of the previous example by allowing the user to pin tagged items and have the system assist the user navigate back to them.
Chapter 4, Building Paper Trash Ball in Unity, walks the reader through the development of a popular smartphone game--Paper Toss--but unlike the traditional game, the setting will be the user’s physical environment.
Chapter 5, Building Paper Trash Ball Using HoloToolkit in Unity, is a continuation of the last chapter, but looks at reimplementing the project using HoloToolkit and a collection of scripts and components intended to accelerate development of HoloLens application development, and extends it by adding an holographic user interface.
Chapter 6 , Interacting with Holograms Using Unity, explores various ways the user can interact with holograms. We start by discussing gestures, along with allowing the user to touch holograms, then introduce the reader to Voice User Interfaces, walking the reader through extending the application to allow the user to manipulate the hologram using their voice.
Chapter 7, Collaboration with HoloLens Using Unity, outlines building a project where HoloLens is shared across multiple devices. The example is based on a collaborative design review, whereby a design (a 3D model) is streamed to multiple HoloLens devices where it is projected into the real world, allowing for richer engagement and collaboration.
Chapter 8, Developing a Multiplayer Game Using Unity, covers our final example of building a fun multiplayer game inspired by the classic Cannon Fodder, where the user will navigate their team around the environment to eliminate their opponent.
Chapter 9, Deploying Your Apps to the HoloLens Device and Emulator, showcases the process of building and deploying applications to an actual HoloLens device and/or emulator.
What you need for this book
Here is a list of software used in this book:
Window 10 Pro
Visual Studio 2015 Update 3
HoloLens Emulator (build 10.0.14393.0)
Unity 5.5.0f3
Holotoolkit-Unity (SHA1 cd2fd2f9569552c37ab9defba108e7b7d9999b12)
Vuforia 6.1
Blender 2.78
Who this book is for
This book is for developers who have some experience with programming in any of the major languages, such as C# and C++. You do not need any knowledge of augmented reality development.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply email [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files emailed directly to you. You can download the code files by following these steps:
Log in or register to our website using your email address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Microsoft-HoloLens-By-Example. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Downloading the color images of this book
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/MicrosoftHoloLensByExample_ColorImages.pdf.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Enhancing Reality
Welcome to Microsoft HoloLens By Example; join me on a journey through this book to learn about Microsoft's HoloLens device and more generally, mixed reality (MR) applications, through a series of exciting example, each uncovering an important concept related to developing mixed reality applications, including:
Understanding the environment and context of the user through image recognition and spatial mapping
Projecting and placing holograms into the real world
Allowing the user to interact with holograms using a variety interaction modes including gaze, gesture and voice
Sharing the experience across devices
We will start our journey by briefly peering into the past to see how our relationship with computers has changed over the past couple of decades, before defining MR with respect to reality and MR. We will then cover some of the core building blocks of MR applications before wrapping up this chapter.
A good place to start is always at the beginning, so let's start there.
A brief look at the past
Let's start our journey by briefly peering into the past, specifically at how our interaction with computers has changed over time and how it might be in the near future.
Early computers, bypassing when people were used as computers, were large mechanical machines, and their size and cost meant that they were fixed to a single location and limited to a specific task. The limitation of a single function was soon resolved with Electronic Numerical Integrator And Computer (ENIAC), one of the world's first general-purpose electronic computers. Due to its size and cost, it was still fixed to a single location and interacted with/programmed through a complex process of rearranging physical switches on a switch board. Mainframes followed and introduced a new form of interaction, the Command Line Interface (CLI). The user would interact with the computer by issuing a series of instructions and have the response returned to them via a Terminal screen. Once again, these computers were expensive, large, and complex to use.
The following screenshot is an example of DOS, an operating system that dominated the personal computing market in the late 1980s:
The era of direct manipulation interfaces and personal computing followed. With the reduced size, cost, and introduction of the Graphical User Interface (GUI), computers had finally become more accessible to the ordinary person. During this era, the internet was born, computers became platforms allowing people to connect and collaborate with one another, to be entertained, and to augment their skills by making data and information more accessible. However, these computers were, despite their name, still far from being personal, neglecting the user and their current context. These computers forced the user to work in a digital representation of their world, fixing them to a single location--the desk.
The following photograph shows the Apple Macintosh, seen as one of the innovators of the GUIs:
BlackBerry Limited (then known as Research In Motion), released its first smartphone in around 2000, which edged us toward the mobile computer. The tipping point was in 2007, when Steve Jobs revealed the iPhone. Smartphones became ubiquitous; personal computing had finally arrived, and it was a platform that was inherently personal. Constraints of technology provided the catalyst to rethink what the role of the application was. Since then, we have been incrementally improving on this paradigm with more capable devices and smarter services. For a lot of us, our smartphone is our primary device-being always-on and always-connected has given us superpowers our ancestors had only dreamed about.
The following is a photograph showing the late Steve Jobs presenting the first version of the iPhone to the world:
Taking a bird's-eye view, we can see the general (and obvious) trend of the following things:
Minimization of hardware
Increase in utility
Moving toward more natural ways of interacting
The shift from us being in the computer world toward computers being in our world
So, what does the next paradigm look like? I believe that HoloLens gives us a glimpse into what computers will look like in the near future, and even in its infancy, I believe that it provides us with a platform to start exploring and defining what the future will look like. Let's continue by clarifying the differences and similarities of virtual reality (VR), augmented reality (AR), and MR.
The following is a photograph illustrating an example of how MR seamlessly blends holograms into the real world:
Defining mixed reality
Let's begin by first defining and contrasting three similar, but frequently misused, paradigms: VR, AR, and MR:
VR describes technology and experiences where the user is fully immersed in a virtual environment.
AR can be described as technology and techniques used to superimpose digital content onto the real world.
MR can be considered as a blend of VR and AR. It uses the physical environment to add realism to holograms, which may or may not have any physical reference point (as AR does).
The differences between VR, AR, and MR are not so much in the technology but in the experience you are trying to create. Let's illustrate it through an example--imagine that you were given a brief to help manage children's anxiety when requiring hospital treatment.
With VR, you might create a story or game in space, with likeable characters that represent the staff of the hospital in role and character, but in a more interesting and engaging form. This experience will gently introduce the patient to the concepts, procedures, and routines required. On the other end of the spectrum, you can use augmented reality to deliver fun facts based on contextual triggers, for example, the child might glance (with their phone or glasses) at the medicine packaging to discover what famous people had their condition. MR, as the name suggests, mixes both the approaches--our solution can involve a friend, such as a teddy, for the child to converse with, expressing their concerns and fears. Their friend will accompany them at home and in the hospital, being contextually sensitive and respond appropriately.
As highlighted through these hypothetical examples, they are not mutually exclusive but adjust the degree to which they preserve reality; this spectrum is termed as the reality–virtuality continuum, coined by Paul Milgram in his paper Augmented Reality: A class of displays on the reality-virtuality continuum, 2007. He illustrated it graphically, showing the spectrum of reality between the extremes of real and virtual, showing how MR encompasses both AR and augmentedvirtuality (AV). The following is a figure by Paul Milgram and Fumio Kishino that defined the concept of Milgram’s reality-virtuality continuum and illustrates the concept--to the far left, you have reality and at the opposite end, the far right, you have virtuality--as MR strides itself between these two paradigms:
Our focus in this book, and the proposition of HoloLens, is MR (also referred to as Holographic). Next, we will look into the principles and building blocks that make up MR experiences.
Designing for mixed reality
The emergence of new technologies is always faced with the question of doing old things in a new way or doing new things in new ways. When the TV was first introduced, the early programs were adopted from radio, where the presenter read in front of a camera, neglecting the visual element of the medium. A similar phenomenon happened with computers, the web, and mobile--I would encourage you to think about the purpose of what you're trying to achieve rather than the process of how it is currently achieved to free you to create new and innovative solutions.
In this section, we will go over some basic design principles related to building MR experiences and the accompanying building blocks available on HoloLens. Keeping in mind that this medium is still in its infancy, the following principles are still a work in progress.
Identifying the type of experience you want to create
As discussed earlier, the degree of reality you want to preserve is up to you--the application designer. It is important to establish where your experience fits early on as it will impact how you design and also the implementation of the experience you are building. Microsoft outlines three types of experiences:
Enhanced environment apps: These are applications that respect the real world and supplement it with holographic content. An example of this can be pinning a weather poster near the front door, ensuring that you don't forget your umbrella when the forecast is for rain.
Blended environment apps: These applications are aware of the environment, but will replace parts of it with virtual content. An application that lets the user replace fittings and furniture is an example.
Virtual environment apps: These types of applications will disregard the environment and replace it completely with a virtual alternative. An application that converts your room into a jungle, with trees and bushes replacing the walls and the floor can be taken as an example.
Like with so many things, there is no right answer, just a good answer for a specific user, specific context, and at a specific time. For example, designing a weather app for a professional might have the weather forecast pinned to the door so that she sees it just before leaving for work, while it might be more useful to present the information through a holographic rain cloud, for example, to a younger audience.
In the next section, we will continue our discussion on the concepts of MR, specifically looking at how HoloLens makes sense of the environment.
Understanding the environment
One of the most compelling features of HoloLens is its ability to place and track virtual/digital content in the real world. It does this using a process known as spatial mapping, whereby the device actively scans the environment, building its digital representation in memory. In addition, it adds anchors using a concept called spatial anchors. Spatial anchors mark important points in the world in reference to the defined world origin; holograms are positioned relative to these spatial anchors, and these anchors are also used to join multiple spaces for handling larger environments.
The effectiveness of the scanning process will determine the quality of the experience; therefore, it is important to understand this process in order to create an experience that effectively captures sufficient data about the environment. One technique commonly used is digital painting; during this phase, the user is asked to paint the environment. As the user glances around, the scanned surfaces are visualized (or painted over), providing feedback to the user that the surface has been scanned.
However, scanning and capturing the environment is just one part of understanding the environment, and the second is making use of it; some uses include the following:
Occlusion
: One of the shortfalls of creating immersive MR experiences using single camera devices (such as Smartphones) is the inability to understand the surface to occlude virtual content from the real world when obstructed. Seeing holograms through objects is a quick way to force the user out of the illusion; with HoloLens, occluding holograms with the real world is easy.
Visualization
: Sometimes, visualizing the scanned surfaces is desirable, normally an internal effect such as feeding back what part of the environment is scanned to the user.
Placement
: Similar to occlusion in that it creates a compelling illusion, holograms should behave like the real objects that they are impersonating. Once the environment is scanned, further processing can be performed to gain greater knowledge of the environment, such as the types of surfaces available. With this knowledge, we can better infer where objects belong and how they should behave. In addition to creating more compelling illusions, matching the experience with the user's mental model of where things belong makes the experience more familiar, thus easing adoption by making it easier and more intuitive to use.
Physics
: HoloLens makes the scanned surfaces accessible as plain geometry data, which means we can leverage the existing physics simulation software to reinforce the presence of holograms in the user's environment. For example, if I throw a virtual ball, I expect it to bounce off the walls and onto the floor before settling down.
Navigation
: In game development, we have devised effective methods for path planning. Having a digital representation of our real world affords us to utilize these same techniques in the real world. Imagine offering a visually impaired person an opportunity to effectively navigate an environment independently or assisting a parent to find their lost child in a busy store.
Recognition
: Recognition refers to the ability of the computer to classify what objects are in the environment; this can be used to create a more immersive experience, such as having virtual characters sit on seats, or to provide a utility, such as helping teach a new language or assisting visually impaired people so that they can better understand their environment.
Thinking in terms of the real world
The luxury of designing for screen-based experiences is that your problem is simplified. In most cases, we own the screen and have a good understanding of it; we lose these luxuries with MR experiences, but gain more in terms of flexibility and therefore opportunity for new, innovative experiences. So it becomes even more important to understand your users and in what context they will be using your application, such as the following:
Will they be sitting or standing?
Will they be moving or stationary?
Is the experience time dependent?
Some common practices when embedding holograms in the real world include the following:
Place holograms in convenient places--places that are intuitive, easily discovered, and in reach, especially if they are interactive.
Design for the constraints of the platform, but keep in mind that we are developing for a platform that will rapidly advance in the next few years. At the time of writing, Microsoft recommends placing holograms between 1.25 meters and 5 meters away from the device, with the optimum viewing distance of 2 meters. Find ways of gracefully fading content in and out when it gets too close or far, so as not to jar the user into an unexpected experience.
As mentioned earlier, placing holograms on contextually relevant surfaces and using shadows create, more immersive experiences, giving a better illusion that the hologram exists in the real world.
Avoid locking content to the camera; this can quickly become an annoyance to the user. Rather, use an alternative that is more gentle, an approach being adopted has the interface dragged, in an elastic-like manner, with the user's gaze.
Make use of
spatial sound
to improve immersion and assist in hologram discovery. If you have ever listened to Virtual Barber Shop Hair Cut (
https://www.youtube.com/watch?v=8IXm6SuUigI
), you will appreciate how effective 3D sound can be in creating an immersive experience and, similar to mimicking the behavior of the objects you are trying to impersonate, use real world sound that the user will expect from the hologram.
Of course, this list is not comprehensive, but has a few practices to consider when building MR applications. Next, we will look at ways in which the user can interact with holograms.
Interacting in mixed reality
With the introduction of any new computing paradigm comes new ways of interacting with it and, as highlighted in the opening paragraph, history has shown that we are moving from an interface that is natural to the computer toward an interface that is more natural to people. For the most part, HoloLens removes dedicated input devices and relies on inferred intent, gestures, and voice. I would argue that this constraint is the second most compelling offering that HoloLens gives us; it is an opportunity to invent more natural and seamless experiences that can be accessible to everyone. Microsoft refers to three main forms of input, including GazeGestureVoice (GGV); let's examine each of these in turn.
Gaze refers to tracking what the user is looking at; from this, we can infer their interest (and intent). For example, I will normally look at a person before I speak to them, hopefully, signalling that I have something to say to them. Similarly, during the conversation, I may gaze at an object, signalling to the other person that the object that I'm gazing at is the subject I'm speaking about.
This concept is heavily used in HoloLens applications for selecting and interacting with holograms. Gaze is accompanied with a cursor; the cursor provides a visual representation of the users gaze, providing visual feedback to what the user is looking at. It can additionally be used to show the state of the application or object the user is currently gazing at, for example, the cursor can visually change to signal whether the hologram the user is gazing at is interactive or not. On the official developer site, Microsoft has listed the design principles; I have paraphrased and listed them here for convenience:
Always present: The cursor is, in some sense, akin to the mouse pointer of a GUI; it helps the users understand the environment and the current state of the application.
Cursor scale: As the cursor is used for selecting and interacting with holograms, it's size should be no bigger than the objects the user can interact with. Scale can also be used to assist the users' understanding of depth, for example, the cursor will be larger when on nearby surfaces than when on surfaces farther away.
Look and feel: Using a directionless shape means that you avoid implying any specific direction with the cursor; the shape commonly used is a donut or torus. Making the cursor hug the surfaces gives the user a sense that the system is aware of their surroundings.
Visual cues: As mentioned earlier, the cursor is a great way of communicating to the user about what is important as well as relaying the current state of the application. In addition to signalling to the user what is interactive and what is not, it also can be used to present additional information (possible actions) or the current state, such as visualizing showing the user that their hand has been detected.
While gazing provides the mechanism for targeting objects, gestures and voice provide the means to interact with them. Gestures can be either discrete or continuous. The discrete gestures execute a specific action, for example, the air-tap gesture is equivalent to a double-click on a mouse or tap on the screen. In contrast, continuous gestures are entered and exited and while active, they will provide continuous update to their state. An example of this is the manipulation gesture, whereby the user enters the gesture by holding their finger down (called the hold gesture); once active, this will continuously provide updates of the position of the tracked hand until the gesture is exited with the finger being lifted. This is equivalent to dragging items on desktop and touch devices with the addition of depth.
HoloLens recognizes and tracks hands in either the ready state (back of hand facing you with the index finger up) or pressed state (back of hand facing you with the index finger down) and makes the current position and state of the currently tracked hands available, allowing you to devise your own gestures in addition of providing some standard gestures, some of which are reserved for the operating system. The following gestures are available:
Air-tap: This is when the user presses (finger down) and releases (finger up), and is performed within a certain threshold. This interaction is commonly associated to selecting holograms (as mentioned earlier).
Bloom: Reversed for the operating system, bloom is performed by holding your hand in front of you with your fingers closed, and then opening your hand up. When detected, HoloLens will redirect the user to the Start menu.
Manipulation: As mentioned earlier, manipulation is a continuous gesture entered when the user presses their finger down and holds it down, and exited when hand tracking is lost or the user releases their finger. When active, the user's hand is tracked with the intention of using the absolute position to manipulate the targeted hologram.
Navigation: This is similar to the manipulation gesture, except for its intended use. Instead of mapping the absolute position changes of the user's hand with the hologram, as with manipulation, navigation provides a standard range of -1 to 1 on each axis (x, y, and z); this is useful (and often used) when interacting with user interfaces, such as scrolling or panning.
The last dominate form of interacting with HoloLens, and one I'm particularly excited about, is voice. In the recent times, we have seen the rise of Conversational User Interface (CUI); so, it's timely to introduce a platform where one of it's dominate inputs is voice. In addition to being a vision we have had since before the advent of computers, it also provides the following benefits:
Hands free (obviously important for a device like HoloLens)
More efficient and requires less effort to achieve a task; this is true for data entry and navigating deeply nested menus
Reduces cognitive load; when done well, it should be intuitive and natural, with minimal learning required
However, how voice is used is really dependent on your application; it can simply be used to supplement gestures such as allowing the user to use the Select keyword (a reserved keyword) to select the object the user is currently gazing at or support complex requests by the user, such as answering free-form questions from the user. Voice also has some weaknesses, including these:
Difficulty with handling ambiguity in language; for example, how do you handle the request of
louder
Manipulating things in physical space is also cumbersome
Social acceptance and privacy are also considerations that need to be taken into account
With the success of Machine Learning (ML) and adoption of services such as Amazon's Echo, it is likely that these weaknesses will be short lived.
Summary
So far, we have discussed a lot of high-level concepts, let's now wrap this chapter up before moving on and putting these concepts into practice through a series of examples.
I hope you're as excited as I am and, with this book, join me in shaping the future. This book consists of a series of examples, each walking through a "toy" example used to demonstrate a specific concept or feature of the HoloLens. As you work your way through this book, I encourage you to dream of what is possible, looking past some of the current nuances, knowing that they will be resolved in the near future. I would also discourage creating horseless carriages; a phrase used by the notable designer Don Norman in reference to how the car was designed (and named) on horse-drawn carriages, highlighting how new technology is always started by making it look like the old technology. So, rather than adapting from the existing apps, be inspired to adapt from the real world--with that said, let's make a start with our first example.
Tagging the World Using DirectX
One of the most significant advancements of HoloLens has little to do with the actual technology and more to do with where it is placed. Having a computer situated conveniently on your head lends itself to many interesting avenues. It means having a computer easily accessible any place and in any situation and, equally important, having a computer that is capable of seeing what you see and hearing what you hear. Smartphones have taken us to a paradigm where we have had always-on and always-connected computers, but this extends this concept even further, whereby our devices can be more aware of our situation, but of course this is only possible if the applications we build for these platforms take advantage of these opportunities. In this chapter, we start exploring this concept by way of example, whereby we will use Microsoft's Cognitive Services (Microsoft's Cognitive Services), to recognize people in the user's view and, if recognized, display associated metadata--name in this case. While reading through this chapter, I encourage you to let your mind drift and imagine how else augmentation can be applied to enhance people's lives.
While working through this chapter (and example), we will touch on the following topics:
A brief walkthrough of the Visual C# Holographic DirectX 11 App template provided by Microsoft
Capturing frames from the HoloLens color camera
Using Microsoft's Cognitive Services to detect and identify faces
Positioning a name tag relative to a recognized face in the users view
