Augmented Reality Development with Unity E-Book

I. Wijesooriya

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 371

Veröffentlichungsjahr: 2025

Beschreibung

This book provides a comprehensive guide to developing augmented reality applications. Starting with an introduction to AR concepts, it explores key tools and platforms, emphasizing Unity as a core development tool. Readers will gain hands-on experience creating marker-based and marker-less AR apps using Vuforia, ARKit, and ARCore.
The book takes a practical approach, covering the creation of custom components, C# programming for Unity, and building mobile AR apps. Advanced topics include world-scale AR with Niantic Lightship and optimization strategies for AR app performance. Each chapter builds skills progressively, ensuring a thorough understanding of the subject.
By the end, readers will have the confidence and skills to design and develop innovative AR applications, understanding the best practices and challenges of AR app development.

Leseprobe

AUGMENTED REALITYDEVELOPMENT WITH UNITY

LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY

By purchasing or using this book and companion files (the “Work”), you agree that this license grants permission to use the contents contained herein, including the disc, but does not give you the right of ownership to any of the textual content in the book / disc or ownership to any of the information or products contained in it. This license does not permit uploading of theWork onto the Internet or on a network (of any kind) without the written consent of the Publisher. Duplication or dissemination of any text, code, simulations, images, etc. contained herein is limited to and subject to licensing terms for the respective products, and permission must be obtained from the Publisher or the owner of the content, etc., in order to reproduce or network any portion of the textual material (in any media) that is contained in the Work.

MERCURY LEARNING AND INFORMATION (“MLI” or “the Publisher”) and anyone involved in the creation, writing, or production of the companion disc, accompanying algorithms, code, or computer programs (“the software”), and any accompanying Web site or software of the Work, cannot and do not warrant the performance or results that might be obtained by using the contents of the Work. The author, developers, and the Publisher have used their best efforts to ensure the accuracy and functionality of the textual material and/or programs contained in this package; we, however, make no warranty of any kind, express or implied, regarding the performance of these contents or programs. The Work is sold “as is” without warranty (except for defective materials used in manufacturing the book or due to faulty workmanship).

The author, developers, and the publisher of any accompanying content, and anyone involved in the composition, production, and manufacturing of this work will not be liable for damages of any kind arising out of the use of (or the inability to use) the algorithms, source code, computer programs, or textual material contained in this publication. This includes, but is not limited to, loss of revenue or profit, or other incidental, physical, or consequential damages arising out of the use of this Work.

The sole remedy in the event of a claim of any kind is expressly limited to replacement of the book and/or files, and only at the discretion of the Publisher. The use of “implied warranty” and certain “exclusions” varies from state to state and might not apply to the purchaser of this product.

Companion files are available for downloading by writing to the publisher (with proof of purchase) [email protected].

AUGMENTED REALITYDEVELOPMENT WITH UNITY

INDIKA WIJESOORIYA

MERCURY LEARNING AND INFORMATION

Boston, Massachusetts

Copyright © 2024 by MERCURY LEARNING AND INFORMATION.

An Imprint of DeGruyter Inc. All rights reserved. Reprinted and revised with permission.

Original title and copyright: Augmented Reality Development with Unity.

Copyright © 2024 by BPB Publications. All rights reserved. ISBN : 978-93-55518-330.

This publication, portions of it, or any accompanying software may not be reproduced in any way, stored in a retrieval system of any type, or transmitted by any means, media, electronic display, or mechanical display, including, but not limited to, photocopy, recording, Internet postings, or scanning, without prior permission in writing from the publisher.

Publisher: David Pallai

MERCURY LEARNING AND INFORMATION

121 High Street, 3rd Floor

Boston, MA 02110

[email protected]

www.merclearning.com

800-232-0223

I. Wijesooriya. Augmented Reality Development with Unity.

ISBN: 978-1-50152-258-1

The publisher recognizes and respects all marks used by companies, manufacturers, and developers as a means to distinguish their products. All brand names and product names mentioned in this book are trademarks or service marks of their respective companies. Any omission or misuse (of any kind) of service marks or trademarks, etc. is not an attempt to infringe on the property of others.

Library of Congress Control Number: 2024938179

242526321 This book is printed on acid-free paper in the United States of America.

Our titles are available for adoption, license, or bulk purchase by institutions, corporations, etc. For additional information, please contact the Customer Service Dept. at 800-232-0223(toll free).

All of our titles are available in digital format at academiccourseware.com and other digital vendors. Companion files are available for downloading (with proof of purchase) by writing to the publisher [email protected]. The sole obligation of MERCURY LEARNING AND INFORMATION to the purchaser is to replace the files, based on defective materials or faulty workmanship, but not based on the operation or functionality of the product.

ToMy beloved wife, Thilini&My mother, Swarna

Contents

Preface

Acknowledgments

Chapter 1: Getting Started with Augmented Reality

Introduction

Structure

Objectives

AR Implementation and Application

History of AR

1968—The Sword of Damocles

1975—Videoplace by Myron Krueger

1999—AR Toolkit

AR Over the Years

AR Across the World

AR in Entertainment

AR in Manufacturing and Logistics

Real Estate

Health

Education

Marketing and Retail

AR Enabling Technologies

Image Recognition and Tracking

Simultaneous Localization and Mapping

Machine Learning and Artificial Intelligence

Conclusion

Key Points

Questions

Chapter 2: Visualizing the AR Environment and Components

Introduction

Structure

Objectives

Basics of Mathematics for 3D Visualization

Understanding the 3D Environment

Vector Mathematics

Terminology of Vectors

Basic Trigonometry

Understanding Different Spaces

Context Recognition Concepts

Placement of Virtual Objects on Physical Worlds

Converting Physical Input to Virtual Interactions

Conclusion

Key Points

Questions

Chapter 3: Exploring Tools and Development Platforms

Introduction

Structure

Objectives

AR Platforms

Smartphones Android, iOS

Magic Leap

Snapchat Spectacles

Other Wearables

WebXR AR

Social AR

AR Development Tools

Native

Social AR—Spark AR, Snapchat Lens Studio, TikTok

Apple Reality Composer

8th Wall

Unity 3D

AR SDKs

Microsoft Mixed Reality Toolkit

ARToolkit

Wikitude

Easy AR

Vuforia

ARCore

ARKit

AR Foundation

Niantic Lightship

Conclusion

Key Points

Questions

Chapter 4: Up and Running with Unity 3D

Introduction

Structure

Objectives

Installing Unity 3D

Installing Android SDK, Java SDK

Installing Xcode in MacOS

Installing Visual Studio/Visual Studio Code

Creating a Project and Setting up the Target Platform

Unity Engine User Interface

Using the Tools to Create and Manipulate Primitive Objects

Understanding Lights

Creating Materials

Manipulating the Virtual Camera

Using the Unity Package Manager

Importing Content into Unity

Importing Assets from the Unity Asset Store

The Structure of the Game Object

Components of Unity

Transform

Mesh Renderer

Colliders

Rigid Body

Creating Prefabs

Conclusion

Key Points

Questions

Chapter 5: Creating a First Custom Component

Introduction

Structure

Objectives

Introduction to C# in Unity

Exploring the MonoBehavior

Serialized Variables

Debug Class of Unity

Time in Unity

GetComponent

Building a Simple 3D Clock

Conclusion

Key Points

Questions

Chapter 6: Refreshing C# Concepts with Unity

Introduction

Structure

Objectives

Setting up the Scene

Movement of the Player

Creating a Bullet for the Player to Shoot

Shooting Bullets

Setting Up the Enemy

Setting Up the Player

Spawning and Destroying Enemies

Game UI

Game Manager

Adding Audio and Polishing

Adding the Main Menu

Switching Scenes

The Next Steps

Conclusion

Key Points

Questions

Chapter 7: Trying Out a First 3D Mobile App Development

Introduction

Structure

Objectives

Unity Build Settings

Android Build Settings

Icon

Resolution and Presentation

Splash Image

Other Settings

Publishing Settings

iOS Build Settings

Identification | Bundle Identifier

Identification | Provisioning Profile and Signing Team ID

Configuration | Target Device, SDK and Minimum iOS Version

Configuration | Camera Usage Description

Configuration | Requires ARKit Support

Building Unity Apps

Building for Android

Building for Android without ADB

Building for iOS

Unity Script Debugging with Visual Studio

Debugging the App for Both Platforms

Using the ADB Console for Android

Using the Xcode Console for iOS

Conclusion

Key Points

Questions

Chapter 8: Building Marker-based AR Apps with Vuforia

Introduction

Structure

Objectives

Introduction to Vuforia

Create a Vuforia Developer Account and Login

Downloading and Installing the Sdk

Setting up a Basic AR Scene

Using Image Targets

Choosing the Best Markers/Targets

Additional Features of Image Targets

Using Cylinder Targets

Using Multi Targets

Development of an AR Treasure Hunt Game

Conclusion

Key Points

Question

Chapter 9: Developing Marker-based Dynamic AR Apps

Introduction

Structure

Objectives

Machine Inspection Tool

Unity Serialization and Deserialization

Creating a Simple Mock API

Using Unity WebRequest to Request Data Based on Input

Vuforia VuMark Introduction

Designing a VuMark Using Adobe Illustrator

Connecting the Marker Recognition with Networking to Request Dynamic Data and Displaying Them in AR

Dynamic AR Billboard

Setting Up the AR Poster Project

Implementing Vuforia Cloud Targets

Creating Asset Bundles in Unity

Uploading the Asset Bundles in the Cloud

Updating the Meta Files to Update the AR Experience

Downloading Content and Displaying in AR

Conclusion

Key Points

Questions

Chapter 10: Marker-less AR Apps with ARKit and ARCore

Introduction

Structure

Objectives

Setting up AR Foundation

Setting up a Marker-less Tracking Scene

Simulating the Scene in the Editor

Plane Detection and Tracking: Building an AR Object Viewer

Placing the Object on a Scanned Plane

Move Object on Touch if the Object is Placed Already

Scaling the Object with Two Touches

Rotating the Object with a Single Touch and Drag

Face Tracking—Creating an AR Face Filter with AR Foundation

Default Face Tracking Setup

Adding Custom Objects Around a Face

Adding Face Interactions

Device Tracking: Building an AR Shooter

Setting Up the Enemy

Setting Up the Game Manager

Additional Components of AR Foundation

AR Human Body Manager

Point Clouds

Anchors

Image Tracking

Conclusion

Key Points

Question

Chapter 11: World-Scale AR App with Niantic Lightship

Introduction

Structure

Objectives

Setting up Niantic Lightship with Unity

Downloading Required Tools and Assets

Object Placement in AR

Object Occlusion

Environment Meshing

Setting Up the Scene to Mesh the Environment Boundaries

Creating a Simple Physics Simulation in AR

Introduction to Niantic Lightship VPS

Scanning an Environment for VPS

The Flow of VPS Scan Recognition

Importing the Scanned Location

Adding Anchors Using the VPS Authoring Tool

Updating the Prefab with Proper Visual Elements

Using Niantic API to Restore Anchor Data

Using Niantic VPS for Scalable AR Applications

Conclusion

Key Points

Question

Chapter 12: Best Practices in Augmented Reality Application Design

Introduction

Structure

Objectives

Safety is Key

Showing Warnings at the Beginning

Showing Warnings While Using the AR App

Guide the User with the Interactions

Be Mindful of the Scale

Use Audio and Visual Cues

Indicating Objects that are out of Camera View

Adding Visual Markers on Interactable Objects

Using 3D Audio Cues

Showing Physical Properties in Visual Cues

Free Flow Restrictions

AR Interactions

Using a Reticle to Interact with Smaller Objects

Interacting with Objects Further from the Camera

UI Considerations

Conclusion

Key Points

Question

Chapter 13: AR App Performance Optimization

Introduction

Structure

Objectives

Performance Metrics

Frames Per Second

Tris and Verts

Draw Calls

Optimization of a 3D App

GPU Instancing

Static and Dynamic Batching

Occlusion Culling and Frustum Culling

Textures and UV Maps

Light Baking

UI Optimizations

Optimization Considerations in Coding

Conclusion

Key Points

Questions

Appendix: Answer Key for Chapter Questions

Index

Preface

Augmented Reality (AR) has been a key component of immersive technologies in the past decade. With the entrance of web3 and related technologies, AR has become a key player in providing extended user experiences for people around the world. Whether it is education, training, manufacturing, healthcare or military, AR can help people augment and enhance the core components related to them. One of the major challenges for an enthusiast to begin development of AR applications is the knowledge of the many tools available in the current context.

This book will introduce AR, its user cases and history to better understand its power and utility. The second part of this book will focus on the available tools and technologies for developing AR applications. Later, the book will introduce the Unity 3D engine, fundamentals of C# programming with Unity, vector mathematics, implementing AR applications with Unity, followed by best practices in creating AR applications.

Using this book, readers will have a comprehensive knowledge for bringing an idea into an augmented reality application, and for using C# and the Unity 3D engine to design and develop a prototype, or a finished application.

Chapter1:GettingStartedwithAugmentedReality

In this chapter, we will learn about the key concepts of AR including the technology behind AR applications and the principles of content recognition. It is important to review the underlying technology to better understand AR’s behavior. It also provides a brief history of AR and the existing and future use cases of the technology.

Chapter 2: Visualizing the AR Environment and Its Components

In this chapter, we will discuss the placement of objects in a virtual world and how these are mapped in order to generate the illusion of an augmented reality. We show how various Augmented Reality setups perform and, to better understand these principles, basic mathematical theories are discussed.

Chapter 3: Exploring Tools and Development Platforms

This chapter provides an in-depth introduction to the tools and technologies available for building AR applications and experiences, along with their respective platforms. We compare the pros and cons of each tool and discuss various use cases to determine which platform and tools are best suited for each scenario.

Chapter 4: Up and Running with Unity 3D

In this chapter, we utilize the Unity3D engine as our primary development tool for the remainder of the book. We cover the steps for installing the necessary tools and setting up your computer for development. Additionally, we will build a 3D scene in Unity3D, providing an overview of the Unity engine, its tools, and an introduction to 3D concepts essential for creating AR applications in future chapters. We also delve into the key components of Unity, exploring the structure of GameObjects, their components, and the main elements within Unity

Chapter 5: Creating Your First Custom Component

Here, we introduce C# with Unity by creating our first custom component. We guide you through the process of creating a base script, understanding the basic components of a C# script, and how C# script execution works. By the end of the chapter, we will have created a simple clock using the concepts learned.

Chapter 6: Refreshing C# Concepts with Unity

In this chapter, we explore C# programming to build functionality within Unity. We will develop an endless top-down shooting game targeted at mobile devices, learning C# concepts, and utilizing key Unity components relevant to building interactive applications.

Chapter 7: Trying Out First 3D Mobile App Development

During this chapter, we continue developing the top-down shooter game and install the application on mobile devices. We cover the requirements for both Android and iOS platforms, detailing how to set up project settings accordingly. Additionally, we explore various debugging methods for running your app on a device.

Chapter 8: Building Marker-based AR Apps with Vuforia

This chapter focuses on building an Augmented Reality treasure hunt game using the Vuforia AR SDK. We introduce marker-based tracking with Vuforia, its key capabilities, and its use of marker-based tracking. We walk through creating a Vuforia account, registering the application, configuring project trackers via the Vuforia portal, downloading, and installing the SDK in Unity, and testing with a PC webcam.

Chapter 9: Developing Marker-based Dynamic AR Apps

In this chapter, we extend the use of marker-based AR to dynamic application design. We will build two AR applications: a sample AR machine inspection tool and a dynamic AR billboard that changes without app updates. For the inspection tool, we use a marker system to identify different objects without significantly altering the marker graphic. For the dynamic billboard, we host files in the cloud to create a dynamic AR app that can update content and markers without needing app updates.

Chapter 10: Marker-less AR Apps with ARKit and ARCore

We will build multiple mini-projects covering the AR detection modes provided by ARKit and ARCore, compatible with iPhones, iPads, and Android devices. We will learn the similarities between both platforms and use AR Foundation as a cross-platform method for building AR apps.

Chapter 11: World Scale AR App with Niantic Lightship

This chapter focuses on developing mini-projects using the Niantic Lightship SDK to create world-scale AR applications that interact with environmental obstacles, including collision and occlusion. The first project involves building a simple AR throwing experience, while the second project covers scanning an environment to create custom area-based AR applications.

Chapter 12: Best Practices in Augmented Reality Application Design

We discuss various best practices in AR application design and development, categorized into design-based practices, development practices, and provide sample project snippets illustrating these best practices.

Chapter 13: AR App Performance Optimization

In this chapter, we learn about key performance metrics and optimization techniques to enhance AR applications, reducing resource requirements. We identify essential actions to address performance-related challenges in your app.

Companion files with code samples and color figures from the book are available for downloading by writing to [email protected].

Acknowledgments

I would like to express my deepest gratitude to my wife, Thilini, whose unwavering encouragement and unconditional support have been instrumental in bringing this book to fruition. Her belief in me and constant motivation were an invaluable source of inspiration throughout this journey. I extend my heartfelt thanks to Chamira Jayasinghe, CEO of Arimac, for opening my eyes to the vast world of immersive technology. His guidance and mentorship have played a pivotal role in shaping my career and fueling my passion for AR development.

I would like to extend my appreciation to Lou, Mike, and Jerry from Circuit Stream. Their support and guidance have helped me discover my passion for teaching and sharing knowledge with others. Their expertise and collaboration have been instrumental in shaping the content of this book. I would also like to acknowledge Dejan, my dear friend, for his unwavering motivation and continuous support. His belief in my abilities has been a driving force throughout this writing process. I am grateful to Mana Saei and Leena Prasad for their insightful reviews and valuable comments on the manuscript. Their expertise and constructive feedback have immensely contributed to the quality of this book.

Finally, I would like to express my sincere appreciation to all the readers and supporters of this book. Your interest and enthusiasm for augmented reality development have been the driving force behind my commitment to sharing knowledge and empowering others in this exciting field. Thank you for joining me on this journey.

CHAPTER    1

Getting Started with Augmented Reality

Introduction

Before building augmented reality (AR) applications, it is important to understand what AR is and how hardware and software technology are combined to present an AR application to the end user. The second decade of the 21st century can be seen as a pivotal period for immersive technology, mainly AR and virtual reality. One of the main reasons for the exponential growth in the AR field can be the escalated CPU power of mobile smartphones. The processing power required to perform real-time image processing calculations while estimating virtual element poses was previously unavailable in mobile devices. Therefore, technology did not have a greater reach toward the masses.

A practical definition for AR can be as follows: “AR is a view of the real physical world in which some elements are computer generated and graphically enhanced, allowing extended capabilities in terms of input and output.”

The definition seems unrealistic, but the motivation for AR has been there for decades through science fiction stories. Almost a century ago, the golden age of comics introduced many superheroes and characters with superhuman and augmented abilities, who were placed in scenarios where they used AR technologies as in science fiction. Holograms have been one of the key elements to demonstrate advanced technologies within the stories. These holograms and augmented elements in the physical world had many things in common, such as interactivity with hands or voice, feedback from the elements with animations, showing information with text or sound effects, and the flexibility to spawn them anywhere, regardless of the context. Even though humans are not as advanced as predicted in the movies, there is still rapid development in research and development to invent such concepts.

Structure

This chapter covers the following topics:

■AR implementation and application

■History of AR

■AR across the world

■AR enabling technologies

■Machine learning and artificial intelligence

Objectives

By the end of this chapter, it will be clear that AR was not invented during the last decade but is a topic with a history of many decades. The chapter will also explain the fundamental technologies used to create AR applications and the use cases of AR in different industries.

AR Implementation andApplication

Consider a holographic AR implementation that may be implemented in the future. The technology behind the implementation must first capture and identify the context. There may be a hardware component with built-in sensors to recognize people, objects, and planes, as well as the world around them.

Once the context has been captured, the collected data must be converted into a virtual environment so that a virtual object can be placed on top of it. This conversion may be done using computer algorithms placed within the holographic device. This process can also be identified as the transformation layer.

The final steps of the implementation may include a presentation layer, which consists of all the audio and visual feedback, interactions, logic, and so on. The users of the holographic device may use this presentation layer to generate AR content. The final layer would be the output layer, which generates the accumulated set of components so that the user can see the final implementation. In a hologram, this is the final holographic output. In 2023, there is still a long way to go and years of research and development to implement a nonblocking output layer for a real hologram. Figure 1.1 represents the conceptual representation of a holographic AR implementation:

Figure 1.1 Conceptual representation of a hologram. Source: https://pixabay.com/photos/science-hologram-artificial-fiction-4642115/.

Currently, existing AR systems are very advanced but not yet able to visualize the AR content without another screen or a wearable in between the actual and virtual worlds. Consider a wearable AR device as an example. Such a device contains a camera and other sensors at the front to capture the world around it and uses software to recognize the context. Like the hologram example in Figure 1.1, objects can be placed in the virtual environment generated within the presentation layer. Finally, the generated 3D content will render on the wearable screen, mapping to the physical world that can be seen through the glass.

A smartphone-based AR application would be the same, except that the final output layer would be through the mobile screen, having the camera input as the background. In these AR applications, the camera input must be captured and rendered inside the 3D space behind the 3D models, giving an illusion of how the 3D models are placed in the real-world context. Figure 1.2 illustrates the structure of an AR application:

Figure 1.2 Structure of an AR application.

History of AR

In order to understand current AR development, it is better to know the brief history of AR and where everything started.

1968—The Sword of Damocles

The first implementation of AR that can be found in the history books is not through a computer screen but a prototype head-mounted display. Ivan Sutherland, an American computer scientist and a widely known computer graphics expert, created the first head-mounted display in 1968, known as The Sword of Damocles. The fundamental nature of such a device was to create an illusion on a transparent display, depending on a concept known as the kinetic depth effect.

The display was fixed to the ceiling of a room and could be worn by the user. The linkage in the ceiling measured the pose of the head and transferred that data to the computer program. A miniature cathode ray tube attached to the side of the display projected the image onto the eyeglass display optics, changing the orientation of the image based on the movement of the head. The research paper on this implementation can be obtained within the chapter titled “A Head-Mounted Three-Dimensional Display” of the Fall Joint Computer Conference Journal in 1968. Figure 1.3 features the display:

Figure 1.3 The first head-mounted display and how a 3D object can be seen through it. Source: Fall Joint Computer Conference Journal, 1968.

1975—Videoplaceby Myron Krueger

Another implementation of artificial reality, even though it is not directly an example of AR, can be seen through one of the first augmented interactive applications, known as Videoplace. The idea behind the application is to use external sensors and capture context and interactivity without the use of goggles, buttons, gloves, or anything attached to the user.

The application used a camera to capture the user in front of a projector screen in front of the person. The captured video is transmitted to a computer, which performs image processing to generate a silhouette of the person with an additional interactive element as a separate layer. The layers are generated in different flat colors to differentiate each. The processed final image is then projected to the same projection screen using a back projector. This does not completely fulfill the requirements of AR, but it can be recognized as a step toward artificial reality and contained image-recognition algorithms. Figure 1.4 is an illustration of the Videoplace system:

Figure 1.4 Structure of the Videoplace system.

1999—AR Toolkit

Ever since the first AR implementations, technology has improved to move away from hardware-only solutions to programmable software solutions. One of the major implementations of such solutions is the AR Toolkit, which is a library that was developed in 1999 by Hirokazu Kato of the Nara Institute of Science and Technology.

AR Toolkit was known as the world’s first mobile AR software development toolkit was incorporated after it, and a version was released to the public. The library was used in early OS-based smartphone devices such as Symbian around 2005, all the way to iOS and Android during the 2010 era.

Early versions of the AR Toolkit focused on predefined physical AR trackers (also known as fiducial markers) to recognize the planes and visualize virtual 3D objects following the orientation of the trackers. Later versions of the AR Toolkit introduced natural feature tracking, which allowed developers to train natural features of colorful images rather than using predefined AR trackers.

Figure 1.5 illustrates an AR Toolkit fiducial tracker, which is an example of an AR target:

Figure 1.5 Example of an AR target (AR Toolkit fiducial tracker).

More information on AR Toolkit can be found at the following link:

http://www.hitl.washington.edu/artoolkit.html

AR over the Years

Ever since the development of the AR Toolkit, many developers and organizations have stepped into the field of research and development of AR software development kits and supportive hardware. This was followed by the development of smartphone devices and their easy accessibility, which encouraged engineers and developers to focus more on handheld devices. Early software-based AR applications leveraged the power of desktop computers and attached Web cameras to generate 3D models using magazine covers as image targets. A magazine advertisement done for Mini in 2008 is an example of one of the first AR implementations as a commercial use case. See the following link:

https://www.youtube.com/watch?v=HTYeuo6pIjY

Esquire magazine worked with Hollywood actor Robert Downey Jr. to color their magazine with AR. See the following link:

https://www.youtube.com/watch?v=wp2z36kKn0s

The introduction of Google Glass in 2014 has become one of the key events in the history of AR. It was the first AR-inspired wearable device targeted at consumers. From 2010 to 2015, many software-based development kits and libraries were developed, targeting mainly mobile operating systems. Qualcomm Vuforia, AR Toolkit open source, MetaIO (acquired by Apple to develop its current AR software development kit, ARKit), Wikitude, and ARMedia are some of the early adaptors of AR in the domain of smart handheld devices.

In 2016, Microsoft joined the AR market by introducing the HoloLens, which is the world’s first wearable AR tracking and meshing device, as can be seen in Figure 1.6:

Figure 1.6 Microsoft HoloLens. Source: Kai Kowalewski Creative Commons BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0,viaWikimediaCommons.

Apple released its native AR software development kit (SDK), known as ARKit, and Google started working on a hardware–software implementation known as Google Tango. The project was discontinued and later announced again as a complete software-based implementation known as ARCore. Google also invested in a wearable AR device, Magic Leap, an alternative to Microsoft HoloLens.

The later chapters of this book will dive into some of the most popular AR development tools currently available to develop AR applications in cross-platform devices.

AR across the World

In order to build AR applications, it is better to research and understand the use cases of AR in various disciplines. Emerging technologies during the last couple of decades have been used in various applications outside the field of computer science. This section explores some of the use cases of AR currently.

ARin Entertainment

Many use cases of AR that are consumer-based circle around entertainment. In 2016, Niantic released its widely popular location-based game Pokémon Go. Niantic had been working with location-based games such as Ingress before stepping into AR.

Pokémon Go innovatively implemented the catching a Pokémon using a pokéball interaction using an AR interface within the game. This was a major boost for AR technology as more people could experience AR for the first time.

With the release of Pokémon Go and smartphones having native AR capabilities, many games were developed that used AR as the backbone. Minecraft World, Harry Potter Wizards Unite, and Jurassic World Alive are some of the popular games that have been released since then. Stepping out of the mass outreach of AR games, the technology has also been used in various physical activations and live events. Video-based AR technologies such as Vizrt and Wtvision have taken over the world, offering real-time AR over broadcast. Various augmented content is appearing on news programs, sports events, and other live shows to make the shows more interesting through television. Vizrt technologies have been used in thousands of TV events around the world, mainly covering sports events and news. Some of their customers include Eurovision, ausbiz, Mediacorp in Singapore, CNN-NEWS18, Al-Jazeera documentaries, and many more.

Figure 1.7 features the use of AR in a museum:

Figure 1.7 AR at Museu de Mataró linking to Catalan. Source: Kippelboy, CC BY-SA 3.0, https://creativecommons.org/licenses/by-sa/3.0>,viaWikimedia Commons.

AR has been used in art galleries and museums around the world. In 2021, the Muséum National d’Histoire Naturelle in Paris launched an AR experience. The experience was called REVIRE and allowed visitors to hire a Microsoft HoloLens and interact with digital animals throughout the museum. The National Museum of Singapore has an AR installation called Story of the Forest. An AR installation known as ReBlink is available at the Art Gallery of Ontario, Toronto, enhancing the art experience of the artwork within the gallery. Google has a virtual museum with AR capabilities, allowing users to visit an AR gallery virtually. It can be found at https://artsandculture.google.com/project/ar. These are just a fraction of the thousands of AR installations all around the world.

Having explored some of the existing AR-related use cases in the world and thinking of the future, it is apparent that AR can replace many entertainment channels such as television, billboards, and live events. Think of an AR wearable device that can turn a living room into a stadium with live matches running on the coffee table. Imagine how a wall can be converted to a large cinema screen where one can watch the latest movies without stepping out of the room. The possibilities of AR in the entertainment world are endless, provided the correct tools and technologies.

AR in Manufacturing andLogistics

Google Glass has had many challenges in reaching the consumer market, even though it is the first ever widely announced wearable display device. The device has a camera mounted in front, and there is no indication to the outside world whether the user is capturing the surroundings and streaming a video to a third party. This caused many conflicts with regard to security and privacy. Ultimately, Google discontinued the product to the public as a consumer device and revived it to be used in enterprise applications, mainly focusing on manufacturing, logistics, and anything that requires hands-free work but with access to additional information. Currently, Google Glass provides real-time collaboration, access to training visualizations, and voice input, enhancing the capabilities of factory workers. Even though it is not a completely AR-enabled device, it has been used by many larger organizations across the world, including DHL, Schenker, Samsung, and Volkswagen.

Mainly, AR has been used in factories for maintenance assistance, machine operation training, and data visualization in real time for machinery data retrieval and preventive maintenance. Dynamic arrangement of manufacturing facilities leverages AR for navigation within the factories. Imagine a logistics facility that rearranges the inside of the facility rapidly based on dynamic adaptations. This may allow the facility to change the navigation system along with any rearrangements needed. This can be easily realized by setting up navigation in AR that can instantly replace the data points. Imagine a wearable within a factory that allows machine supervisors to just look at the machine and visualize its sensory and machinery data in real time next to the machine itself. This reduces the overhead time the teams spend searching through data logs through a separate interface matching the data to the machines. Imagine a 3D machine or vehicle construction that can be previewed through AR in a collaborative environment. This allows the designers and engineers working together to visualize the elements on a real-life scale and thus reduces the time required to make foam models of the machine. There are limitless possibilities where AR can be beneficial in manufacturing.

Real Estate

In 2020, Unity, a leading 3D development engine, teamed up with Autodesk Revit, a specialist in building information modeling (BIM), to create a tool known as Unity Reflect. This allowed creators to link BIM model data to a 3D environment in real time. Unity, known for AR, used the same links to visualize 3D models of buildings and architecture in real time in an AR environment. A great use case of this turned out to allow the client to visualize their building design exactly on the building site as a 1:1 scale in AR while the architects change the model data in real time according to the client’s needs.

Similarly, many real estate agencies use AR to enhance the customer experience in buying real estate by providing AR virtual tours. Imagine an application allowing people to go to real estate locations and visualize the price charts and how different buildings appear without spending for concept creators. Imagine, as a real estate agency, customers being able to freely walk around within any available houses with the AR app providing additional information based on their needs. Using AR technologies, the realtors get a competitive advantage in providing more unique personalized experiences to their customers.

Health

Imagine visualizing a 3D reconstruction of the bone structure of a patient while walking around the subject instead of taking multiple X-ray images. AR can allow that to happen, provided the hardware and software required to obtain scan data of a patient are available. AR is a great alternative to current healthcare imaging solutions. Many of the imaging output channels can be replaced by a simple AR wearable that can use its surroundings to import as many 3D visualizations of the environment as possible. Imagine a surgeon performing surgery and being able to visualize a patient’s vital data on top of the patient’s body parts. The surgery can be done faster, reducing the time required for the professionals to look away from the patient. Also, such use cases reduce surgeon errors due to misreading information.

AR can also be used in medical training. Instead of using custom-made physical dummies for health-related simulations, AR glasses can be used to quickly change training scenarios for the trainees to perform interactions and get trained more efficiently.

Education

As suggested in the previous subsections, training can be offered in any competency. It can be the health industry, manufacturing, logistics, or a common school classroom. Schools invest heavily in chemical and physics laboratories, not only by building new labs but also by maintaining them. Imagine an enhanced school physics laboratory where the students can create AR-based tabletop physical simulations while changing the parameters of the objects dynamically. This allows the students to learn the activities by trying out limitless scenarios without changing any physical object in the lab. Imagine a wearable AR headset that can convert a living room into a classroom. Microsoft HoloLens provides shared experiences with the help of spatial anchors allowing people to connect, share, and learn the same experience. Science books make use of marker-based AR to allow the experiments to be viewed outside the book by using a mobile application. The ability to move around freely to observe 3D constructions enables student engagement to learn by doing rather than by memorizing.

Marketing and Retail

One widely used AR application can be found within marketing and retail use cases of AR. IKEA Place is a great initiative done by the company to allow people to add furniture to their own space to visualize whether the piece of furniture fits properly. Dulux has built an AR app to scan any wall and immediately change the color to any available color it offers. There are many marketing-enabling AR apps currently available, such as Blippar, 8th Wall (acquired by Niantic), Zappar, and Augment, which allow companies to use their platforms as authoring tools to build content for their marketing campaigns. Children’s cereal producers use AR apps to extend their cereal boxes for children to collect and play custom-made games for rewards. E-commerce sites make use of AR product viewers to spawn 3D models of products into a user’s space for visualizing them on a real-life scale. Companies like Nike and Adidas use the power of AI and AR and have built apps such as Wannakicks and Wyking AR to visualize shoes directly attached to a customer’s feet. Product visualization, product configurators, and marketing engagement activities have become key entry points for AR in retail. Unlike decades ago, the spectrum of possibilities has expanded with the introduction of AR. It not only provides the wow factor to customers but also creates unforgettable memories in their minds about the products and services that have been marketed with AR. Many businesses have been leveraging the use of AR, providing marketing and retail solutions around AR to bigger companies around the world. Leading service-oriented companies, such as Accenture and Volume Global, have already filled their portfolios with AR case studies, and there is more to come.

Any readers of this book who hope to provide services with AR solutions can think of the problems that are faced in different disciplines across the world and evaluate how those problems can be solved using AR. However, there are many AR use cases that create problems rather than offering solutions. Therefore, it is vital to understand the requirements and measure the importance of AR in replacing existing workflows. Considering the costs, maintainability, and usefulness of AR technology, any industry can leverage it for greater savings, higher profits, and increased engagement.

AR Enabling Technologies

AR relies heavily on computer vision. The use of additional data, such as depth sensing for meshing, catalyzes the accuracy of the trackers. Ever since the use of machine learning and AI, the accuracy of target detection and tracking and the extended capabilities of AR have expanded. The following section explores some of the computer vision techniques that enable AR.

Image Recognitionand Tracking

In order to enable AR in a predefined target image (for example, a poster, magazine cover, photo, QR code, and so on), the features of the image must be matched and recognized. This process is known as feature detection and matching. Once the image has been detected, the data can be used to track the image with its movement with respect to the position of the camera. These algorithms are used not only for AR but in many applications. Robot navigation systems, image retrieval, object tracking, motion detection, and segmentation are some of the other applications of feature detection and matching. A feature of an image can be recognized as a piece of information in the image that can be used by the computer program to recognize the image. As an example, the points and edges of an image can be used as features of the image. The challenge of image detection is to detect and capture the features, collect information about the appearance around the feature point, and match similar features with a known feature distribution set. These feature points are identified by various algorithms based on either the brightness of the image or the boundary/edge details of the image.

A feature recognition algorithm gets an image as an input and outputs a series of vectors. The output of an image would be a series of encoded data points in a machine-understandable format. This is known as a feature descriptor. The information that is generated from an image is supposed to be independent of the orientation and the transformation of the image. Therefore, the output data remains the same regardless of the movement of the image. This provides one of the fundamental abilities of AR, which is the ability to track a detected image based on the image features by recognizing and tracking the image detected. Figure 1.8 illustrates a colored poster and how its transformed feature points are aligned together:

Figure 1.8 Features of an image (features are marked with x).

The neighborhood around a feature point is known as a local descriptor, which resembles the appearance of an area around its feature point. The basic flow of feature detection and matching can be listed as follows:

■Use image processing methods to detect certain points. They can be based on the contrast of the image, edge detection, and so on.

■Capture the information around the detected feature point.

■Generate a local descriptor, an encoded vector that describes the region around the feature point.

■Match the local descriptors based on another factor, such as the time, predefined descriptor, and so on.

Following are the image processing algorithms that support feature detection and tracking:

■Harris corner detector

■Scale invariant feature transform (SIFT)

■Speeded-up robust feature (SURF)

■Binary robust independent elementary features (BRIEF)

■Features from accelerated segment test (FAST)

■Oriented FAST and rotated BRIEF (ORB)

■Binary robust invariant scalable key points (BRISK)

OpenCV is a library of computer vision algorithms that is under Apache open-source license. In-depth details of all the listed algorithms can be read via OpenCV documentation at https://docs.opencv.org/.

SimultaneousLocalization and Mapping

Simultaneous localization and mapping(SLAM) has been an integral part of not only AR but of many research areas in the engineering field, such as robotics. The basic idea of SLAM is to use the cameras and other sensors of the computational device to generate a virtual map of the surroundings. Once the map has been generated, the information is then fed into its own movement system, where it can localize itself within the virtual map.

SLAM can be achieved in many ways depending on the device. For robots, it uses cameras, LiDAR sensors, motion, and other sensors to map the virtual world. Some systems use beacons placed around the world to help the device map the environment more accurately. For mobile AR, however, it is only the camera, the gyroscope, the accelerometer, and other movement-based sensors that can be used to map the environment.

For example, consider a mobile AR solution that works on top of a SLAM integration. The ultimate goal is to capture the outside world and convert it into a map. The flow of the process can be listed as follows:

■Use the camera of the device to capture the image and the gyroscope and the accelerometer to capture the pose and the movement vectors of the device.

■Extract the features based on the input data. As described in the image processing section, a feature descriptor is used to convert the image data to vector data, such as key points and neighborhood data.

■Association of data is done to neglect the features that have been captured before. This is a repetitive process. Therefore, the features that were captured in one frame (in one iteration) should be passed to the other frame. Instead of duplication, similar data points must be recognized. This is also known as a loop closure.

■The captured data are used to determine the movement of all the feature points and transferred to map construction. Based on the algorithm, the generated map will be a point cloud or a sparse reconstruction based on the distribution of the feature points. Figure 1.9