The Infinite Retina E-Book

Irena Cronin

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Veröffentlichungsjahr: 2020

Beschreibung

What is Spatial Computing and why is everyone from Tesla, Apple, and Facebook investing heavily in it?

In The Infinite Retina, authors Irena Cronin and Robert Scoble attempt to answer that question by helping you understand where Spatial Computing?an augmented reality where humans and machines can interact in a physical space?came from, where it's going, and why it's so fundamentally different from the computers or mobile phones that came before.

They present seven visions of the future and the industry verticals in which Spatial Computing has the most influence?Transportation; Technology, Media, and Telecommunications; Manufacturing; Retail; Healthcare; Finance; and Education.

The book also shares insights about the past, present, and future from leading experts an other industry veterans and innovators, including Sebastian Thrun, Ken Bretschneider, and Hugo Swart. They dive into what they think will happen in Spatial Computing in the near and medium term, and also explore what it could mean for humanity in the long term.

The Infinite Retina then leaves it up to you to decide whether Spatial Computing is truly where the future of technology is heading or whether it's just an exciting, but passing, phase.

Leseprobe

The Infinite Retina

Spatial Computing, Augmented Reality, and how a collision of new technologies are bringing about the next tech revolution

Irena Cronin

Robert Scoble

BIRMINGHAM - MUMBAI

The Infinite Retina

Copyright © 2020 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Producer: Ben Renow-Clarke

Acquisition Editor – Peer Reviews: Suresh Jain

Content Development Editor: Dr. Ian Hough

Technical Editor: Gaurav Gavas

Project Editor: Radhika Atitkar

Proofreader: Safis Editing

Indexer: Dominic Shakeshaft

Presentation Designer: Sandip Tadge

First published: May 2020

Production reference: 1060520

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-83882-404-4

www.packt.com

packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from more than 4,000 industry professionalsLearn better with Skill Plans built especially for youGet a free eBook or video every monthFully searchable for easy access to vital informationCopy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.Packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.Packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Foreword

When I was young, I told my dad that I'd rather own a computer than a home. Of course, that was back when a 4K Data General Nova cost more than a house in Silicon Valley.

Today, just a few decades later, we all carry a computer that is thousands of times more powerful than the computers of my childhood. Computers used to be huge, but today's tech is getting so small, we can wear them. The $250 Apple AirPods Pros have more compute power inside them than an iPhone 4 and certainly a lot more than that Data General computer that cost tens of thousands of dollars back in the 1970s.

People often ask me what I would engineer if I were starting out today. When I worked at HP we built calculators and huge laser printers. Currently, there's tons of research being done in Spatial Computing, which includes the hardware and software surrounding Robotics, Autonomous Vehicles, Augmented Reality, and Virtual Reality. All of these bring interesting engineering problems that would keep me interested for sure. I see the potential in all of them to bring computing to people in a way that is ubiquitous, more powerful, and far more personal.

Spatial Computing will move computing from something that sits on a desk, or that you hold in your hands, to something you move around with. There, you'll be joined by robots, virtual beings, and assistants. I'm optimistic that Spatial Computing will be used to serve us with a deeper understanding of ourselves, and give us wild new ways to experience the world and new ways to become more productive in businesses and classrooms.

Children soon will have "imagination machines" on their faces. That will be in contrast to the toys I grew up with. One of my favorites was a transistor radio, which had six transistors (a modern phone has billions of transistors, by comparison). That radio brought the world to me. Spatial Computing will open up new worlds to today's kids that simply didn't exist when I was growing up.

To me I'm excited by what the children of today will do with these technologies and hope they remember that life is about happiness. Smiling and laughing is good. Hearing music in new ways is good. Telling new jokes to each other is good. Spatial Computing can do all that and can bring new ways to learn, work together, and travel the world to boot.

What Irena Cronin and Robert Scoble have done in their book The Infinite Retina is provide readers with an accessible, useful and entertaining guide to all of the latest advances in Spatial Computing and how this new technology is influencing and priming the way forward for a host of new innovations that will affect how we live our lives, how we go to work, and how we communicate with each other for years to come.

– Steve Wozniak

Contributors

About the authors

Irena Cronin serves as CEO of Infinite Retina, which provides research and business strategy to help companies succeed in Spatial Computing. She was previously CEO of Transformation Group, which advised decision makers on business strategies related to Artificial Intelligence, Augmented and Virtual Reality, Machine Learning, Facial Recognition, Robotics, Autonomous Vehicles, and related disruptive technologies. Previous to this, she worked for several years as an equity research analyst and gained extensive experience of evaluating both public and private companies.

Cronin has a joint MBA/MA from the University of Southern California and an MS in Management and Systems from New York University. She graduated with a BA from the University of Pennsylvania with a major in Economics (summa cum laude). She has near-fluent proficiency in Mandarin, intermediate in Japanese, and beginner in Korean.

Robert Scoble is Chief Strategy Officer at Infinite Retina and works with companies that are implementing Spatial Computing technologies. He is a futurist and technology strategist and the author of three books about technology trends, being the first to report on technologies from autonomous vehicles to Siri. Previous positions include being a strategist at Microsoft, a futurist at Rackspace, and the producer and host of a video show about technology at Fast Company.

About the reviewer

Robert Crasco, futurist, thought leader, influencer, social media manager, and occasional developer and designer. His focus is on Spatial Computing (Virtual Reality, Augmented Reality, Mixed Reality, 360 Video, VR180, Volumetric Capture, and virtual worlds), Artificial Intelligence, Machine Learning, Robotics, 5G and technology's impact on society. He has a background in computer science and marketing and has worked for various tech companies, including AT&T, Ziplink, NewsCorp, and iBasis. He was a contributing author on Convergence: How The World Will Be Painted With Data. He has over a decade experience working with virtual worlds. He currently works with technology-focused individuals and brands on social media management, social amplification, social promotion, content curation, and content creation.

Contents

Introduction

Spatial Computing – The New Paradigm

Exploring Technological Change

Transformation Is Coming

New Vision

Pervasive Change – Shopping, Healthcare, and Finance

New Ways to Learn

Meeting the Pioneers

Thinking Ahead

Starting the Journey

Part I: Why Spatial Computing and Why Now?

Prime Directive

What Makes Us Human?

Drivers and Benefits

Potential Dangers

Understanding the Natural World

Faster Wireless Leads to New Affordances

Data, Data, Data Everywhere

The Impulse to Capture, Understand, and Share

Dancing Into Different Worlds

The Frustrated Pioneer

Breakthroughs in Sensors

Driven to Improve

Hands-on Use

Beginning

Four Paradigms and Six Technologies

Introduction

The Four Paradigms

Paradigm One – The Arrival of the Personal Computer

Paradigm Two – Graphical Interfaces and Thinking

Paradigm Three – Mobile

Paradigm Four – Spatial Computing

Six Technologies

Optics and Displays

Wireless and Communications

Control Mechanisms (Voice, Eyes, and Hands)

Sensors and Mapping

Computing Architectures

Artificial Intelligence (Decision Systems)

Evolution

Part II: The Seven Visions

Vision One – Transportation Automates

A Future Vision

The Paths to Autonomy

Falling Car Transportation Costs

Autonomous Vehicle Round-up

Zoox

Ford

General Motors' Cruise

Uber

Waymo

Tesla

Lyft, Magna, and Aptiv

Hyundai, Pony.ai, Aptiv, and Aurora

Aurora

Driven and Oxbotica

Autonomous Trucks

Flying Cars (eVTOLs)

The Autonomous and Electric Culture Shock

The Future City

The Changing City

Auto Dealers

Smart Roads

Interstates

Factories

Real Estate

Remote Working

Travel Times and Ranges

Navigation

Green Thinking

AI Assistants

Entertainment

Fewer Cars

"Data Bubbles" Will Abound

Vision Two ‒ Virtual Worlds Appear

From 2D to 3D

Increasing the Bandwidth

Change is Coming

Emergence as a Human Hook

The Brain and VR

The Right Tool for the Job

The "Glassholes" Show the Way

Defeated Expectations

Where Did Things Go Wrong?

What It Means to Be Human

Embodiment, Immersion, Presence, Empathy, and Emergence

Embodiment and Immersion

Evolution

Facebook has the Edge

Market Evolution

2014's New Wave

Inside-out

Mistakes Were Made

The Spectrum ‒ Augmented Reality from Mobile to Mixed Reality

Closer Computers

The Origins and Limitations of Augmented Reality

Wearable AR

Too Much and Too Little

Getting Past the Setbacks

Vergence and Accommodation

Fast, Cheap, or Good

Differences To Be Aware Of

From Kinect to the HoloLens

Technology Takes a Leap

Patents and Problems

Volumetric and Light Field: Capturing Our World in a New Way

Shrinking Concepts, Expanding Horizons

Shooting Some Volumetric

Camera Magic

AR Clouds (3D Mapping) and Internet Infrastructure Including 5G

A New Kind of Map

Building a Map

Ready for 5G

New Audio Capabilities Arrive

Audio Augmentation

Spatial Audio

Gaming and Social VR Experiences

A World Without Rules

The Ripples of the VR Phenomenon

Building a VR Community

Great Expectations

Location-based VR

The Void

Dreamscape Immersive

Sandbox VR

Cinematic and Interactive Experiences

Festivals That Include VR

Marketing the Unmarketable

Engineering Experiences

Content Production and Previsualization

The Merging of the Virtual and the Real

Vision Three – Augmented Manufacturing

Robots and Other Augmentation

Augmented Reality Visions and Informed Robots

The Mirror World Comes to Factory Floors

The Human Robot

Vision Four ‒ Robot Consumers

What Drives Consumers?

Getting Emotional

Now in 3D

Retail Destruction? No, But Radical Change Underway

Trying to Keep Up

Looking Ahead

Variations on a Retail Theme

Amazon Go is Watching You

Spatial Computing for Logistics

Spatial Computing Delivery Methods – Robots, Drones, Future

In Your Living Room and Elsewhere ‒ Spatial Computing Marketing Experiences

Starting in the Living Room

A Personalized Experience

The VR World

Making an Impact

A "No Install Needed" AR World

The Past and The Future

Will Anything Remain the Same?

Vision Five ‒ Virtual Healthcare

The Virtualist

Fixing Humans – Surgery Preparation and Surgery

Brain Tricks/Virtual Medicine

The Virtual Coach and the "Check Engine Light"

Data, Data, Everywhere

Precision and Personalization

Vision Six – Virtual Trading and Banking

3D Data Visualization

Virtual Trading

ATM Security and Facial Payment Machines

Virtual Branch Functionality and Customer Service

Future Adoption

Vision Seven ‒ Real-Time Learning

Experiential Digital Learning

No Teachers, Books, or Movies? New Approaches in the Offing

Brainier Learning

The Future of Education

The Grabbable Classroom

Workplace Training and Simulation

A Virtual Arm for Human Resources

Our Learning Futures

Part III: The Spatial Business

The Always Predicted World

The Predictive Breakthrough

Glasses That Understand You

(Human) Assistance Not Required

Pervasive

Proactive

Connected

Data Dance With Seven Verticals

Transportation

Technology, Media, and Telecommunications (TMT)

Manufacturing

Retail

Healthcare

Finance

Education

The "Bubbleland"

The Bubble Network

Unified Learning, Shared Experiences

Pervasive and Proactive Future Assistance

Spatial Computing World-Makers

The Market Illuminator

The Pain Reducer

The Investing Visionary

The Immersive Genius

The Synthetics Craftsman

The Chip Innovator

The Future Flyer

Spatial Computing Paths

How Human?

Transformative Experience

Privacy and Security

On Facebook and Cambridge Analytica

OJ Simpson and Location Data

Apple and the FBI

Identity and Ownership

Blurred Faces on Google Street View

China and Facial Recognition

Issues with Facial Recognition in the US and Europe

Human Social Good

Epilogue

Index

Introduction

We live in a world that is today undergoing, or perhaps by the time you read this, has undergone, a shift due to the novel coronavirus disease, officially called COVID-19. COVID-19 has caused a change in our way of communicating and the way in which we conduct business, albeit, most everyone is still using two-dimensional visual communication apps, such as Zoom, versus three-dimensional visuals at this point and will be for the next few years. Following the outbreak in 2020, schools and universities were shut down for weeks, in some cases months (depending on the location), with classwork expected to be done online. Many venues closed and events were postponed or cancelled, from Disneyland, to conferences and music festivals, to baseball and all other live sports. Governments around the world ordered people to stay home. The world was not prepared for a pandemic like COVID-19.

As we stayed home for weeks, many of us discovered new ways to work, play, educate, entertain, and shop. We also saw new technology become more important, whether it be Artificial Intelligence being used to look at CAT scans, to look for the virus, or to create a vaccine, or new wearable computers used in China and elsewhere to diagnose people with the disease. Plus, having autonomous vehicles or robots preparing our food or delivering it seemed a lot smarter to us all of a sudden.

At home, music artists released performances for VirtualReality (VR) headsets to keep fans entertained during the weeks when people had to stay home. New virtual events, some of which were being trialled in Virtual Reality, were announced to replace the physical conferences lost for the year. These are early signs of a set of new technologies that promise to deeply transform all of computing. We call this set of new technologies "Spatial Computing."

Spatial Computing – The New Paradigm

Spatial Computing comprises all software and hardware technologies that enable humans, virtual beings, or robots to move through real or virtual worlds, and includes Artificial Intelligence, Computer Vision, Augmented Reality (AR), VR, Sensor Technology, and Automated Vehicles.

Seven industry verticals will see transformational change due to Spatial Computing: Transportation; Technology, Media, and Telecommunications (TMT); Manufacturing; Retail; Healthcare; Finance; and Education.

These changes are what is driving strategy at many tech companies and the spending of billions of dollars of R&D investment. Already, products such as Microsoft'sHoloLens AR headset have seen adoption in places from surgery rooms to military battlefields. Devices like this show this new computing paradigm, albeit in a package that's currently a little too bulky and expensive for more than a few of the hardiest early adopters. However, these early devices are what got us to be most excited by a future that will be here soon.

Our first experiences with HoloLens, and other devices like it, showed us such a fantastic world that we can predict that when this new world arrives in full, it will be far more important to human beings than the iPhone was.

We were shown virtual giant monsters crawling on skyscrapers by Metaio years ago near its Munich headquarters. As we stood in the snow, aiming a webcam tethered to a laptop at the building next door, the real building came alive thanks to radical new technology. What we saw at Metaio had a similar effect on Apple's CEO, Tim Cook. Soon, Apple had acquired Metaio and started down a path of developing Augmented Reality and including new sensors in its products, and the capabilities of this are just starting to be explored. Today's phones have cameras, processors, small 3D sensors, and connectivity far better than that early prototype had, and tomorrow's phones and, soon, the glasses we wear will make today's phones seem similarly quaint.

In Israel, we saw new autonomous drones flying over the headquarters of Airobotics. These drones were designed to have no human hands touching them. Robots even changed memory cards and batteries. New Spatial Computing technology enabled both to "see" each other and sense the world around us in new ways. The drones were designed to fly along oil pipelines looking for problems, and others could fly around facilities that needed to be watched. Flying along fences and around parking lots, their Artificial Intelligence could identify things that would present security or other risks. These drones fly day or night and never complain or call in sick.

Focusing on just the technology, though, would have us miss what really is going to happen to the world because of these technologies. Our cities and countryside will reconfigure due to automation in transportation and supply chains as robot tech drives our cars, trucks, and robots rolling down sidewalks delivering products. We'll spend more time in virtual worlds and metaverses. More of our interfaces, whether they are the knobs on our watches, cars, doors, and other devices, will increasingly be virtualized. In fact, many things that used to be physical may be virtualized, including stores and educational lessons from chemistry experiments to dissection labs.

Computing will be everywhere, always listening, always ready to talk back, and once we start wearing Spatial Computing glasses, visual computing will always be there ready to show us visualizations of everything from your new designs to human patterns in stores, on streets, and in factories and offices. Some call this "invisible computing" or "ambient computing," but to us, these systems that use your eyes, voice, hands, and even your body as a "controller" are part of Spatial Computing.

At the same time, all this new computing is joined by radically fast new wireless technology in the form of 5G. The promises of 5G are threefold. First, we'll have more than a gigabit of data if we have the highest bitrates, and even the lowest rates promise to give us more bandwidth than current LTE phones. Second, wireless will soon add almost no latency, which means that as soon as you do something, like throw a football in a virtual game, it'll happen, even if you have many players viewing the football in real time. Third, 5G supports many more devices per tower, which means you will be able to live stream even a Taylor Swift concert with tens of thousands of fans filling a stadium.

When you combine 5G with all the new things under the Spatial Computing umbrella, you get a big bang. All of a sudden, cars can work together, one sending detailed 3D imaging of streets instantly to others behind it. New kinds of virtual games will be possible in the streets, where hundreds of people can play virtual football games in parks and other places. Crazy new virtual shopping malls appear where virtual celebrities show you around, and you can imagine new products directly in a virtual scan of your home and in other places as well.

A range of new capabilities will appear over the next few years in devices you wear over your eyes. There will be very light ones that are optimized for utility, showing you navigation, notifications, reminding you where you have left things, or nagging you to do some exercise or meditation to keep on top of your physical and mental health. There also will be heavier devices that will be more optimized for everything from detailed design or architecture work to entertainment and video game work. We can even imagine owning several different kinds of Spatial Computing devices, along with some smart contact lenses, that will let us go out on a date night without looking like we have any computing devices on at all.

As the 2020s dawn, we have VR devices that cost a few hundred dollars that are great for games and a few other things, like corporate training. On the more expensive side of the scale, we have devices that can be used by car designers, or even as flight simulators to train airline pilots. The expensive ones, though, will soon look as out of date as one of the first cell phones does today. By 2025, the computing inside will shrink to a fraction of the size of today's devices and the screens inside will be much sharper and capable of presenting virtual and augmented worlds to us that far exceed what we can experience today.

It is this next wave of devices that will usher in the paradigm shift in computing and in human living that we are discussing here. Already these changes are benefiting many enterprises, raising productivity. Inside many warehouses, hundreds of thousands of robots scurry about, moving products from trucks to packages. These new warehouses have evolved over the past decade and enable retailers to keep up with floods of new online orders that, back in 2000, were only dreamt of in futuristic books like this one.

The productivity gains will spread to many jobs. At Cleveland Clinic, surgeons are already using similar technology that shows them digital views from ultrasound, CAT scans, and other sensors. Like the warehouse worker who sees a blue line on the floor telling her how to find the product she's looking for, in this case, when a surgeon navigates to the right place to cut out a cancerous tumor, it lights up like a missile guidance system and tells the surgeon they are in the right place.

Other systems help workers "phone a friend" with new remote assistance features. This can help companies that have expensive machinery, or other work forces, including surgeons, architects, and engineers, save money. At some plants, the savings will be substantial. It took us 30 minutes to simply walk across the Boeing floor where it builds airliners. Asking someone for advice virtually might save someone an hour of walking just to come over and see your problem in a plant like that.

New devices let these remote helpers see what you are dealing with, and they can often show you visually what to do. Imagine trying to remove an engine while holding a phone or tablet in your hand. These systems, because they use wearable glasses, can let workers use both of their hands while talking and showing the remote assistant what is happening. The savings in downtime can be extreme. Imagine a problem that is causing a shutdown in a line at Ford. Every minute it is down, Ford loses about $50,000.

Even for salespeople and managers, the cost savings add up. A flight from San Francisco to Los Angeles usually costs about $700, including airfare, hotel, a decent meal, and an Uber/taxi ride or two. Increasingly, these meetings will be replaced by ones held in Virtual Reality. Two headsets are $800. In just a couple of virtual meetings, the headsets pay for themselves. Social VR from Facebook, Spatial, Microsoft, and others are rapidly improving to make these virtual meetings almost as good as physical ones. The time and cost saved will add up to big numbers, and workers will be happier. Workforces will be less likely to pick up viruses from travelers, too, which will also add up to big savings for corporations and reduced risks.

More lives could be saved, too. Mercedes-Benz had an Augmented Reality system built to show first responders how to cut apart a wrecked car. The app showed where fuel and electrical lines were so that firefighters working to free an accident victim wouldn't start a fire or electrocute themselves. This isn't the only example we have of this technology helping first responders, either through better training or by giving them various assistance on scene. One such system helps police gather evidence and then be able to recreate crime scenes for juries that they can virtually walk around.

Here, we've given you a taste of just how much the world is about to get reconfigured because of Spatial Computing technologies. Let's dig into the details of the chapters ahead.

Exploring Technological Change

Spatial Computing's technological change is laid out in Chapter 1, Prime Directive. Mobile phones soon will give way to headsets and glasses that bring computing to every surface. What is driving all of this new technology? We have a need for complex technologies to keep us around on this planet longer and in a more satisfied and productive state. What will drive us to build or buy new headsets, sensors, and vehicles, along with the connected systems controlled by Artificial Intelligence? Augmentation is coming, and that can mean a lot of different things, which we will explore.

We look back in Chapter 2, Four Paradigms and Six Technologies, at the previous three foundations of personal computing and include the new Spatial Computing paradigm. The six technologies discussed are those that enable Spatial Computing to work: Optics and Displays, Wireless and Communications, Control Mechanisms (Voice and Hands), Sensors and Mapping, Compute Architectures (new kinds of Cloud Computing, for instance), and Artificial Intelligence (Decision Systems).

It all started with the personal computer of the late 1970s. That paradigm shift was followed by graphical user interfaces and networking in the 1980s and the mobile phone and other devices that started arriving in the 1990s, culminating with the iPhone in 2007. Then, we look forward to the next paradigm and why it will be so different and why so many more people will be able to get more out of Spatial Computing than the laptops, desktops, and smartphones that came before.

Human/machine interfaces are radically changing, and we visit the labs that brought us the mouse to understand the differences between how humans interfaced with computers with keyboard and mice to how we'll interface with cloud computing that is hyper-connected by using voice, eyes, hands, and other methods, including even wearing suits with sensors all along our bodies. It's amazing to see how far we've come from the Apple II days, where there were very few graphics, to Spatial Computing where cameras see the real world, decipher it, and decide how to drive a car around in it.

Transformation Is Coming

Because of this new technology, cities and even the countryside will change, autonomous vehicle pioneers tell us in Chapter 3, Vision One – Transportation Automates. Soon you will tell your glasses, "Hey, I need a ride," and you'll see your ride arrive. Sounds like Uber or Lyft, right? Look closer, there isn't a driver inside. Now think of the cost and the other advantages of that. Economists see that such a system could be a fraction of the cost, and could do many other things as well: "Hey car, can you go pick up my laundry and then dinner for our family?" The problem with such a world is that it is probable, many tell us, that we'll see much more traffic near cities as we use transportation to do new things, like pick up our laundry. This is why Elon Musk came up with the Boring Company to build tunnels under cities. We show some other solutions pioneers have come up with, including special roads for these vehicles and new kinds of flying vehicles that will whisk commuters into city centers, passing above all that new traffic.

Transportation soon will include more than just cars and trucks, too. Already, lots of companies are experimenting with new kinds of robots that will deliver products much more efficiently and, in a world where viruses are a new threat, without human hands touching them either. We talk with a company that is already rolling out such robots on college campuses and elsewhere.

Autonomous cars might look like they are rolling around the real world, but often they are developed by rolling them around inside a simulation. Simulations are how engineers are able to test out AI systems and come up with new ways to train the AIs. After all, you don't want to wreck 500 cars just to come up with how to handle someone running a red light, do you? If you walk around the simulations built by Nvidia and others, they look like real streets, with real-looking and acting traffic, pedestrians, and even rainwater and puddles after rain. This technology has many new uses other than training robots and autonomous vehicles, though. The technology inside is a radically different form of computing than was used to make Microsoft Windows for the past few decades, too.

Here, new AI systems fuse dozens of sensor and camera readings together and then look for patterns inside. Some of the cars rolling around Silicon Valley and other cities, like Phoenix, Arizona, have more than 20 cameras, along with half a dozen spinning laser sensors that see the world in very high-definition 3D.

Is that a stop sign or a yield sign? Humans are good at that kind of pattern recognition, but computers needed to evolve to do it, and we dig into how these systems work for autonomous cars and what else this kind of technology could be used for―maybe for playing new kinds of games or visiting new kinds of virtual amusement parks where virtual actors interact with you? How will such things be possible? Well, let's start with the huge amount of bandwidth that will soon appear as 5G rolls out and new devices show up on our faces and in our pockets to connect us to these new Spatial Computing systems. Yes, 5G can support these new kinds of games, but it also can tell cars behind you that there's a new hazard in the road that needs to be avoided.

New Vision

Games aren't the only things that better devices will bring. Chapter 4, Vision Two – Virtual Worlds Appear, provides details on Technology, Media, and Telecommunications, another of our seven industry verticals to be disrupted. We start out by detailing the different kinds of devices that are available to bring a spectrum of Spatial Computing capabilities to your face, from Virtual and Augmented Reality headsets to lightweight smart information glasses, and even contact lenses with displays so small that it will be very hard to tell that your friend is wearing one.

There are pretty profound trade-offs made as manufacturers bring devices to the market. VR headsets emphasize immersion, or the feeling you get when you see something beautiful wrapped all around you. Augmented Reality headsets focus on the virtual layer that they reveal on top of the real world. Often it's amazing and magical, albeit usually with less of that "I'm in a dark movie theater with a huge screen" feeling. Then there are a few other devices that focus mostly on being lightweight, bringing navigation and notification-style functionality. Our guide isn't designed to be comprehensive, but rather to you to understand the market choices that both businesses and consumers will have to soon make.

While cataloging the device categories, we also show some of the new entertainment capabilities that soon will come, which will be captured with new arrays of volumetric and light-field cameras. We visited several such studios and delivered you into a new entertainment world, one where you can walk around in, and interact with, objects and the virtual beings inside.

These new media and entertainment experiences are arriving with a bundle of novel technologies, from AR Clouds, which contain both 3D scans of the real world and tons of virtual things that could be placed on top, to complete metaverses where users can do everything from build new fun cities to play new kinds of games with their friends. In enterprises, they are already building a form of an AR Cloud, called a "Digital Twin," which is changing a lot about how employees are trained, work together, and manage new kinds of factories.

We have visited the world's top manufacturing plants, and in many of them, we see new kinds of work being done with lots of robots that didn't exist just a few years ago, with workers walking around wearing new devices on their faces helping them learn or perform various jobs. In Chapter 5, Vision Three – Augmented Manufacturing, you'll learn about how Spatial Computing is changing how factories are even designed. Increasingly, these factory floors are using robots. The robots are different than they used to be, too. The older ones used to be kept in cages designed to keep humans away. Those can still be found welding, or like in Ford's Detroit factory, putting windshields into trucks. Newer robots work outside cages and sometimes, can even touch humans. These types of robots are called "cobots" because they cohabit with humans and can greatly assist workers.

In the upcoming years, humans will both be trained to work with these new robots using new VR and AR technologies as well as train the robots themselves in new headsets with new user interfaces that let humans virtually control factory floors. As these new Spatial Computing technologies are increasingly used on factory floors, they bring new capabilities, from virtual interfaces to physical machines and new productivity enhancers. For instance, in many of these systems, workers can leave videos, 3D drawings, and other scans, and other notes for workers on the next shift to see. "Hey Joe, the cutting machine is starting to misbehave. I ordered a new motor for it so you can fix it when the line is down at 2 p.m."

Pervasive Change – Shopping, Healthcare, and Finance

The products that Joe makes will eventually be found in retail stores, many of which will be quite different than the ones that Sears ran decades ago. Walk into an Amazon Go store and look at the ceiling. You'll see hundreds of cameras and sensors aimed back at you, watching and categorizing every move you make. In the Go store, they charge you for the products you take without you having to talk to a checkout clerk or pull out a credit card, or really do anything other than walk out of the store. In Chapter 6, Vision Four – Robot Consumers, we detail these changes, along with others that make retail stores, even traditional ones, more efficient and better for both consumers and sales for producers, and useful new Augmented Reality technologies that make shopping at home much easier.

As we were finishing up this book, Apple announced a new iPad with a 3D sensor, and it demonstrated why people should buy one of these new iPads by showing off one of these futuristic shop-at-home experiences where the person holding the tablet could drop chairs and other items into their real home to see how they fit. The changes, though, don't stop at the shopping experience. Behind the scenes, these technologies are making things more efficient, helping logistics companies pick and pack products faster and getting them to stores faster and with fewer losses, and we detail how Spatial Computing is making a difference there.

Similar changes are underway in healthcare. They are so profound that Dr. Brennan Spiegel, Director of Health Research at Cedars-Sinai, says that we should expect a new kind of healthcare worker: "The Virtualist." This new kind of practitioner will perform several roles: help patients prevent disease, help doctors deliver a new form of healthcare, and help the nurses, doctors, and other staff perform their jobs more efficiently using new Spatial Computing technology. For instance, let's say you need surgery. Well, a new surgery team at Cleveland Clinic is already using MicrosoftHoloLens 2 headsets to see inside you, thanks to images from scanners being visualized inside the surgeon's headgear. We discuss, in Chapter 7, Vision Five – Virtual Healthcare, how its system guides the surgeon to the right spot to cut out a patient's cancerous tumor.

That's just a tiny piece of what's happening in healthcare due to Spatial Computing technology. On another wing of the hospital, doctors are using VR to address mental illnesses and ailments from PTSD to dementia, with more applications on the way. At the University of Washington, they discovered it often is much better at treating pain than using opiates, which are much more dangerous, killing tens of thousands of Americans every year due to addiction. In other places, nurses noticed that patients going through tough procedures preceding childbirth felt a lot less pain, too, if they were watching a 360-video experience during the process. Other doctors even found ways to enhance athletes' perception. These brain tricks and virtual remedies have the capacity to significantly change healthcare. Pfizer's head of innovation told us that she views Augmented and Virtual Reality as the future of medicine.

How does all this work? Largely on data. New predictive systems will watch your health by having sensors look into your eyes, watch your vascular system, or blood streams, and sense other things, too, maybe to the point where they see that you are eating too much sugar or smoking too many cigarettes. One could even warn your doctor that you aren't taking your medicine or performing the exercises that she prescribed. Are we ready for new tough conversations with our doctors? Remember, in this future, maybe your doctor is a virtual being from whom you don't mind hearing the harsh truth. One of the studies we found showed that patients actually are much more honest about their mental health problems when talking with a virtual being, or in a chat that's run by Artificial Intelligence.

These AIs won't just be helping us keep our health on track, either. Similar systems might let us know about market changes that we need to pay attention to or, like banks already do when they notice that buying behavior is out of bounds, warn you about other things. The financial industry is generally a relatively conservative one, so adoption of Spatial Computing technologies there is contingent upon demonstrated and clear utility, and they must be additive to the bottom line. Currently, there is very little Spatial Computing that is being actively used there; however, the possibilities are very promising. In Chapter 8, Vision Six – Virtual Trading and Banking, we cover the future uses for Spatial Computing in the financial industry.

In this chapter, we review the functional areas where we think Spatial Computing will have its greatest impact, including 3D data visualization, virtual trading, ATM security and facial payment machines, and virtual branch functionality and customer service.

Someday soon, we may never go into a physical bank again due to these changes, but could the same happen with our schools? Already, teachers are using Augmented and Virtual Reality to teach all sorts of lessons, from chemistry experiments to math visualizations, to even virtual dissections of real-looking animals.

New Ways to Learn

COVID-19, though, showed us that sometimes we might need to rely completely on teaching virtually, and in Chapter 9, Vision Seven – Real-Time Learning, we talk with educators and others who are using technology aggressively to make learning more virtual. It isn't only for kids, either. Soon, because of automation, we'll need to retrain millions of adults around the world, and schools and universities are responding with new curricula, new learning programs for Virtual and Augmented Reality, and new support systems to enable even truck drivers to change careers. Speaking of careers, already at companies like Caterpillar, it is using Augmented Reality glasses to train workers to fix their expensive tractors in real time. Many new VR-based training systems are being developed, from simulators to help police learn how to deal with terrorist situations to ones that show quarterbacks how to perform better, to training at Walmart that shows retail workers how to manage stores better. Verizon even trained its retail store workers on what to do if they are being robbed using VR-based training. What if, though, the system could do even more, we asked, and predict what we might do next and assist us with that?

How can computers predict our next move? Well, truth be told, we are somewhat predictable. We buy groceries at the same store every week, visit the same gas stations, go to the same churches, schools, offices, movie theaters, laundries, and head home at pretty much the same time every evening.

Watching our friends, we can usually predict what they will order from menus or how they will complain when we try to get them off pattern. Ever try to take someone who prefers steak and potatoes to a sushi restaurant ? Can't we predict that they will have the same preferences tomorrow night? Yes, and so can computer algorithms, but Spatial Computing systems could soon know a lot more about us than even our best friends do, since they could watch every interaction, every product touched, every music song picked, and every movie watched. In Chapter 10, The Always Predicted World, we show how that data will be used in each of our seven disruptable industries to serve users in radically new ways.

Meeting the Pioneers

We predict that you will enjoy learning more about the seven people who are pushing Spatial Computing further in Chapter 11, Spatial Computing World-Makers. Instead of ratcheting through a list, we'd like to tell you why we picked the people here. One spends time in retail stores the world over and uses technology to help them become not only more profitable, but more customer-centric. Another developed Google's autonomous vehicle technology and has gone on to further build out a huge vision of the future of transportation. You'll meet one leader who, from their perch at Qualcomm, sees literally every new product coming before the rest of us do. Also on the list are a couple of investors, one East Coast, one West, who are pouring resources into entrepreneurs who are bringing us the future of virtual beings, robots and the AI that runs them. Finally, we have a doctor who is pushing the healthcare system forward into a world of Augmented and Virtual Reality and a successful innovator who builds companies that have immersiveness and VR at their core. We picked them out of the thousands that we've studied because they represent a guiding hand that will bring "superpowers" to us all.

Thinking Ahead

With these new superpower-like capabilities come responsibilities that the companies and organizations who create and use Spatial Computing owe to the people who use their technologies. New technologies bring with them new ways of doing things, and the more significant the change, the more unchartered and unknown are the ramifications that occur as a result of their use. In Chapter 12, How Human?, we provide a philosophical framework put forth by L.A. Paul of Yale University in her book, Transformative Experience, that explains why human beings tend to have cognitive issues with radical new technologies. We then discuss recent issues regarding privacy, security, identity, and ownership, and how they relate to Spatial Computing. Finally, we take up how Spatial Computing technologies can be utilized to bring about human social good.

Starting the Journey

We wrote The Infinite Retina for a wide audience of non-technical people, rather than for engineers. By the end of the book, you should understand the technologies, companies, and people who are changing computing from being something you do while sitting and facing a flat computer screen or while holding a phone, to computing that you could move through three-dimensionally. We focus on how Spatial Computing could be used by enterprises and effectively radically change the way human beings learn from information and visuals and understand their world. We very strongly believe that enterprise use of Spatial Computing will lead to massive consumer use, and we are excited to share our learning in this book with you.

Part I

Why Spatial Computing and Why Now?

The Significance of Spatial Computing

1

Prime Directive

A perfect storm of change will arrive during the 2020s. We predict that we will see computing switch from staring into a four-inch piece of glass in our hands to using devices on our face that bring computing to every surface. Along with that, we will see advances such as vehicles moving without humans in them for the first time. A fourth paradigm of the personal computing age is upon us, Spatial Computing, and it is one that truly makes personal computers even more personal.

With the coming of Spatial Computing and its two purest members, Virtual and Augmented Reality, we believe that businesses and human cooperation need to be aimed at one thing: working together to build complex technologies to keep us around on this planet longer and in a more satisfied and productive state, while paying attention to the effects that these technologies have on ourselves and the planet. Spatial Computing in the 2020s will see immense challenges, but great opportunities will be available for brands to use new technologies for combined social good. Just what is it about Spatial Computing that is making human beings crave it?

At stadiums around the world, they are getting ready for it. We visited a place in Las Vegas because of it. Hollywood is getting ready for it. We visited a huge studio in Manhattan Beach, California, to see it. Investors and entrepreneurs are getting ready for it. We visited several in Seattle, New York, Los Angeles, and Silicon Valley. Hospitals, shopping malls, hotels, cities, automotive companies, power companies, banks, and more are investing in it. Tens of billions of dollars are being invested in it by both big and small tech companies.

That "it," which is Spatial Computing, is computing that you, virtual beings, or robots can move around in. It includes all the software and technology needed to move around in a digital 3D world.

That is software and technology associated with AI, including Machine Learning and Natural Language Processing, Computer Vision, Augmented Reality, Virtual Reality, and all other apps that support the creation and maintenance of a digital 3D world. We see great strides that will be made in Spatial Computing uses for many industry verticals, including Technology, Media, and Telecommunications (TMT), Transportation, Manufacturing, Retail, Finance, Healthcare, and Education.

Before we dig into everything that's happening that caused us to write The Infinite Retina, let's back up and think about the Prime Directive that is driving billions of dollars in human effort into Spatial Computing. Why do human beings need robots delivering food and building things in factories, and why do we need Spatial Computing devices on our faces so that we can work, entertain ourselves, educate ourselves, and collaborate with each other in new ways?

What is the Prime Directive? Does it have something to do with why humans spend more and more on technology or tools every year? Are any new trends, like our changing understanding of climate change, causing this change? Does culture itself change in a major way because of the Prime Directive?

Photo credit: Robert Scoble. Attendees at the 2019 Game Developers Conference in San Francisco use a Magic Leap Spatial Computing headset.

What Makes Us Human?

Human beings are classified as Homo sapiens, which in Latin means "knowing man." Modern Homo sapiens are believed to have appeared a little over 300,000 years ago. The distinction between Homo sapiens and what came before has to do with the relatively sophisticated use of tools―tools that were used to survive more efficiently and with which humans gained control of their surroundings. Tools were also used by early humans to make art on cave walls and carve statuettes of female fertility goddesses. The tools served as augmenting devices―augmenting humans' chances of survival and also of expression.

With modern humans, this augmentation can take the form of education, which in turn is used to gain knowledge. With knowledge, our chances of survival should be better. In many ways, our Prime Directive is to know how to better survive and how to better express ourselves by willfully creating and using tools for those purposes. It is a dual directive, for it cannot be proven that one gives rise to the other, but rather both are mutually beneficial. And it is for both very practical and expressive reasons that tools have continued to be created from the time of early man to today. An example of a human being's ingenuity that traverses both the practical and the expressive are the iterative inventions of the writing "pen and paper" combination. This combination tool, which goes back millennia, started out with cave walls, some form of patchworked dried grasses, as well as stone, serving as the "paper" and natural dye and sturdy reed, as well as a stone or metal chisel, serving as the "pen." "Pen and paper" has been used to record both business and legal matters, as well as nonfictional and fictional narrative, and poetry, as well as visual art, such as paintings, when the "pen" is conceived as pigments. With the advent of the typewriter, there was even more of a separation between the practical and textually expressive and the visually expressive. A machine, the typewriter, was then replaced by the word processor and then the computer. And here we are these days utilizing our computers and their smaller counterparts―the smartphone. Computers did not only replace typewriters; they are also in the process of causing people to question the continued existence of physical books and newspapers, as well as movie theaters.

Our Prime Directive to know how to better survive and how to better express ourselves now has a new channel―Spatial Computing. With Spatial Computing, the uses of the technologies of Virtual Reality, Augmented Reality, and Artificial Intelligence eclipse those of the computer we know today. In the near future, we will no longer have to use a physical computer to do our work and browse the internet. And we will be able to do so much more with the three-dimensionality of Spatial Computing and speech recognition software. It turns out that our need to better express ourselves appears to include a need to experience a replicated reality.

Replicating of reality in the forms of paintings, fiction, and films, as well as other forms, has existed as long as human beings have had the need to express the conditions of both their individual and social existence in an effort to better understand themselves. Experiencing a replicated reality also turns out to be a very good way to achieve a new skill and to get knowledge in general. Spatial Computing is the next generation of imaging that is able to replicate reality, allowing the movement from two-dimensional imaging to three-dimensional. With three-dimensional imaging, the replication of reality is able to be more closely related to the reality it is trying to represent.

Human beings seem to get satisfaction out of presenting and experiencing narratives that have the appearance of being real. An example of this is a movie. It is difficult to say exactly why we get such pleasure out of viewing a "good" movie. Perhaps it is empathy, but the question still remains why empathizing with movie characters that appear to be real should make us feel good, much less entertained. With Spatial Computing, the visuals are even more true-to-life and we are able to move through them (Virtual Reality) or incorporate and manipulate non-real objects into our real world (Augmented Reality). Artificial Intelligence adds another layer to the existing reality by organizing previously unconnected data into meaningful systems that could then be utilized in Spatial Computing to feed our Prime Directive needs.

Photo credit: Robert Scoble. Here, you can see the slums and other residential buildings as seen from the Four Seasons luxury hotel in Mumbai, India. Billions of people live in similar accommodations around the world and they will experience the world far away soon in Spatial Computing devices.

Drivers and Benefits

The benefits of Spatial Computing play right into our Prime Directive of knowing how to better survive and how to better express ourselves through our creation and use of tools. Our need to have replicated three-dimensional worlds and objects in order to master our understanding and manner of expression is one that could be served by the software and technologies of Augmented Reality, Virtual Reality, and Artificial Intelligence.

Noted investor and Netscape founder Marc Andreessen has told markets he has a contrarian view to Silicon Valley's belief that Augmented Reality represents a better investment than Virtual Reality. He noted that it is a privileged view―that in Silicon Valley, residents have tons of beautiful places within an hour's drive, from beaches to vineyards. Most people in the world, he said, don't have those advantages.

Walk through neighborhoods, even middle-class ones, and you will see millions living in small homes in high rises. Telling them that they will want to wear computing devices while walking through a beautiful area won't be hard. Instead, Andreessen sees a world where people will wear headsets to visit the natural beauty well out of reach somewhere else in the world. Even in the United States, only about 20 percent own a passport, so asking them to visit historic sites in, say, Egypt or Israel, won't be possible for most. We can, instead, take them there with Spatial Computing.

However, unlike Andreessen, we don't view the question of whether Virtual Reality or Augmented Reality will be the "winner" between the two. We see that, by the mid-2020s, our Spatial Computing devices will let you float between putting virtual things on top of, or replacing things, in the real world.

Where we are going with this argument is that the hunger for these devices, services, technologies, and experiences that Spatial Computing affords will probably be far greater among the billions who can't afford a private jet to fly them to a Davos, Switzerland ski vacation, or even afford a Tesla for a weekend jaunt to Napa or Yosemite. That seems to be Andreessen's greater point; that the investment opportunity here is grand because it not only will improve the lives of billions, but may lead to us saving ourselves with new education and new approaches to living, including being able to take courses using Virtual Reality and "travel" to different locations in the world without having to jump on an airplane.

However, an argument could be made that human beings are social―our social natures have aided us greatly in our need to survive and thrive and that Spatial Computing is too isolating. With Spatial Computing, though, we could choose to experience and learn something solo or networked with others.

Spatial Computing on its own can serve as the medium to interface with ideas, locations, processes, people, and AI characters, or as Betaworks' John Borthwick likes to call them, synthetic characters. These synthetic characters will replace the Machine Learning text bots that are currently ubiquitous―in effect, putting a three-dimensional body to the words. The benefit of this is that we will feel like we are engaging with a real being that feeds our social natures.

Along these same lines, entertainment that utilizes Spatial Computing will make it even more true-to-life. Characters in these new kinds of narratives will be more real to us, allowing us to gain even more insights into the human condition. Spatial Computing is a major innovation; the latest in a long line of ideas and inventions aimed at improving the human condition. It makes our lives better, bringing knowledge to us faster and with less expense overall.

Potential Dangers

It could be said that with great knowledge comes great responsibility. There are several associated potential dangers that come with our use of Spatial Computing. Areas we will touch upon here include potential loss of personal control, dilution of the natural, and population segmentation.

In terms of potential loss of personal control, the major one that everyone has at the front of their minds because of Facebook's transgressions is the disappearance of privacy. Especially with Augmented Reality, unauthorized use of data and media by companies or authoritarian use by governments could potentially be a problem. Location and spending data, along with videos made with the knowledge of the viewer and recording speech when headsets/glasses are worn could present another wave of Google Glass-like uproar. However, we do not think this is going to happen since the uses of Augmented Reality have been lauded over the last few years and there also seems to be better advance acceptance of glasses-like headsets due to the public's understanding that privacy issues will be addressed by Augmented Reality hardware and software companies, along with much deeper utilities that today's technology affords. Companies will need to be especially clear as to what their data policies are and have appropriate opt-in policies to meet the expectations of the public. There is so much reward that could be received by providing data for Augmented Reality purposes―these rewards should be heralded while understanding that there are those that would prefer to not share their data.

Another potential loss of personal control is one of over-advertising. As Keiichi Matsuda's 2016 six-minute nightmare concept film "Hyper-Reality" portrayed, a world where Augmented Reality advertising is constantly overlaid over the real world is one that is unbearable.

Having an opt-in system should solve this, but it might still be an issue when there is a willful exchange of some kind of visual goods with advertising. Companies will probably push the advertising threshold with potential consumers to see how far they can go in this case.

Talking about extremes, this brings up the point of addiction that has come so naturally with the advent of the smartphone. And as with any other kind of addiction, it is certainly the case that personal loss comes attached with it. There are many cases of people who have died while taking selfies while in the throes of being distracted by their digital addictions. There is the possibility that a person walking around in a future blockbuster Augmented Reality game will walk into traffic or even off a cliff, so technical safeguards based in Computer Vision will definitely have to be built-in as an alert device.

Along with this kind of addiction comes dilution of the significance of the natural objects and environments of the world as they appear in reality. This is relevant with regard to both Augmented and Virtual Reality. Dystopian visions on this abound, ranging from people never wanting to be in the real world again, to the death of learning, to the abandonment of care about pollution and global warming. In Virtual Reality, a person could potentially hurt and even kill a digital character without having the full effect of what these actions would be like in real life. A worry might be that these actions via Augmented and Virtual Reality could become so common stance that the line between the imaginary and the real could become blurred to the extent that real people would then get hurt and killed. Industry oversight organizations might spring up to create ratings of Spatial Computing experiences that would rate the level of violent content so that viewings these experiences could be better managed and controlled. In this way, possible negative societal effects could be mitigated.