102,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Serie: Wiley Series in Display Technology
- Sprache: Englisch
A COMPREHENSIVE AND AUTHORITATIVE GUIDE TO THE FOUNDATIONS, TECHNOLOGIES, AND TRANSFORMATIVE APPLICATIONS OF VIRTUAL AND AUGMENTED REALITY.
Virtual and Augmented Reality: Fundamentals and Applications presents an in-depth exploration of the core principles, enabling technologies, and real-world use cases of VR and AR. Designed for use as both a textbook and a professional reference, it offers a rigorous and structured foundation for students, researchers, engineers, and industry practitioners alike.
Grounded in neuroscience and computational perception, the text takes readers from first principles to advanced topics in immersive system architecture and experience design. It emphasizes human-centered systems and multisensory engagement, covering vision, hearing, balance, touch, and smell, along with natural interaction modalities such as gesture, voice, gaze, facial expressions, and direct brain interfaces.
Key features include:
- Foundations of human sensory perception and their technological counterparts in immersive systems
- Advances in 3D graphics, high-fidelity display technologies, spatial audio, computer vision, and motion tracking
- Multimodal interaction and semantic understanding powered by artificial intelligence
- System design and usability considerations for end-to-end VR and AR platforms
- Case studies spanning entertainment, healthcare, education, workforce training, virtual tourism, e-commerce, and more
Authored by a leading innovator in immersive technologies, this book is ideal for advanced instruction and serves as an essential reference for those shaping the future of virtual and augmented reality.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 1355
Veröffentlichungsjahr: 2026
Ähnliche
Table of Contents
Cover
Table of Contents
Series Page
Title Page
Copyright Page
Series Editor’s Foreword
About the Author
Preface
Acknowledgments
Section 1: Overview and Evolution
1 Overview of Virtual and Augmented Reality: Toward Immersive and Interactive Experiences
1.1 Introduction
1.2 Key Concepts in Immersive Technologies
1.3 Multisensory Immersion and Interaction
1.4 Goals of the Book
1.5 Overview of the Structure and Content
1.6 Conclusion
References
2 The Evolution of Virtual and Augmented Reality: Accomplishments and Challenges
2.1 Introduction
2.2 Virtual and Augmented Reality in Science Fiction
2.3 The Early Concepts Through to the 1950s and 1960s
2.4 The 1980s and 1990s: Commercial Exploration
2.5 The 2000s: Expansion into New Fields
2.6 The 2010s: Technological Innovations and Mainstream Adoption
2.7 The 2020s: Integration, Convergence, and Pushing the Frontiers
2.8 Challenges and Limitations
2.9 Summary
References
Section 2: Sensory‐Perceptual Immersion
3 Visual Immersion: Foundations of Human Vision
3.1 Introduction
3.2 Anatomy of the Human Eye
3.3 Visual Information Processing in the Brain
3.4 Field of View (FOV) for Human Vision
3.5 Perception of Depth and Space
3.6 Motion Perception
3.7 Color Vision
3.8 Brightness and Contrast Perception
3.9 Summary
References
4 Visual Immersion: Displays and Optics
4.1 Introduction
4.2 Display System Configurations
4.3 Key Attributes of Displays for VR and AR
4.4 Display Technologies
4.5 Optics for VR and AR Systems
4.6 Advanced Topics and Future Trends
4.7 Summary
References
5 Visual Immersion: Creating Virtual Worlds with 3D Graphics
5.1 Introduction
5.2 Fundamentals of Graphics in VR and AR
5.3 Rendering Pipeline
5.4 Animation and Motion
5.5 Advanced 3D Graphics Techniques
5.6 Computational Hardware and Software Tools
5.7 Future Trends and Innovations
5.8 Summary
References
6 Visual Immersion: Understanding and Augmenting the World with Computer Vision
6.1 Introduction
6.2 Fundamentals of Computer Vision
6.3 3D Reconstruction
6.4 Augmenting the Reality: Overlaying Virtual on Real
6.5 Summary
References
7 Auditory Immersion: Spatial Sound and 3D Audio
7.1 Introduction
7.2 Basic Principles of Sound Waves
7.3 Fundamentals of Human Auditory Perception
7.4 Sound Technologies in Virtual and Augmented Reality
7.5 Summary
References
8 Vestibular Immersion: Motion Sensing and Stimulation
8.1 Introduction
8.2 The Human Vestibular System
8.3 Creating Vestibular Immersion
8.4 Motion Tracking Techniques
8.5 Technologies for Vestibular Stimulation
8.6 Summary
References
9 Somatosensory Immersion: Touch and Haptic Feedback
9.1 Introduction
9.2 Physiology of Somatosensory Perception
9.3 Haptic Feedback for Immersive Systems
9.4 Designing Haptic Feedback Systems
9.5 Applications of Haptic Feedback Technologies
9.6 Summary
References
10 Olfactory Immersion: Smell Simulation
10.1 Introduction
10.2 Basics of Olfactory Perception
10.3 Enhancing Virtual Experiences with Olfactory Feedback
10.4 User Experience Considerations
10.5 Future Directions and Innovations
10.6 Summary
References
11 Multisensory Integration: Creating Unified Immersive Experiences
11.1 Introduction
11.2 Foundations of Multisensory Integration
11.3 Challenges in Multisensory Integration
11.4 Synchronization Across Modalities
11.5 Applications of Multisensory Integration
11.6 Summary
References
Section 3: Human Inputs and Interactions
12 Hand Gesture Recognition: Intuitive Interfaces
12.1 Introduction
12.2 Physiology of the Human Hand and Gesture Categories
12.3 Fundamentals of Hand Gesture Tracking
12.4 Gesture Tracking System Architectures
12.5 Key Components of Gesture Tracking Systems
12.6 Gesture Recognition Algorithms
12.7 Applications of Hand Gesture Tracking
12.8 Challenges and Limitations
12.9 Future Trends and Research Directions
12.10 Summary
References
13 Eye‐Gaze Tracking: Enabling Adaptive Interactions
13.1 Introduction
13.2 Physiology of Eye Movement
13.3 Types of Eye‐Gaze Tracking Systems
13.4 Technologies and Methods
13.5 Calibration and Accuracy
13.6 Integration with VR and AR Systems
13.7 Applications of Eye‐Gaze Tracking in VR and AR
13.8 Challenges and Limitations
13.9 Future Trends and Research Directions
13.10 Summary
References
14 Speech Recognition: Voice as a Natural Input Modality
14.1 Introduction
14.2 Fundamentals of Voice Recognition
14.3 Types of Voice Recognition Systems
14.4 Technologies and Methods
14.5 Integration with VR and AR Systems
14.6 Applications of Speech Recognition in VR and AR
14.7 Challenges and Limitations
14.8 Future Directions
14.9 Summary
References
15 Facial Expression Recognition: Avatars and Emotional Presence
15.1 Introduction
15.2 Physiology of the Human Facial Expression
15.3 Technologies for Facial Expression Tracking
15.4 Real‐Time Expression Rendering in Avatars
15.5 Personalization and User Representation
15.6 Impact on Immersion and Presence
15.7 Challenges and Limitations
15.8 Summary
References
16 Brain–Computer Interfaces: Direct Neural Interactions
16.1 Introduction
16.2 BCI Principles and Systems
16.3 Hardware Components
16.4 Software Components
16.5 BCI Applications
16.6 Challenges and Limitations
16.7 Future Trends and Research Directions
16.8 Summary
References
17 Multimodal Interactions: Bridging Inputs for Holistic Experiences
17.1 Introduction
17.2 The Case for Multimodal Interactions
17.3 Principles of Multimodal System Design
17.4 Combining Interaction Modalities
17.5 Key Technologies for Multimodal Systems
17.6 Applications of Multimodal Interactions
17.7 Challenges in Multimodal Integration
17.8 Future Directions
17.9 Summary
References
Section 4: Systems and Applications
18 System Architectures and Designs: Integration Challenges
18.1 Introduction
18.2 Integration Challenges and Usability Considerations
18.3 Designing for Scalability and Collaboration
18.4 Case Studies in System Integration
18.5 Summary
References
19 Applications of Virtual and Augmented Reality: Innovating Across Industries
19.1 Introduction
19.2 Healthcare and Medicine
19.3 Education and Training
19.4 Enterprise and Industrial Applications
19.5 Entertainment and Media
19.6 Retail and E‐Commerce
19.7 Military and Defense
19.8 Arts, Culture, and Tourism
19.9 Summary
References
Section 5: Reflections and Outlook
20 Accessibility and Inclusivity: Designing Safe and Equitable Systems
20.1 Introduction
20.2 Principles of Accessible and Inclusive Design
20.3 Accessibility Features in Virtual and Augmented Reality Systems
20.4 Inclusivity in Multicultural and Global Contexts
20.5 Ethical Considerations and User Safety
20.6 Summary
References
21 The Simulation Hypothesis: Are We Already Living in a Virtual World?
21.1 Introduction
21.2 Origins and Theoretical Foundations
21.3 Technological Feasibility
21.4 Supports and Counterarguments
21.5 Societal Implications
21.6 Cultural and Psychological Impact
21.7 Future Directions and Research
21.8 Summary
References
22 Toward the Future: Emerging Directions in Virtual and Augmented Reality
22.1 Introduction
22.2 Technology Trends Shaping the Future
22.3 Expanding Applications
22.4 Human‐Centric Designs
22.5 Challenges and Research Opportunities
22.6 The Broader Societal and Cultural Impact
22.7 A Vision for the Future
22.8 Summary
Index
End User License Agreement
List of Illustrations
Chapter 2
Figure 2.1
Ready Player One
(2018), directed by Steven Spielberg, portrays a f...
Figure 2.2
Oliver Wendell Holmes' 1859 redesign of the stereoscope simplified ...
Figure 2.3
Morton Heilig's Sensorama, introduced in the early 1960s, is consid...
Figure 2.4
The Sword of Damocles, developed by Ivan Sutherland and Bob Sproull...
Figure 2.5
NASA Ames Virtual Interface Environment Workstation (VIEW), 1985, f...
Figure 2.6
The DataGlove, developed by VPL Research, was an early VR device th...
Figure 2.7
Nintendo Virtual Boy and Red Alarm Gameplay. The 1995 Virtual Boy u...
Figure 2.8
Released in 2013, Google Glass was one of the first major attempts ...
Figure 2.9
Microsoft HoloLens was a pioneering mixed reality device, blending ...
Figure 2.10
The HTC Vive Lighthouse Tracking system used a series of base stat...
Figure 2.11
Intel's Project Alloy, introduced in 2016, revolutionized virtual ...
Figure 2.12
The Magic Leap One offered a mixed reality experience, using waveg...
Figure 2.13
The Meta Quest series revolutionized consumer VR with its untether...
Figure 2.14
Apple Vision Pro revolutionized spatial computing by blending digi...
Figure 2.15
The Meta Orion prototype aims to deliver an immersive AR experienc...
Chapter 3
Figure 3.1
Schematic diagram of the human eye, showing the key anatomical stru...
Figure 3.2
Layers of photoreceptors and nerve cells in the retina (Betts et al...
Figure 3.3
Distribution of rod and cone photoreceptors in the human retina, il...
Figure 3.4
The human visual pathway starts with the retina, which captures inc...
Figure 3.5
Visual processing pathways in the brain (Yang et al., 2022). Visual...
Figure 3.6
Human field of vision, illustrating different regions of the visual...
Figure 3.7
Various depth perception cues, including both binocular and monocul...
Figure 3.8
Depth contrast sensitivity across different spatial ranges (Cutting...
Figure 3.9
Stereopsis based on binocular disparity. This stereoscopic card fro...
Figure 3.10
Monocular depth cues, such as perspective, relative size, occlusio...
Figure 3.11
Normalized spectral response of human cone photoreceptors. This gr...
Figure 3.12
Log‐Log plot of spatial contrast sensitivity functions for luminan...
Chapter 4
Figure 4.1
Virtual and augmented display architectures for 3D object visualiza...
Figure 4.2
The Cave Automatic Virtual Environment at the Electronic Visualizat...
Figure 4.3
The use of augmented reality on a smartphone, where digital element...
Figure 4.4
Layered structure of a color TFT LCD. (1) Glass plates provide supp...
Figure 4.5
A top‐emission OLED display architecture with micro‐cavity structur...
Figure 4.6
Optical microscope images of an active‐matrix microLED display (Dur...
Figure 4.7
Digital micromirror device structure and chip (Allen, 2017). The di...
Figure 4.8
System configuration for a two‐dimensional RGB laser beam scanning ...
Figure 4.9
Examples of geometric optical designs for augmented reality combine...
Figure 4.10
Diffractive waveguide‐based augmented reality near‐eye display (Xi...
Figure 4.11
Comparison of natural vision vs. VR headset vision (Daler, 2022). ...
Figure 4.12
Demonstration of a liquid crystal lens with adjustable focal dista...
Figure 4.13
Basic architecture of a near‐eye light field display utilizing a m...
Chapter 5
Figure 5.1
Illustration of the unprecedented progress in graphics processing t...
Figure 5.2
Illustration of global (world) and local (object) coordinate system...
Figure 5.3
Example of polygonal modeling. This illustration of a dolphin demon...
Figure 5.4
Illustration of the six degrees of freedom in virtual and augmented...
Figure 5.5
Visualization of a view (camera) transformation in a 3D scene (Mari...
Figure 5.6
Illustration of the concept of clipping of primitives (Marie, 2008)...
Figure 5.7
Illustration of window‐viewport transformation (Marie, 2008). The 3...
Figure 5.8
Diagram illustrating the vectors used in defining the Bidirectional...
Figure 5.9
Diagram illustrating Lambertian diffuse reflection (GianniG46, 2011...
Figure 5.10
Comparison of Flat Shading and Gouraud Shading. The flat shading (...
Figure 5.11
The three components of the Phong model: Ambient, Diffuse, and Spe...
Figure 5.12
The visual differences between the Blinn‐Phong (left) and Phong (r...
Figure 5.13
Process of texture mapping a 2D image onto a 3D model. The top‐lef...
Figure 5.14
Illustration of hand posing using joints and control handles in a ...
Figure 5.15
Visualization of particle emission from a cube. Left: The cube emi...
Figure 5.16
Illustration of foveated rendering technique. This image demonstra...
Chapter 6
Figure 6.1
A pair of RGB‐D images captured using a RealSense 3D camera (Bhowmi...
Figure 6.2
Illustration of depth estimation using a stereo camera system with ...
Figure 6.3
Examples of lens distortion types in images. Left: no distortion, s...
Figure 6.4
Representation of the RGB color model in a cubic form, where each a...
Figure 6.5
Comparison of HSL (a–d) and HSV (e–h) color models (Rus, 2020). (a,...
Figure 6.6
Illustration of a typical convolutional neural network (CNN) archit...
Figure 6.7
Example of a convolution operation on a single‐channel padded image...
Figure 6.8
A Vision Transformer architecture for image classification and loca...
Figure 6.9
A dense 3D reconstruction of an office environment created using a ...
Figure 6.10
Integration of real‐world elements into a virtual environment usin...
Figure 6.11
Example of real‐world augmentation with 3D virtual objects. RealSe...
Chapter 7
Figure 7.1
Representation of a sound wave, illustrating its key characteristic...
Figure 7.2
Anatomy of the human ear, depicting the key elements that enable us...
Figure 7.3
Cross‐sectional diagram of the cochlea showing the scala vestibuli,...
Figure 7.4
Lateral view of the human brain showing the exposed primary auditor...
Figure 7.5
Illustration of a spherical coordinate system to represent spatial ...
Figure 7.6
Fundamental binaural cues for auditory localization: (a) Interaural...
Figure 7.7
Illustration of the contribution of the outer ear (pinna) to HRTF. ...
Figure 7.8
The primary elements of sound localization: timing, loudness, and t...
Figure 7.9
The dynamic range of human hearing illustrated through frequency an...
Figure 7.10
Schematic of 3D spatial audio computation using head‐related impul...
Chapter 8
Figure 8.1
Key components of the vestibular system responsible for balance and...
Figure 8.2
Representation of the six degrees of freedom (6‐DOF) system, encomp...
Figure 8.3
Diagram of a MEMS accelerometer, illustrating its working principle...
Figure 8.4
Architecture of a vibrating gyroscope consisting of a single proof ...
Figure 8.5
Schematic view of a resonant magnetic field sensor utilizing movabl...
Figure 8.6
Comparison of inside–out tracking (a) and outside–in tracking (b) i...
Figure 8.7
Example of a motion simulator chair designed to enhance virtual rea...
Figure 8.8
Two examples of omnidirectional treadmills designed for virtual rea...
Figure 8.9
An example of VR shoes, which simulate natural walking in virtual e...
Figure 8.10
Illustration of galvanic vestibular stimulation (GVS) and its effe...
Figure 8.11
Block diagram of a Proportional‐Integral‐Derivative (PID) controll...
Chapter 9
Figure 9.1
Somatosensory pathways from the peripheral nervous system to the co...
Figure 9.2
Mean two‐point discrimination (2PD) thresholds measured for pain an...
Figure 9.3
Illustrative examples of skin‐integrated haptic interfaces for virt...
Figure 9.4
A haptic‐feedback smart glove with integrated piezoelectric actuato...
Chapter 10
Figure 10.1
Anatomy of the human olfactory system.(a) The nasal cavity gui...
Figure 10.2
Concept of virtual reality applications of olfactory interfaces, l...
Figure 10.3
A virtual olfactory generation system, utilizing a bionic fibrous ...
Figure 10.4
A study design demonstrating the impact of integrating olfactory s...
Chapter 11
Figure 11.1
Diagram of the human brain illustrating key cortical areas involve...
Figure 11.2
Cross‐modal processing areas in the human brain (Calvert et al., 2...
Figure 11.3
An illustration of the ventriloquism effect, where visual input do...
Figure 11.4
The vestibulo-ocular reflex mechanism (Häggström, 2007). When head...
Figure 11.5
Multimodal synchronization in the balancing process (Zampogna et a...
Chapter 12
Figure 12.1
Illustration of the anatomy of the human hand and wrist bones, hig...
Figure 12.2
Examples of static and dynamic hand gestures along with visual det...
Figure 12.3
An early demonstration of natural hand interactions using computer...
Figure 12.4
A VPL DataGlove connected to a Macintosh computer, with the outer ...
Figure 12.5
RGB and depth images captured with RealSense 3D camera (Bhowmik, 2...
Figure 12.6
Different representations of a hand for gesture recognition system...
Figure 12.7
A labeled dataset for American Sign Language (ASL) gestures (Laksh...
Chapter 13
Figure 13.1
Diagram illustrating the vestibulo‐ocular reflex (Häggström, 2007)...
Figure 13.2
An example of fixations and saccades during reading, illustrating ...
Figure 13.3
Lateral and anterior views of the extraocular muscles of the right...
Figure 13.4
Eye movement patterns in the study of Yarbus in 1967 demonstrating...
Figure 13.5
Schematic diagram of the four Purkinje images (P1, P2, P3, P4) cre...
Figure 13.6
Illustration of the “dark pupil effect” and “bright pupil effect” ...
Figure 13.7
Electrooculograms for the left eye (LEOG) and the right eye (REOG)...
Figure 13.8
An example of an eye‐tracking system integrated into a virtual rea...
Chapter 14
Figure 14.1
Representation of sound as a mixture of sinusoidal waves with vary...
Figure 14.2
Spectrogram of the spoken words “nineteenth century.” The vertical...
Figure 14.3
Anatomy of the human speech articulatory system, highlighting the ...
Figure 14.4
Diagram of a one‐unit recurrent neural network (Deloche, 2017). Th...
Figure 14.5
Diagram of a one‐unit long short‐term memory network (Deloche, 201...
Figure 14.6
Diagram of a one‐unit gated recurrent unit (Deloche, 2017). The GR...
Figure 14.7
Diagram of the Transformer model architecture, showing its main co...
Chapter 15
Figure 15.1
Experimental images of facial expressions induced by electrical st...
Figure 15.2
Side view of the muscles of facial expression (Betts, 2013). This ...
Figure 15.3
Examples of modeled facial configurations for 13 emotion categorie...
Figure 15.4
A diverse set of digital avatars generated using the Meta Avatar S...
Figure 15.5
Avatars can vary widely in fidelity, from simplified, abstract rep...
Figure 15.6
A pipeline for personalized avatar generation (Ding et al., 2024)....
Chapter 16
Figure 16.1
A wireless EEG headset.
Figure 16.2
An example of EEG signal decomposition into its constituent freque...
Figure 16.3
Common artifacts in human EEG recordings (Cherninskyi, 2015). (1) ...
Figure 16.4
Intracranial electrode grid used for electrocorticography (ECoG), ...
Figure 16.5
The Utah electrode array (Normann & Fernandez, 2016). (a) It featu...
Figure 16.6
Typical flow of BCI software stages for signal analysis, in this e...
Figure 16.7
Two applications of hybrid brain–machine interfaces (Nicolelis, 20...
Chapter 17
Figure 17.1
A conceptual model of a multimodal interaction system illustrating...
Figure 17.2
An example of multimodal interaction in immersive environments. Th...
Chapter 18
Figure 18.1
Latency and motion prediction in virtual systems (Warburton et al....
Figure 18.2
Histogram of interpupillary distances (in millimeters) for 3,976 i...
Figure 18.3
Accurate alignment between the optical system of a head‐mounted di...
Figure 18.4
Illustration of various viewing postures tested for neck joint tor...
Chapter 19
Figure 19.1
The Stanford virtual heart project uses immersive VR to revolution...
Figure 19.2
A virtual classroom facilitating a problem‐solving lesson (Davis e...
Figure 19.3
Augmented reality applications provide industry technicians with v...
Figure 19.4
Virtual and augmented reality transform entertainment with immersi...
Figure 19.5
An example of a mixed reality game, Job Simulator by Owlchemy Labs...
Figure 19.6
IKEA’s AR app allows users to visualize virtual furniture in their...
Figure 19.7
Military and defense sectors are adopting virtual and augmented re...
Figure 19.8
Virtual reality flight training solution utilizing a dynamic six d...
Figure 19.9
Example of a virtual tour of natural parks in Chile (Stappung et a...
Chapter 20
Figure 20.1
Percentage of participants reporting various access barriers in da...
Figure 20.2
Pure‐tone air‐conduction hearing thresholds by age and sex for Jap...
Figure 20.3
An example of a task-based virtual environment designed for biomet...
Chapter 21
Figure 21.1
A scene from The Matrix illustrating the concept of reality as a d...
Guide
Cover Page
Table of Contents
Series Page
Title Page
Copyright Page
Series Editor’s Foreword
About the Author
Preface
Acknowledgments
Begin Reading
Index
Wiley End User License Agreement
Pages
ii
iii
iv
v
vi
xxiii
xxv
xxvii
xxviii
xxix
1
3
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
57
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
525
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
587
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
615
616
617
618
619
620
621
622
623
624
625
627
628
629
630
631
632
633
634
635
636
637
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
Wiley ‐ SID Series in Display Technology
Series Editor: Susan K. Jones
Achintya K. BhowmikVirtual and Augmented Reality: Fundamentals and Applications
Paul Alivisatos, Eunjoo Jang, Ruiqing Ma (Eds.)Quantum Dot Display Science and Technology
2025
Phil Green (Ed)Fundamentals and Applications of Colour Engineering
Darran R. Cairns, Dirk J. Broer, Gregory P. Crawford (Eds.)Flexible Flat Panel Displays
2023
Bo‐Ru Yang (Ed.)E‐Paper Displays
Hideo Hosono, Hideya Kumomi (Eds.)Amorphous Oxide Semiconductors: IGZO and Related Materials for Display and Memory
Ernst Lueder, Peter Knoll, Seung Hee LeeLiquid Crystal Displays: Addressing Schemes and Electro‐Optical Effects, 3ed
2022
Jiun‐Haw Lee, David N. Liu, Shin‐Tson WuIntroduction to Flat Panel Displays 2ed
2020
Jun Souk, Shinji Morozumi, Fang‐Chen Luo, Ion BitaFlat Panel Display Manufacturing
2018
Takatoshi TsujimuraOLED Displays: Fundamentals and Applications 2ed
Shunpei Yamazaki, Tetsuo TsutsuiPhysics and Technology of Crystalline Oxide Semiconductor CAAC‐IGZO: Application to Displays
2017
Shunpei Yamazaki, Masahiro FujitaPhysics and Technology of Crystalline Oxide Semiconductor CAAC‐IGZO: Application to LSI
Noboru Kimizuka, Shunpei YamazakiPhysics and Technology of Crystalline Oxide Semiconductor CAAC‐IGZO: Fundamentals
2016
Dmitry A. Yakovlev, Vladimir G. Chigrinov, Hoi‐Sing KwokModeling and Optimization of LCD Optical Performance
2015
Achintya K. BhowmikInteractive Displays
Temkar N. RuckmongathanAddressing Techniques of Liquid Crystal Displays
Deng‐Ke Yang, Shin‐Tson WuFundamentals of Liquid Crystal Devices
2014
P. Bodrogi, T. Q. KhanIllumination, Color and Imaging: Evaluation and Optimization of Visual Displays
2012
Ernst Lueder3D Displays
Robert H. ChenLiquid Crystal Displays: Fundamental Physics and Technology
2011
Zhibing Ge, Shin‐Tson WuTransflective Liquid Crystal Displays
Ernst LuederLiquid Crystal Displays: Addressing Schemes and Electro‐Optical Effects 2e
2010
Shunsuke Kobayashi, Shigeo Mikoshiba, Sungkyoo Lim (Eds.)LCD Backlights
2009
Vladimir G. Chigrinov, Vladimir M. Kozenkov, Hoi‐Sing KwokPhotoalignment of Liquid Crystalline Materials: Physics and Applications
Mathew S. Brennesholtz, Edward H. StuppProjection Displays, Second Edition
Achintya K. Bhowmik, Zili Li, Philip Bos (Eds.)Mobile Displays: Technology and Applications
2008
David Armitage, Ian Underwood, Shin‐Tson WuIntroduction to Microdisplays
Shin‐Tson Wu, Deng‐Ke YangFundamentals of Liquid Crystal Devices
2006
Michael G. Robinson, Jianmin Chen, Gary D. SharpPolarization Engineering for LCD Projection
2005
Gheorghe BerbecelDigital Image Display: Algorithms and Implementation
2003
Robert L. MyersDisplay Interfaces: Fundamentals and Standards
Phil Green, Lindsay MacDonald (Eds.)Colour Engineering: Achieving Device Independent Colour
2002
Lindsay W. MacDonald, Anthony C. Lowe (Eds.)Display Systems: Design and Applications
1997
Virtual and Augmented Reality
Fundamentals and Applications
Achintya K. Bhowmik
This edition first published 2026© 2026 John Wiley & Sons Ltd
All rights reserved, including rights for text and data mining and training of artificial intelligence technologies or similar technologies. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Achintya K. Bhowmik to be identified as the author of this work has been asserted in accordance with law.
Registered OfficesJohn Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USAJohn Wiley & Sons Ltd, New Era House, 8 Oldlands Way, Bognor Regis, West Sussex, PO22 9NQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
The manufacturer’s authorized representative according to the EU General Product Safety Regulation is Wiley‐VCH GmbH, Boschstr. 12, 69469 Weinheim, Germany, e‐mail: [email protected].
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
Limit of Liability/Disclaimer of WarrantyWhile the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication Data applied for
Hardback ISBN: 9781119615927
Cover Design: WileyCover Image: © QtraxDzn/Shutterstock
Series Editor’s Foreword
Virtual and augmented reality (VR/AR) have moved rapidly from the margins of speculation into the center of technological innovation. No longer confined to research labs or experimental prototypes, immersive systems are reshaping how we learn, design, communicate, and heal. The convergence of advances in displays, computation, sensors, and human–computer interaction has made this a defining moment to examine both the scientific foundations and the transformative applications of immersive technologies.
Virtual and Augmented Reality: Fundamentals and Applications is a timely response to that need. It is not simply a survey of devices or software platforms, but a structured guide to how immersive technologies are conceived, built, and applied. By grounding the discussion in fundamentals—perception, interaction, and system design—while also exploring real‐world applications across industries, this book provides readers with both depth and breadth. It equips professionals and academics alike to understand the principles, evaluate current systems, and imagine what comes next.
The value of this volume lies in its balanced perspective. It does not treat VR and AR as isolated inventions but as part of a continuum in human–machine interaction, one that integrates multiple senses and modalities. At the same time, it does not shy away from the societal and ethical questions that arise as immersive systems move toward mainstream adoption. Issues of accessibility, inclusivity, user safety, and privacy are given attention alongside technical progress—a recognition that innovation carries responsibility as well as opportunity.
Dr. Achin Bhowmik brings a unique vantage point to this subject. His pioneering work in computational perception and immersive interfaces has advanced the field, while his teaching at Stanford and through international forums has shaped how new generations of researchers and practitioners approach these technologies. He has led the development of novel human‐computer interaction and mixed‐reality technologies, taught graduate‐level courses and seminars, and published well‐cited papers in the field.
For the reader, this book is both a foundation and an invitation. A foundation, because it consolidates decades of technical progress into a coherent framework of fundamentals and applications. An invitation, because it challenges us not only to study immersive systems but also to reflect on their broader implications and to imagine new directions. The future of VR and AR will be determined not only by engineering achievements but by the choices of the diverse community of thinkers, builders, and dreamers who shape them.
Susan K. JonesSeries Editor
About the Author
Achintya K. Bhowmik, Ph.D., is the Chief Technology Officer and Executive Vice President of Engineering at Starkey, where he leads the development of advanced hearing devices and human–technology interactions powered by artificial intelligence. He is also an Adjunct Professor at the Stanford University School of Medicine and an affiliate faculty member of both the Stanford Institute for Human‐Centered Artificial Intelligence and the Wu Tsai Neurosciences Institute. His research and professional interests span computational perception, artificial intelligence, and human sensory enhancement and augmentation technologies.
Previously, Dr. Bhowmik served as Vice President and General Manager of Perceptual Computing at Intel, where he created and led the RealSense product line, pioneering breakthroughs in 3D sensing, computer vision, and intelligent systems. He also serves on the boards of companies developing next‐generation sensing, microdisplay, and AI‐enabled technologies, as well as several academic and nonprofit organizations.
Dr. Bhowmik is a Fellow of the IEEE, the Society for Information Display, and the Asia‐Pacific Artificial Intelligence Association. He has authored more than 200 publications, including 3 books and over 80 patents worldwide. His work has been recognized with numerous honors for innovation in technology and engineering, including TIME’s Best Inventions, the Gold Globee Award for Most Innovative Person in Healthcare, and the Red Dot Design Award.
Preface
There’s something timeless about the human urge to create new realities. For centuries, we have imagined worlds beyond our own, sometimes through myths and legends, sometimes through philosophy, and more recently through science and technology. I have often wondered why immersive experiences hold such fascination for us. Perhaps it speaks to a deep part of human nature: the desire to reshape the world and to live within it in new ways.
Virtual reality (VR) and augmented reality (AR) technologies are part of this long story. What once lived mostly in fiction and imagination has become something tangible, something that is beginning to transform how we learn, communicate, heal, and create. When I first encountered early VR and AR systems, the imperfections were obvious, yet the potential was undeniable. Today, the pace of progress is extraordinary, and the possibilities ahead are even more profound. This book, Virtual and Augmented Reality: Fundamentals and Applications, is an attempt to capture both where we have come from and where we are heading. It is written for a wide audience: researchers, engineers, designers, students, and industry leaders, all of whom are shaping this rapidly evolving field.
Rather than treating VR and AR technologies as isolated inventions, the book follows the natural building blocks of immersive experience. It begins with historical and conceptual foundations, then explores how the senses, including vision, hearing, touch, balance, and smell, can be simulated and extended. From there, it examines the ways humans interact with virtual environments, using gestures, gaze, voice, facial expressions, and even direct brain–computer interfaces. Later chapters address system architectures, practical applications across industries, and the broader social, ethical, and philosophical questions that immersive technologies inevitably raise.
When I look back at my work in human-computer interactions and virtual and augmented reality, what stands out most is how much of it has revolved around a simple idea: helping machines sense and understand the world more like people do, and in turn enhancing and augmenting human sensory experiences. Computational perception became a kind of north star in that effort, though the journey has been anything but straightforward. Along the way, I had the chance to teach these evolving subjects at Stanford University and elsewhere, sometimes in structured academic courses, sometimes in the faster‐paced setting of seminars and conference workshops like those at the Society for Information Display’s Display Week events.
Teaching, surprisingly, taught me as much as working in the lab did. It forced me to explain complicated ideas simply, to audiences who didn't always have the same technical background. That experience mirrored what I saw happening in the field itself: VR and AR, once experimental playgrounds, began finding serious footholds across healthcare, education, entertainment, and more. The pace of change has been breathtaking and at times disorienting, marked by stunning technological advancements alongside products that fell short of expectations.
And yet, despite decades of relentless breakthroughs, it often feels as though we are still at the very beginning. As virtual and physical experiences blend more deeply, the basic questions grow harder. What should we be building? Why are we building it? Who benefits, and who might be left out? Issues like accessibility, inclusivity, mental well‐being, and user privacy aren't side conversations anymore; they have to be front and center. Innovation on its own isn't enough. What matters is how wisely, how thoughtfully, we choose to guide it so that it enhances and augments human experience.
I wrote this book hoping to share what we know about immersive systems and to invite readers to grapple with what these technologies might mean as they become woven into the fabric of everyday life, for better or for worse.
It is my hope that this book offers a foundation for understanding immersive systems, as well as an invitation to think more deeply about the possibilities they unlock and the responsibilities they demand. The journey forward will be shaped by a diverse community of thinkers, builders, artists, and dreamers. In their hands, the future of immersive reality will find its true meaning.
Acknowledgments
The creation of this book has been a long journey shaped by many generous people, thoughtful collaborators, and inspiring communities. I am deeply grateful to all who supported, challenged, and guided me along the way.
First, I want to thank my family, whose encouragement, understanding, and unwavering support made it possible to devote the time and energy needed to bring this book to life. I owe more to them than words can express.
I am grateful to the members of the RealSense and Project Alloy teams, an eclectic and immensely talented group I had the privilege of creating and leading, who shared a relentless passion for developing technologies that enable intelligent, natural human–computer interactions and immersive experiences. I am also thankful to the research and development team at Starkey, whose dedication, creativity, and commitment to improving human communication continue to inspire my work every day.
My thanks go to my students at Stanford University and elsewhere, whose curiosity and insightful questions continually pushed me to think more deeply about the future of computational perception and immersive technologies. Teaching these subjects in classrooms, seminars, and workshops helped refine many of the ideas that found their way into these chapters.
I would like to express my appreciation to the global community of engineers, scientists, designers, and artists whose research, insights, and breakthroughs have laid the foundations upon which this book builds. Many of the concepts discussed here stand on the shoulders of their remarkable contributions.
My sincere thanks to the editorial and production team at Wiley for their guidance, professionalism, and patience throughout the development of this project. I am also grateful to the members of the Society for Information Display, whose dedication to advancing the science and technology of visual systems, and with whom I had the honor of serving as president, helped shape many of the perspectives reflected in this work.
I am grateful to my mentors and collaborators who have shaped my thinking over the years. Their examples of intellectual integrity and creative exploration have left lasting marks on my approach to science and technology.
To everyone who contributed in ways large and small: thank you. This book is as much yours as it is mine.
Section 1Overview and Evolution
1Overview of Virtual and Augmented Reality: Toward Immersive and Interactive Experiences
1.1 Introduction
Welcome to a journey into the future of human–computer interactions, where the boundaries between reality and imagination blur, and our sensory perceptions are enhanced and augmented. In this first chapter of the book, we will present an overview of the key concepts in virtual reality (VR) and augmented reality (AR) technologies that promise to revolutionize the way we experience and interact with multimedia information and the world around us. As we explore the landscapes these technologies create, we will discover how VR and AR extend our digital capabilities and craft new dimensions of immersive sensory‐perceptual experiences. We will explore how these technologies can redefine our relationship with both the digital and physical worlds, offering unprecedented opportunities for innovation and creativity.
The evolution of VR and AR has been marked by significant advancements in transduction, computation, and communication technologies, which have collectively propelled these immersive and interactive experiences from conceptual ideas to practical applications. From gaming and entertainment to education, healthcare, and industry, the applications are vast and varied, promising to enhance every aspect of our lives. This chapter will provide an introduction to these transformative technologies, laying the groundwork for a deeper understanding of their principles, potential, and impact.
As we embark on this exploration, we will examine the technical foundations of virtual and augmented reality and consider the implications these technologies hold for the future. By understanding the mechanisms behind sensory‐perceptual immersion, we can appreciate how these technologies can create more engaging and realistic experiences, as well as the significant challenges associated with this endeavor.
We start by reviewing the key concepts and terminologies that are widely used in the field of immersive technologies. Then, we discuss the goals of the book and present an overview of the structure and content flow that will follow.
1.2 Key Concepts in Immersive Technologies
The landscape of immersive technologies has expanded rapidly over the past few decades, driven by advancements in computing, graphics, displays, audio, sensors, and haptics technologies. These innovations have given rise to a range of new tools and platforms that transform how we interact with digital content and the physical world. Central to this transformation are the concepts of VR, AR, merged or mixed reality (MR), extended reality (XR), the metaverse, and spatial computing. Each of these concepts represents unique capabilities and experiences, but they also share common goals: enhancing user immersion, interactivity, and engagement.
Virtual Reality refers to technologies that create a fully immersive digital environment, effectively replacing the user’s real‐world surroundings with a computer‐generated simulation (Burdea & Coiffet, 2003; Sherman & Craig, 2018; Greengard, 2019; LaValle, 2023). This is typically achieved through the use of head‐mounted displays (HMDs), which present the user with a stereoscopic view and spatial sound of the virtual world, along with motion tracking systems that capture and interpret the user’s movements and map them into the virtual environment. Additional sensory input devices, such as gloves, controllers, and even haptic suits, can enhance the interactivity and realism of the experience. The primary goal of VR is to create a convincing sense of presence, making users feel as though they are truly inside the simulated environment. VR is widely used in various fields, including gaming, where it offers immersive gameplay experiences; simulations for training and education, providing safe and controlled environments for practicing skills; virtual tourism, allowing users to explore distant or inaccessible locations; therapy and rehabilitation, offering controlled environments for therapeutic interventions; social interaction platforms, enabling virtual meetups and collaborative activities; and numerous other immersive applications.
Augmented Reality overlays digital information onto the real‐world environment, enhancing and augmenting the user’s perception of reality without replacing it (Craig, 2013; Schmalstieg & Hollerer, 2016; Chen et al., 2019). AR devices, which can include smartphones, tablets, smart glasses, and head‐up displays (HUDs), project digital elements such as images, videos, sounds, or data onto the user's view of the real world. Unlike VR, which immerses the user in a completely virtual environment, AR integrates digital content with the physical world, providing contextual information and interactive elements that users can see and interact with. AR enhances real‐world experiences by adding layers of information that can be accessed in real time. For example, AR can be used in navigation to overlay directions onto the real‐world streets, in retail to allow virtual try‐ons of clothing or accessories, in education to bring learning materials to life with interactive content, in industrial applications to provide maintenance and repair guidance, and in entertainment through AR games which overlay digital creatures onto real‐world environments. AR’s ability to blend digital and physical worlds opens up numerous possibilities for enhancing how we interact with our surroundings.
Mixed Reality, a subset of Extended Reality and sometimes referred to as Merged Reality, blends physical and digital worlds, allowing real and virtual elements to coexist and interact seamlessly in real time (Milgram & Kishino, 1994; Wang & Schnabel, 2008; Ohta & Tamura, 2014). MR encompasses both AR and VR, providing a spectrum where digital and real‐world objects can interact dynamically. Unlike traditional AR, which can be limited to overlaying digital content onto the real world, MR integrates and anchors digital objects to the physical environment, allowing for more natural and intuitive interactions. MR devices incorporate advanced hardware which can understand and respond to the physical environment through a combination of cameras, sensors, and advanced computing power. This technology enables users to manipulate and interact with both real and virtual elements in a cohesive experience, creating a more immersive and interactive environment. MR is utilized in various fields such as collaborative design, where multiple users can interact with virtual prototypes in a shared physical space; remote assistance, where experts can provide guidance overlaid onto the physical objects being worked on; interactive storytelling, where narratives can unfold around the user; education, providing interactive and engaging learning experiences; and complex data visualization, where abstract data can be visualized in a spatial context.
The Metaverse is the concept of an expansive shared simulated space created by the convergence of physically persistent virtual reality and virtually augmented physical reality (Jaynes et al., 2003; Hazan, 2010; Ball, 2022; Park & Kim, 2022). It represents an interconnected network of immersive digital environments where users can interact, socialize, create, and transact using avatars or digital representations of themselves. The metaverse is envisioned as a successor to the current internet, with an emphasis on shared, persistent, and immersive experiences. Unlike traditional online experiences, which are often isolated and platform‐specific, the metaverse aims to be an interconnected and persistent virtual space where activities and interactions continue regardless of individual user presence. Users can move seamlessly between various virtual worlds and environments, each with its own unique characteristics and purposes. The metaverse has the potential to impact numerous aspects of life, including social interactions, where virtual meetups and events can take place; gaming, providing vast and immersive virtual worlds; virtual economies, where users can trade digital goods and services; remote work, offering virtual office spaces and collaboration tools; education, providing immersive learning environments; and entertainment, with virtual concerts, movies, and interactive experiences. Early examples of metaverse concepts include platforms like Second Life, Decentraland, and Meta's Horizon Worlds, which offer glimpses into the potential of these interconnected virtual spaces.
Spatial Computing refers to technologies that enable computers to understand and interact with the physical world in three dimensions, integrating digital information with the user's environment in a way that feels natural and intuitive (Zambonelli & Mamei, 2005; Shekhar & Vold, 2020; Hackl & Cronin, 2024; Xu et al., 2024). It encompasses a range of technologies, including AR, VR, MR, and other sensor‐driven environments that recognize the spatial relationship between objects and the environment. Spatial computing allows for more intuitive and natural user interactions with digital content by understanding and responding to the physical context of the user. Devices equipped with spatial computing capabilities can map and understand the physical environment, recognizing objects, surfaces, and spatial relationships. This spatial awareness enables dynamic and context‐aware interactions, where digital content can adapt to the user's movements and environment. Spatial computing supports interactions that mimic real‐world behaviors, such as gestures, movements, and voice commands, making the interaction with digital systems more seamless and intuitive.
In addition to the terminologies defined above, several other concepts are widely used in discussions of immersive technologies and experiences. While not exhaustive, the following list provides an overview of important terms that help frame the evolving landscape of virtual and augmented reality, offering insight into the systems, hardware, software, and principles that shape user experiences.
Presence refers to the feeling of being physically present in a virtually simulated world, often used in the context of VR (McCreery et al., 2013; Erickson‐Davis et al., 2021; Wilkinson et al., 2021). It is a key measure of the effectiveness of an immersive experience, indicating how convincingly the virtual environment can make users feel as though they are truly “there.” Achieving a high sense of presence involves realistic perceptual experiences, including immersive visuals, spatial audio, other sensory cues, and intuitive interactions, all with minimal latency and near‐real‐time responsiveness. As an alternate terminology, Telepresence is used in the context of various applications, including remote collaboration, virtual meetings, telemedicine, and remote robotic control, allowing users to interact with distant environments and people as if they were physically present.
Avatar refers to a digital representation or persona adopted by a user to interact within virtual environments such as VR, AR, MR, video games, social media platforms, and the Metaverse (Lin & Wang, 2014; Hoffman, 2021; Fraser et al., 2024; Makani et al., 2024). Avatars can range from simple 2D icons to highly detailed 3D models that mimic the user's physical appearance or embody entirely fictional characters. They enhance user interaction and engagement by providing a sense of presence and personal identity in digital spaces. Users can extensively customize their avatars to reflect their preferences and personality, including physical attributes, clothing, and expressions. In social and collaborative applications, avatars facilitate real‐time communication and interaction, replicating users' movements and expressions for more authentic experiences. The development of avatars also considers privacy, representation, and inclusivity to ensure diverse and safe digital environments. As technology evolves, avatars will continue to become more sophisticated, offering enhanced realism and interactivity.
A Digital Twin is a virtual representation of a physical object or system that is used to understand, analyze, and simulate real‐world conditions and operations (Vohra, 2023; Sabri et al., 2024). Digital twins are used in industries such as manufacturing, healthcare, and urban planning to monitor performance, predict outcomes, and optimize operations. By providing a detailed digital counterpart to physical assets, digital twins enable real‐time data analysis and decision‐making.
Human–Computer Interaction (HCI) is the interdisciplinary study and practice of designing, implementing, and evaluating the interfaces through which people interact with computers and digital systems (Coomans & Timmermans, 1997; Dix et al., 2003; Jacko, 2012; Bhowmik, 2014). In the field of immersive technologies, HCI is particularly concerned with creating intuitive and effective interfaces for interactive experiences. The goal is to ensure that these interactions are as natural, seamless, and user‐friendly as possible, thereby enhancing the overall user experience. This involves the meticulous design of input devices such as motion controllers, haptic gloves, and eye‐tracking systems that allow users to navigate and manipulate virtual environments with ease and precision. Additionally, HCI in immersive technologies explores innovative interaction techniques, such as gesture recognition, voice commands, and spatial interactions, to create more engaging and immersive experiences. User experience principles play a critical role in this process, guiding the development of multimodal interfaces that are not only functional but also accessible and enjoyable for a wide range of users. By focusing on the ergonomic and cognitive aspects of interaction, HCI aims to bridge the gap between complex digital systems and human capabilities, making immersive technologies more approachable and effective for everyday use.
It is worth acknowledging that the system definitions narrated above may not be consistent with how various companies across disparate industries describe and promote their respective products. As this space is continuing to evolve, new ways to articulate new applications are expected to emerge.
This book predominantly uses the terms “Virtual and Augmented Reality” and is titled as such to reflect the foundational and most widely recognized technologies in the domain of immersive experiences. While terms like mixed reality, extended reality, merged reality, metaverse, and spatial computing represent important and evolving aspects of this technological landscape, virtual reality and augmented reality remain the core components around which these concepts are built. VR and AR are well‐established, with distinct and clear definitions that provide a solid basis for understanding the broader scope of immersive technologies. By focusing on VR and AR, this book aims to offer a comprehensive introduction that is accessible to a broad audience, including those who may be new to these technologies. It also sets a strong foundation for exploring more advanced and integrated concepts, helping readers to gradually build their knowledge from fundamental principles to more complex ideas.
1.3 Multisensory Immersion and Interaction
Immersive experiences in VR and AR are most compelling when they engage multiple senses, creating environments that feel natural, intuitive, and realistic. Multisensory immersion is central to achieving a convincing sense of presence and enhancing the user's ability to interact with virtual or augmented environments seamlessly. The human brain processes information from multiple sensory modalities, such as vision, hearing, touch, smell, and motion, to construct a coherent representation of the world (Stein & Meredith, 1993; Ernst & Bülthoff, 2004). Effective VR and AR systems aim to mimic this natural integration process by aligning sensory inputs in ways that reinforce the user’s perception of the virtual or augmented environment. When sensory cues are synchronized and believable, they enhance the user's sense of presence, the feeling of “being there” in a virtual or augmented space.
Vision plays a primary role in VR and AR systems, delivered through high‐resolution displays, stereoscopic rendering, and realistic lighting effects. Innovations in head‐mounted displays, wide field‐of‐view optics, and real‐time 3D graphics contribute to creating visually immersive environments. In addition to visual elements, auditory input adds another critical dimension to immersion. Spatial audio technologies recreate realistic soundscapes by simulating directional sound sources and environmental acoustics, complementing visual elements with auditory cues such as footsteps that align with a character's location or background music that sets the mood. Haptic feedback further enhances immersion by allowing users to feel textures, resistance, and vibrations through touch‐based devices such as haptic gloves, vests, and controllers. These devices provide tactile sensations that enhance interactivity and realism, allowing users to engage with virtual environments in a more tangible manner. Though still in development, olfactory input adds an additional layer of immersion, with applications in training environments where scent provides critical context, such as the smell of smoke in fire safety drills. Vestibular input, on the other hand, engages the user's balance and motion perception through motion tracking and dynamic simulation technologies, which enhance experiences such as flight training or racing games by replicating realistic movement and acceleration. True immersion occurs when sensory inputs are not only realistic but also seamlessly integrated. For example, a VR experience might combine visual cues of a falling object with spatialized audio and haptic feedback when the object “lands.” AR applications, such as navigation systems, enhance user experience by overlaying directional arrows onto the real world while simultaneously providing vibration cues and spoken directions.
The integration of multiple input methods, such as voice recognition, hand gestures, eye‐gaze tracking, and brain–computer interfaces, further enhances natural user interactions (Bolt, 1980; Poupyrev et al., 1998). A user could, for instance, point at an object in an AR environment, give a voice command to move it, and observe the system responding in real time. Brain–computer interfaces, still in research, add another dimension by enabling direct neural interaction, allowing users to control virtual elements through thought alone, bypassing traditional input devices. These multimodal interactions make immersive systems more intuitive and responsive, bridging the gap between digital and physical interactions and opening new possibilities for accessibility, cognitive enhancement, and intuitive control.
Achieving seamless multisensory integration presents several challenges. One of the most significant issues is synchronization, where misalignment between sensory inputs, such as delayed haptic feedback following a visual event, can disrupt immersion and reduce the sense of presence. Hardware limitations also pose constraints, as devices capable of delivering accurate sensory inputs across multiple modalities can be bulky and expensive. Latency is another crucial factor; any delay between user actions and system responses can break immersion, reduce the effectiveness of the experience, and cause discomfort (LaViola, 2000; Chang et al., 2020
