How Augmented and Virtual Reality Works

This book is an essential guide to the science, mathematics, and engineering behind virtual and augmented reality. Written in everyday language, we explore the optics, algorithms, mathematics, biology, and the difficult art of convincing the human brain that a digital world is real. At its heart, we answer the question: what does it take to build a believable virtual world? The book begins with the new generation of headsets and smart glasses, then opens the black box behind them: cameras, LiDAR, motion sensors, eye trackers, displays, lenses, chips, batteries, and software systems that must all work together in perfect synchrony. The book also explains how these devices know where you are.
We explore 6-degrees-of-freedom tracking, coordinate systems, sensor fusion, and SLAM, the mathematical loop that lets a headset build a map of a room while simultaneously locating itself inside that map. We also look at what happens when this math fails, from drifting virtual objects to broken immersion. We discuss how virtual images are created through micro-OLED displays, pancake lenses, waveguides, distortion correction, and foveated rendering.
We explore the rendering engine, where digital worlds must be rebuilt 90 to 120 times every second, and why latency, reprojection, and asynchronous timewarp are essential for keeping the illusion stable and comfortable. Finally, the book asks why building the future of computing is so difficult. It examines hand tracking, eye tracking, voice control, neural wristbands, motion sickness, eyestrain, heat, battery life, cloud computing, privacy, social acceptance, and the hard lessons learned from products that solved the technology but failed the human test.