1.

INTRODUCTION

Defense has been the first application sector for AR and VR, as far back as the 1950s [1]. Based on such early developments, the first consumer VR/AR boom has expanded in the early 90s and contracted considerably throughout the decade, as a poster child of a technology endeavor ahead of its time, and also ahead of its markets [2]. However, due to the lack of available consumer display technologies and related sensors, novel optical display concepts had been introduced throughout this decade [3], [4], which are still considered today as state of the art, such as the “Private Eye” smart glass from Reflection Technology (1989) or the “VR Boy” from Nintendo (1995), both based on scanning displays rather than flat panel displays. Although such display technologies were well ahead of their time [5], [6], [7], the lack of consumer grade IMU sensors, low power 3D rendering GPUs and wireless data transfer technologies contributed to end this first VR boom in the 90s. The other reason was the lack of digital content, or rather the lack of a clear vision of an adapted VR/AR content for enterprise or consumer [8], [9].

This is very similar to the demise of the early iPod developments at General Magic Corp. in the late 90s, which provided an initial hardware concept however without wireless connectivity or on-line music store. Once all three conditions were met at Apple 10 years later (hardware, WIFI connectivity, on line music store), the iPod concept resonated with a strong consumer market, as a music experience ecosystem, not solely as a desirable music playing product, no matter how excellent it might have been designed and produced.

No matter how good a consumer AR/MR headsets can be, if it is nto shipped with its corresponding “iTune Store App”, the consumer market will not take off.

The only AR/VR sector that saw sustained efforts and developments throughout the next decade was the defense industry (flight simulation and training, Helmet Mounted Displays -HMD- for rotary wing and Head Up Displays – HUD- for fixed wing aircrafts) [10]. The only effective consumer efforts during the 00s has been in the field of automotive HUDs and personal binocular headset video players.

Today’s engineers, exposed at an early age to ever invading flat panel displays technologies, tend to act as creatures of habit much more than their peers 20 years ago who had to invent novel immersive display technologies from scratch. We therefore saw since 2012 the initial implementations of immersive AR/VR HMDs based on readily available smartphones display panels (LTPS-LCD, IPS-LCD, AMOLED) and pico-projectors microdisplay panels (HTPS-LCD, mu-OLED, DLP, LCoS), and IMUs, camera and depth map sensors (structured light and TOF). Currently, HMD display architectures are evolving slowly to more specific technologies, which might be a better fit for immersive requirements than flat panels were, and resembling the display technologies invented throughout the first AR/VR boom 2 decades earlier (inorganic mu-LED panels, 1D scanned arrays, 2D laser/VCSEL MEMS scanners, …).

The smart phone technology ecosystem, as in display, connectivity and sensors, shaped the emergence of the second VR/AR boom and have been certainly the lowest hanging fruits to be used as the first building blocks by early product integrators. Such traditional display technologies will serve as an initial catalyst for what is coming next.

The immersive display experience in AR/VR is however a paradigm shift from traditional panel displays experiences since more than half a century, starting from CRT TVs, to LCD computer monitors and laptop screens, OLED tablets and smart phones, LCoS, DLP and MEMS scanner digital projectors, to iLED smart watches (See Figure 1).

Figure 1:

Immersive NTE displays: a paradigm shift in personal information display

When flat panel display technologies and architectures (smart phone or micro-display panels) are used to implement immersive Near To Eye (NTE) display devices, étendue, fixed focus and low brightness become severe limitations. Addressing the needs for NTE immersive displays to match the specifics of the human visual system requires alternative display technologies.

The second emergence of VR/AR/smart-glasses boom in the early 2010s has also introduced new naming trends, more inclusive than AR or VR: Mixed - or Merged - Reality (MR), more generally referred today as “XR”, a generic acronym as in “Extended Reality”. Smart Eyewear (including both digital information display and prescription eyewear) tend also to replace the initial Smart Glasses naming convention.

2.

SMART GLASSES/AR/MR/VR MARKETS

Today, unlike in the previous AR/VR boom, investors, market analysts and AR/VR/MR system integrators as well as enterprise users are expecting to see a real Return On Investment (ROI) for these unique technologies in the next 5 years, as underlined by the 2017/18 Gartner Hype Cycles (see Figure 2).

Figure 2:

Gartner Hype Cycle showing VR, AR and MR technologies for 2017-2018.

The 2017 Gartner graph shows clearly AR and VR poised to reach the plateau of productivity within 2 to 10 years, with VR preceding AR by a few years. This is a notion shared by most AR/VR market analysts. It is interesting to note that the revised 2018 graph does not show VR anymore, instead introduced MR as departing from the peak of inflated expectations. VR approached in 2018 a more mature stage, even becoming a commodity, moving it thus off the emerging technology class of innovation profiles.

However, one has to take these expectations with a word of caution; the only market sector that has been proven to be sustainable today is MR for enterprise, where the ROI is mainly cost avoidance:

Enterprise sectors that have already shown tangible MR ROI are concentrated in the manufacturing (automotive, avionics, heavy industrial products), power, energy, mining and utilities, technology, media and telecom, as well as in healthcare and surgery, financial services, and retail / hospitality / leisure fields.

Proof of an existing consumer market for smart glasses/AR/MR is less obvious; hardware experiments have been tried out with mixed results for smart glasses (Google Glass, Snap Spectacles, Intel Vaunt or North “Focals”). VR headsets developments have also slowed down recently (Oculus/Facebook VR, Google Daydream, Sony Playstation VR). Other VR efforts have been halted, such as the Video See through (Merged Reality) project Alloy from Intel and the ACER/StarVR wide FOV VR headset. However, the potential of video see through MR remains strong in the long term. 2018 also saw major shut downs of medium sized AR headset companies such as MetaVision Corp (Meta2 MR headset), CastAR Corp and ODG Corp (ODG R8 and R9 glasses), after all three had a strong initial product introduction and VC capital support. Others went through major re-structuration such as Avegant Corp. (Multifocal AR headset). MetaVision and CastAR saw also a recent rebirth in mid 2019, showing that the field is still uncertain and full of surprises. Others (Vuzix Corp) saw continuous growth as well as continuous VC backing (Digilens Corp) throughout 2019.

On the smart glass front, audio only smart eyewear has made a strong return. Audio smart glasses, providing audio immersion as well as world locked audio (based on IMU solely) are not new concepts, but have recently shown upgrades such as surround sound leaving the ear free (no bone conduction) and noise cancelling features towards the world (such as the Bose Frames). They can provide essential input and commands for both consumer and enterprise, and are an essential part of the augmented world experience. Major companies as Huawei have introduced their own version of audio augmented reality smart glasses recently. Camera glasses such as the Snap Spectacles (1^st and 2^nd generations) have had a tough time to show consumer acceptance, as did Google Glass Explorer version.

In addition to spatial world locked audio, if an IMU is present (as in the Bose AR Frames), a number of different head and body gestures that can also be detected including push-ups, squats, nod, shake, double tap, look up, look down, spin around, roll head around.

Table 1 summarizes the current adoption rate in consumer/enterprise/defense sectors for the different types of smart glasses/AR/VR/MR headsets available today.

Table 1:

Current AR/VR/MR/Smart Glasses product offerings in consumer/enterprise/defense markets.

Small form factor Smart Glasses including minimal display (around 10 deg FOV monocular) and prescription correction saw a renewal in 2018 (after the 2014 Google Glass failure in the consumer market sector) with the hyped project Vaunt at Intel. However, this project has been halted later that year, as Intel invested rather in a very similar architecture the “Focals” developed by North Inc. in Kitchener, Canada. But after a price drop of nearly 50% early 2019, and a strong workforce layoff by North, the short term outlook for consumer smart glasses remains uncertain. Other smart glass concepts targeted solely to enterprise sectors have had a quiet but steady growth, such as the rugged RealWear headsets (Vancouver, Canada) and the more stylish Google Glass Enterprise V2 glasses.

On the other hand, the current VC investment hype fueling frenetically single start-ups such as Magic Leap Inc. (totaling >$3B VC investment pushing up a >$7B company valuation before seeing any revenues) is a harsh reminder of the ever present “Fear Of Missing Out” behavior from late stage investors (Alibaba, Singapore Temasek and Saudi funds) eager to jump on the bandwagon fueled by the early investment decisions from major tech VC firms (Google ventures, Amazon, Qualcomm). It is also interesting to note that the two last VC investments in Magic Leap (late 2018 and mid 2019) were by major communication companies (ATT/USA for an unknown amount and NTT/Docomo-Japan for $280M), pointing out that large bandwidth communication channels (5G, WiGig, etc…) will be fueled by future demanding AR markets, allowing also sustained ROI over MR cloud services (AR/MR hardware returns being razor thin).

No matter the investment hype, it might very well take a major consumer electronics company to create at the same time the ultimate consumer headset architecture (addressing visual / wearable comfort and immersion experience) and the subsequent consumer market. Unlike for the enterprise market where the content is provided by each individual enterprise through custom application developments for specific needs, the consumer market needs to rely solely on the entire MR ecosystem development, from generic hardware to generic content and applications.

Even though Q3 2018 saw for the first time a worldwide decline in both smart phone and tablet sales, hinting at Apple’s Q4 2018 30% stock fallout, it is unclear if such an MR consumer hardware has the potential (or even the will) to replace the existing smart phone/tablets, or alternatively be the ultimate companion to a smart phone, providing an immersive experience that is out of reach for any other traditional display screen concept.

Apart the consumer and enterprise markets discussed here, there is still today a considerable defense market for MR headsets. Microsoft has secured in Q4/18 a $480M defense contract to develop and provide the US Army special versions of HoloLens, dubbed IVAS (Integrated Visual Augmentation System). Being the largest contract ever in AR/VR/MR, consumer, enterprise and defense combined, this will boost the entire MR ecosystem worldwide.

3.

THE EMERGENCE OF MR AS THE NEXT COMPUTING PLATFORM

Smart glasses (also commonly called Digital Eyewear) are mainly an extension of prescription eyewear, providing a digital contextual display to the prescription correction for visual impairment (see Google Glass in Figure 3). This concept is very different from either AR or MR functionality. Typical smart glass FOV remains small (less than 15 deg diagonal), often off-set from the line of sight. The lack of sensors (apart the IMU) allows for approximate 3DOF head tracking, and lack of binocular vision reduces the display to simple overlaid 2D text and images. Monocular displays do not require rigidity in the frames as a binocular vision system would (to reduce horizontal and vertical retinal disparity that can produce eye strain). Many smart glass developers provide also prescription correction as a standard feature (“Focal” by North or Google Glass V2).

Figure 3:

The emergence of Smart Glasses, AR/MR and VR headsets.

The combination of a strong connectivity (3G, 4G, WiFi, Bluetooth) and the addition of a camera makes it a convincing companion to a smart phone, for contextual display functionality or as a virtual assistant, GPS and social network companion (thanks to the camera). A smart glass does not aim at replacing a smart phone, but contributes to a good addition to it, as a smart watch would do.

VR headsets are an extension of gaming consoles, as shown by major gaming providers such as Sony, Oculus, HTC Vive and Microsoft Windows MR, with gaming content partners such as Steam VR. Such headsets are often also sold with gaming controllers (see Figure 3). Early outside-in sensors (as the standalone with Oculus CV1 and HTC Vive 2016) lead the way to inside-out sensors in newer generation headsets, providing a more compact hardware (Windows MR headsets such as the Samsung Odyssey). Although these high-end VR systems still require high end GPU in costly desktop or laptop gaming PCs, standalone VR headsets have recently been introduced (2018), such as the Oculus Go (3DOF-IMU) and the HTC Vive Focus, able to seduce a burgeoning VR consumer market base. More recently, further extensions of standalone VR headsets with inside-out sensors led to 2019 products such as the Oculus Quest (6DOF standalone VR headset).

However, tethered high end VR headsets with inside-out sensors have also been updated, by both Oculus and HTC providing in 2019 high end products such as the Oculus Rift “S” and the HTC Vive Pro. Other updates in Windows MR headsets have been done by Samsung “Odyssey 2019 with double the resolution of the first 2017 version).

AR and especially MR systems are poised to become the next computing platform, replacing the ailing desktop and laptop hardware, even the now aging tablet computing hardware. Such systems are untethered for most of them (see HoloLens V1 in Figure 3), and require high-end optics for the display engine and the combiner optics as well as for the sensors (depth scanner camera, head tracking cameras to provide 6DOF, accurate eye trackers and gesture sensors). These are today the most demanding headsets in terms of hardware, especially optical hardware. Eventually, if technology permits, these three categories will merge into a single hardware concept. This will however require improvements in connectivity (5G, WiGig), visual comfort (new display technologies) and wearable comfort (battery life, thermal management, weight/size).

The worldwide sales decline for smartphones and tablets in Q3 2018 is an acute signal for major consumer electronics corporations and VC firms to fund and develop the “next big thing”, whatever it may be. But MR is a good candidate!

4.

THE KEYS TO THE ULTIMATE MR EXPERIENCE

The ultimate MR experience, for either consumer or enterprise, is defined along two main axes: comfort and immersion. Comfort comes in two declinations: wearable and visual. Immersion comes in various declinations, from display to audio, gestures, haptics, etc…

At the confluence of comfort and immersion, three main features are required for a compelling MR experience:

Most of this can be achieved through a global sensor fusion process [5] integrated through dedicated silicon, as it has been implemented in HoloLens with the HPU (Holographic Processing unit) [11].

4.2

Display immersion

Immersion is the other key to the ultimate MR experience, and is not only based on FOV: the FOV is a 2D angular concept, immersive FOV is a 3D concept, including the Z distance from the user’s eyes, allowing for arm’s length display interaction through VAC mitigation).

Immersion experience for the user comes in many flavors:

• - Wide angular Field of View (WFOV) including peripheral display regions with lower pixel count per degree (resolution) and lower color depth.

• - Foveated display in either fixed version (foveated rendering) or dynamic version (through display steering, mechanically or optically).

• - World locked holograms and hologram occlusion through the use of depth map scanners

• - World locked spatial audio

• - Accurate gesture sensing through dedicated sensors

• - Haptics

Figure 4 summarizes the main requirements to enable the analyst’s optimist version of the VR/AR/MR/Smart glasses market, at the of confluence of immersion experience and wearable/visual comfort.

Figure 4:

Comfort and immersion requirements for the ultimate MR experience.

The dark grey items in Figure 4 are based on a critical enabling optical technology for next generation MR headsets: fast, accurate and universal eye/pupil/gaze trackers.

5.

HUMAN FACTORS

In order to design a display architecture aiming at providing the ultimate MR comfort and immersion experience described in the previous section, one needs to consider the optical design task as a human centric task. This section analyzes some of the specifics of the human vision system [12], and how one can take advantage of such in order to reduce the complexity of the optical hardware as well as the software architecture, without degrading in any way the user’s immersion and comfort experience [13].

5.1

The human visual system

The human Fovea, where resolution perception is at a maximum due to its high cone density, covers only 2-3 deg, and is set off-axis from the line of sight by about 5 deg. The human fovea is a result of early life visual experience, and grows from a small age to form a unique area set apart of the line of sight.

Line of sight and optical axis

The human vision specifics are based on the cone and rod density over the retina, as described in Figure 5. The optical axis (or pupillary axis, normal to the vertex of the cornea) is slightly offset from the light of sight [14], (close to the visual axis) by about 5 degrees, and coincides with the location of the fovea on the retina. The blind spot, where the optic nerve is located, is offset by about 18 degrees off from the center of the Fovea.

Figure 5:

Rod and Cone cells density on the retina (left), optical axis and line of sight (right).

It is interesting to note that the human fovea grows slowly in early life based on specific human visual behavior, and is not a feature of our visual system given at birth. Therefore, the location of the fovea might drift to new positions on the retina with novel visual behaviors radically different from millennia of human evolution, such as the use of small digital displays held at close range by toddlers. Another severe change would be early childhood myopia due to the same cause [15] [16].

Lateral Chromatic Aberrations or LCA

Chromatic aberrations result in the separation of colors through Fresnel rings, gratings or traditional refractive lenses, all of which are dispersive. The “L” in LCA reflects both “lateral” and “longitudinal” chromatic spread, where different colors are focusing at different depths, depending on the Abbe V number of the lens (refractive elements having the opposite dispersion of diffractive elements). Reflective optics do not produce LCA, and are therefore used extensively in AR display applications.

Correcting for LCA is usually done in software, by pre-compensating each color frame in a field sequential mode or precompensating the entire color image in an RGB display panel (equivalent of having three distortion map compensations, one for each color). However, this can lead to display artifacts such as color aliasing, and needs high angular resolution to achieve a good effect. Optical dispersion compensation is a better way, but requires also more complex optics (hybrid refractive diffractive) or symmetric coupling architectures such as in waveguide combiners using grating or holographic couplers, or replacing refractive optics by reflective optics.

As the main detector is here the human eye, it is also interesting to analyze the natural LCA spread of the eye, which is surprisingly strong. Figure 6 shows the measured LCA of the human eye (left, as an aggregate of measurements from the past 50 years), yielding a 2 diopters spread over the visible spectrum, and how color images appear on the retina, on axis and off-axis (center), and how they appear as “seen” or “experienced” by the viewer after being processed by the visual cortex (right).

Figure 6:

Natural LCA of the human eye

The natural LCA of the human eye is also the basis of an interesting digital rendering 3D depth cue, called “Chromablur”. The blue and red blur (optical or rendered), depending on which side of a white object they appear, can tell a far defocus from a near defocus, and thus inform the eye’s oculomotor muscles on the sign of the accommodation power change to get the image back in focus (that is, the green part of the image focused on the retina).

The human eye LCA varies slightly from one individual to the other: therefore, changing slightly the natural LCA by increasing or decreasing the spectral spread with external optics will not affect dramatically the visual acuity. However, if part of the field has a specific LCA (see through in AR), and the other part has a different LCA (digital image in AR), this could lead to some visual discomfort.

Visual acuity over fovea and peripheral region

The measured polychromatic MTF of the human eye is plotted in Figure 7. This MTF represents mainly photopic vision over the on-axis field (optical axis of the eye or pupillary axis), which is close to the line of sight field centered on the foveated area on the retina (around 3-5 degrees offset).

Figure 7:

Polychromatic Modulation Transfer Function (MTF) of the human eye for various pupil diameters

The ability of the eye to resolve small features is the “visual acuity.” The teenage and early adult human eye can distinguish patterns of alternating black and white lines as small as one arcmin (or 30 cycles per degree or 60 PPD). That is also the definition of 20/20 vision. A few of us might distinguish smaller patterns (having a higher MTF at these cycles per degree), but most of us will see these patterns as grey shades (having and low MTF, below 0.1).

For all pupil sizes, the MTF at 20/20 for photopic vision is higher than 30%, only for scotopic vision for pupils over 5mm would the MTF drop to lower levels.

Note that MTF50 (50% of the MTF for low frequency) or MTF50P (50% of MTF from the peak value) are good indicators, but rather used for cameras. The human eye especially as it moves constantly, can still distinguish features well in the 30% MTF levels. Unlike cameras, the eye’s MTF also drops for very low frequencies (mainly due to lack of movement at such low frequencies).

Due to high aberrations in the human eye, the higher the pupil size the lower then resulting MTF, which is the opposite to diffraction limited high end camera objectives where the MTF increases with the aperture. It is interesting to note that higher spatial frequencies are actually distinguishable when the pupil increases in photopic vision, allowing over 20/20 vision over lesser bright fields.

This said, the human vision system cannot be limited to its pure optical properties as a simple camera would, but as a computational imaging system (where the CPU is the visual cortex). The impressive LCA discussed in the previous section is a testimony to this effect.

Specifics of the human vision FOV

Figure 9 shows the horizontal extend of the different angular regions of the human binocular vision system. Although the entire FOV spans more than 220 deg horizontally, the binocular range spans only 120 deg in most cases (depending on the nose geometry). Stereopsis (the left and right monocular vision fusion [19] providing 3D depth cue) is however more limited (+/-40 deg) [17].

Figure 8:

Human vision FOV (H and V)

Figure 9:

Human binocular field of view with fixed foveated region including unconstrained eye motion allowing for sustained gaze and accommodation.

The vertical FOV is similar in size to the horizontal FOV, and is set off-axis from the standard line of sight, by about 15 deg downwards (relaxed line of sight)

The human FOV is a dynamic concept, best described when considering the constrained and unconstrained eye motion ranges [14] (unconstrained: motions that do not produce eye strain and allow for steady gaze and subsequent accommodation reflex). While the mechanical eye motion range can be quite large (+/-40 deg H), the unconstrained eye motion over which gaze is possible without inducing the head turning reflex is much smaller, and covers roughly +/-20 deg FOV H. This in turn defines the static foveated region, ranging 40-45 deg FOV H. Fig. x shows the human binocular vision FOV, as the overlap of the left and right fields, as well as the parafovea and the center fovea region ranging 3 degrees full angle [18] (see Figure 9).

Figure 10:

Typical FOVs for current state of the art Smart glasses, AR, MR and VR headsets, overlaid on the human binocular vision and the fixed foveated display region.

The binocular FOV [20] is a quite large region, horizontally symmetric and vertically asymmetric, spanning +/-60 deg over a +55 deg upper and -30 deg lower, with a central lower region reaching also -60deg but over a smaller horizontal span of 35 deg full angle. The white circle showing the fixed foveated region over which sustained eyegaze is possible defines also the state of the art diagonal FOV for most high end AR/MR devices today, which also provide a 100% stereo overlap. Furthermore, for a given gaze angle, color recognition spans over a +/-60 deg FOV, shape recognition over a +/-30 deg FOV, and text recognition over a much smaller FOV of +/-10 deg.

5.2

Adapting display hardware to the human visual system

Various FOVs from existing HMDs are shown in Figure 10. Standard VR headsets (Oculus CV1, HTC Vive, Sony Playstation, Microsoft Windows MR) have all FOVs around 110 deg FOV diagonal, stretching towards 200 deg FOV for some others (PiMax and StarVR). Large AR FOV up to 90 deg can be produced by a large cell phone panel display combined with a large single curved free space combiner (Meta2, DreamGlass, Mira AR, NorthStar Leap Motion AR), to smaller FOV high end AR/MR systems with microdisplay panels such as Microsoft HoloLens V1 and Magic Leap One. Smart glasses have typically D-FOVs starting from 10-15 deg (Zeiss “Tooz” Smart Glasses, Google Glass, North Focals) to larger FOV starting at 25 deg, up to 50 deg D-FOV (Vuzix Blade, Digilens, Optinvent ORA, Lumus DK50, ODG R9).

One other way to describe the FOV experience is to overlap the amount of unobstructed see though FOV over the actual display FOV (see Figure 1). The fixed foveated region (scanned through eye movements) is shown in dotted lines.

Figure 11:

Display FOV and see through FOV for various VR, smart glasses and AR headsets.

For a VR system, there is no see through, and thus the display FOV can be quite large, 110 D-FOV to 150 D-FOV (left), up to 200+ FOV with products from PiMax or StarVR. For a smart glass (Google Glass, center left), the see through – or rather see around-experience is wide, only hindered by the lateral display arm, with an ex-centered display D-FOV of 15 deg. For the Magic Leap One MR headset (center right), the tunneling effect due to the circular mechanical enclosure of the glasses reduce the see through considerably, to about a 70 deg circular cone, while the display has a D-FOV of 50 deg. For Hololens V2 (right), the lateral see-through (or see-around) FOV, is equal to the natural juman FOV of 220 deg, with a display diagonal D-FOV of 52 deg., covering most of the foveated region; only the top part of the FOV is capped by the mechanical enclosure holding the sensor bar, the laser/MEMS display engines and the system board. There is no limitation to the bottom FOV. One other particularity of HoloLens V2 is the ability to move the entire display visor up to reveal a totally unobstructed view.

In order to optimize an HMD optical architecture for large FOV, the various regions of the human FOV described in Figure 9 have to be considered in order not to overdesign the system. This allows, through a “human centric optimization” in the optical design process, to produce a system which is closely matched to the human vision system in terms of resolution, MTF, pixel density, color depth and contrast. Vergence Accommodation Conflict can also be considered as foveated

6.

CONCLUSION

This paper served as an introduction to the EDO19 conference “Digital Optical Technologies” dedicated specifically to digital optics for AR/VR and MR systems, in all their incarnations: Display, Imaging and Sensors. We introduced the field of AR/VR/MR optics historically, and reviewed the most important criteria for AR/VR/MR optical design: matching the performances of the optics to the ones of the human visual system. This can be done for all Digital Optical Elements reviewed throughout this conference: free space optics, waveguide optics (waveguide combiners), diffractive, holographic and metasurface optical elements, tunable, switchable and reconfigurable optics, and novel computational techniques for display and imaging.