3.2.1.

Anatomic mapping and navigation

Augmented and MR modalities of immersive technology allow for the digital augmentation of a surgery or procedure. Perhaps the most recognized utilization of immersive technology in the operating room is the superimposition of medical imaging data onto the patient, providing augmented surgical navigation and disease localization. Patient-specific imaging data can help in pinpointing important anatomic structures and identifying areas of disease versus healthy tissues,⁹ optimizing the surgical operation by enabling faster and more accurate anatomic identification.^15,16 This has been used prior to the start of a surgical operation to align the location of surgical incisions,¹⁷ to inform appropriate instrument placement for minimally invasive procedures,¹⁸ and to provide an optimal surgical setup for implant placement.^19,20 It can also be used to provide guidance on surgical and ablative resection boundaries²¹ as well as prevent damage to adjacent structures. Studies have been completed crossing multiple surgical disciplines including neurosurgery,¹⁸ orthopedic surgery,¹⁷ thoracic surgery,¹⁵ and vascular surgery.¹⁶ Commercial companies have already entered this space hoping to aid interventional radiologic and surgical procedures.²²

3.2.2.

Augmentation of the operative field

As the surgeon navigates the complexities of the human anatomy during a procedure, patient data can be implemented into the surgeon’s field of vision, offering a blend of the live surgical view with supplementary information (see Fig. 2).²³ This can range from displaying vital signs such as heart rate and blood pressure readings, incorporating images, videos, or external technology for procedural reference,²⁴ information related to procedural efficiency,²⁵ or to review patient data such as labs or imaging.^26,27 These types of data are available in the operating room already, but the ability for the surgeons to interact with this data is limited due to the sterile environment and reliance on operating room staff and anesthesia. XR allows for a more integrated access to information during the procedure. Furthermore, XR can assist with instrument navigation, indicating the optimal path for surgical instruments and potentially reducing the risk of inadvertent damage to critical structures, especially for cancer operations²⁸ as well as minimally invasive and percutaneous operations. By seamlessly integrating this wealth of real-time data into the surgical view, immersive technology can not only bolster the surgeon’s spatial awareness and precision but can also expedite decision-making and improve overall surgical outcomes.¹⁵

Fig. 2

VR (top left) of a heart anatomy training module. AR (top right) showing a visual overlay of the positioning of surgical instruments for vascular access during an extracoporeal membrane oxygenation (ECMO) cannula insertion. MR (bottom) showing a step-by-step guide to performing a central catheter line insertion procedure. This shows how the various immersive modalities have differences in the level of immersion provided as well as the haptic feedback that can be afforded by the technology. Each technology has advantages and disadvantages as to how it can be applied to both pre-surgical planning and surgical augmentation in the operating room.

3.2.3.

Ergonomic improvement

A critically important feature of XR’s ability to provide clinical information and patient data to the surgeon’s immediate point-of-view, without distracting away from the surgical operation, is its impact on ergonomics. Iqbal et al.²⁹ demonstrated the integration of holographic operating data from a surgical robot to the surgeon using AR, enhancing the surgeons’ ability to see the operative field and have added information present, without having to go to an alternate screen. By having the ability to optimize how and where information is presented, the surgeon can improve their personal ergonomics during the operation, limiting unnecessary and strainful movements.³⁰ This both helps to facilitate the speed and smoothness of the operation as well as the longevity of the surgeon in the physically challenging environment of the operating room.³¹ Postural feedback mechanisms may also be provided in the future with the addition of positional monitors to help provide information to the surgeon to optimize their posture and decrease mechanical stress and positional strain.³²

3.2.4.

Remote surgical mentoring/collaboration

We have previously highlighted the advantages immersive technologies provide for remote collaboration in both diagnostics, training, and planning methods. The same applies during the surgical operation as XR allows for surgeons at all levels of training and anywhere in the world to provide remote guidance and instruction in an immersive format.^33,34 One study even evaluated using immersive technology to overlay a virtual physician’s hands into the surgical operative environment to facilitate instruction and teaching as a method of enhancing global access to state-of-the-art care.³⁵ Remote telementoring even has military applications as the US Military has used it for surgical augmentation through their system for telementoring with an augmented reality system.³⁶ As has been stated previously, remote surgical collaboration using immersive technologies may provide democratization of surgical skills and expand procedural proficiency and competency among surgical peers.³⁷ This can be applied at a local, regional, national, and even international level.

In summary, immersive technology stands poised to improve surgical operations and interventional procedures, offering multifaceted enhancements to the proceduralist’s experience. Visual overlays utilizing medical imaging data grant surgeons and others an augmented perspective of the operative field. By seamlessly integrating patient data and medical imaging into the surgeon’s line of sight, real-time decision-making becomes more informed, reducing the need to divert attention to external monitors or charts. Ergonomically, XR addresses challenges faced by surgeons by offering postural feedback, reducing physical strain, and facilitating more natural interactions with the surgical environment. In addition, immersive technology breaks down geographical barriers through remote telementoring, allowing novice or inexperienced surgeons to receive guidance from experienced mentors in real time. Immersive technology, through its various modalities, lays a foundational framework to provide augmented surgical care strategies in the future.

3.3.1.

Cost/Affordability

This is a common theme in the use of XR technologies within both preoperative and operative use cases. Despite the initial costs required for hardware and software, with the rapid scalability and enhanced methods of communications and sharing of patient data with immersive technologies, there are various methods that can be utilized to provide cost savings to healthcare systems, especially with repeatable and scalable environments and the ability to provide remote collaboration.^38–40 One study has demonstrated the benefits of immersive VR in training scrub nurses on technically challenging knee surgeries, resulting in significant improvement in real-world skills for everyone in the operating room team.⁴¹ Others have shown that VR can be used to reduce costs associated with designing operating rooms and other healthcare environments by allowing “interactive virtual prototyping” of novel spaces.⁴²

3.3.2.

Accessibility

Cloud computing has hastened the speed at which we can share and utilize digital resources and current information transfer technology now allows for both live and recorded shared XR experiences. This not only allows for locoregional collaborations using XR at a multidisciplinary level but allows users to connect from remote locations anywhere in the world rather than requiring presence at a physical facility. Previous studies have cited the ability to coordinate complex surgical cases and discuss them at both an institutional and multidisciplinary level, as well as at a global level, sharing knowledge and clinical expertise at an intercontinental scale.^26,27 This allows for the democratization of radiological and surgical skill sets through shared medical imaging interpretation as well as surgical planning and operative collaboration.¹²

3.3.3.

Telementoring and remote collaboration

Immersive technology provides the capacity for remote collaboration which, when applied in the training environment, allows for telementoring and remote coaching/teaching. When comparing immersive collaboration with traditional, in-person methods, studies have shown no changes in communication application and performance while citing the numerous benefits provided by the virtual environment.⁴³ Immersive telementoring has been used in the clinical operative setting between faculty physicians through remote collaboration, providing the quick sharing of surgical skills and expanding procedural proficiency and competency among surgical peers.¹²

3.3.4.

Multidisciplinary Planning and Communication

Because of its accessibility, XR provides a method to conduct multidisciplinary, interdisciplinary, and peer-to-peer communication, allowing multiple users from various specialties, levels of expertise, or regionality to communicate, learn, plan, and execute patient care within the same 3D environment. Ghaderi et al.⁴⁴ piloted a VR-based multidisciplinary orthopedic trauma meeting and demonstrated feasibility, excellent data visualization, and effective decision making. Remote collaboration enables faster and easier methods of interprofessional and multidisciplinary communication, providing an enhanced level of flexibility while breaking down logistical barriers.⁴⁵ As healthcare delivery continues to become more digitized, XR will play an increasingly important role in fostering digital methods of communication for accessible and expedited patient care.⁴⁶

3.3.5.

Telemedicine and patient education

Patients often have difficulty understanding and analyzing medical imaging, leading it to be a poor resource for explaining and describing disease pathology and proposed treatment interventions. Approaches of providing 3D visualizations of imaging findings to explain human anatomy and proposed treatment/procedural strategies have been shown to alleviate anxiety from the patient and promote informed decision-making while also providing an excellent educational environment for both the patient and medical trainees.⁴⁷

3.3.6.

3D interactivity

XR’s primary advantage over traditional medical imaging interpretation lies in its capacity to allow image interpretation in a 3D environment with a high level of interactable experiences for user navigation.⁴⁸ Surgical cases with complex, aberrant anatomy, such as congenital cardiac disease, that require significant knowledge of the disease and the ability to imagine complex 3D structures, benefit from significant advancements in enhanced data interpretation and interactivity with 3D modeling.^49,50 This not only provides a more accurate analysis of patient-specific anatomy and physiology⁵¹ but also provides visual elements that aid localization and can be combined with other localizing technologies.⁵²

3.3.7.

Enhanced visualization and localization

The 3D environment allows for the augmentation of surgical operations with the use of 3D, patient-specific medical imaging models and the superimposition of these models onto the operative field for surgical guidance and navigation.⁹ Immersive technologies allow for improved planning for surgical procedures as well as augmentation of the operation by being able to better visualize internal anatomy in relation to the physical location of the patient. This allows for localization of the pathology and the appropriate setup (incision, instrument placement, etc.) to facilitate the surgical operation and ensure operative efficiency and success.¹⁵ Further, AR systems are well-suited for visualizing thin medical devices (e.g., catheters) that require high-precision tracking. Techniques have been developed that overlay devices onto a user’s field of view, leaving the device functionally unchanged while accurately and responsively relaying its position.⁵³

3.3.8.

Realism/reliability/accuracy

Immersive technology can also be used to improve medical image analysis by immersing the user in realistic 3D models of patient-specific imaging including computed tomography and magnetic resonance imaging.^54,55 This interactive visualization can lead to a better understanding of the anatomical and pathological features of patient-specific diseases crucial for accurate diagnosis and treatment planning.⁵⁶ The interactive, immersive, and realistic environment enables healthcare professionals to visualize and interact with complex, patient-specific anatomy, leading to a deeper understanding of pathology, diagnosis, prognosis, and therapeutics.^57,58

3.3.9.

Complex (or low volume) scenarios

Immersive simulation offers a digital arena in which rare and/or complex training scenarios can be rapidly deployed and shared in the XR environment while minimizing expenditures related to setup and execution. Complex surgical cases can be stored and reviewed collaboratively across disciplines or across institutions to share knowledge and know-how in managing rare or complex disease states.^26,27

In summary, immersive technologies are heralding a new era in healthcare by offering advantages over traditional clinical care methods (Table 3). These technologies merge the digital and physical worlds, granting healthcare professionals enriched visualization, real-time data access, and interactive clinical and training environments to improve patient care delivery. Immersive technologies offer the potential to elevate the standard of clinical care, drive improved outcomes, enhance training, and offer a more personalized and informed patient experience.

Table 3

Benefits of XR in healthcare delivery overall.