How Extended Reality Is Transforming Surgery

That nagging back pain has become too intrusive to ignore. A patient has tried conservative remedies like NSAIDS and cortisone injections with little success. They need surgery. But they’re worried. More than 4,000 preventable mistakes occur in surgery every year, and at least 39 times a week a surgeon leaves foreign objects like towels or sponges inside their patients, according to a still-relevant 2012 study by Johns Hopkins. ¹

So they seek out an experienced, competent surgeon who has successfully completed that particular surgery many, many times. Surgeons in mid-career between the ages of 35 and 50 with 10 years of experience are the safest for patients, according to a peer-reviewed study published in BMJ. ²

But how did these surgeons actually get this experience without harming their patients while they perfected their technique? For many years, the solution has included training simulators. Simulations reduce the risks associated with surgical training by allowing novice surgeons the opportunity to practice before applying their skills to a live patient. Makes sense.

There are simulators for virtually every training need. For instance, laparoscopic trainers are used to teach the intricacies of gall bladder or colon surgery. But most traditional simulators have limitations. For example, many simulators have been criticized for presenting tasks that are too simple and not closely related to real procedures.

So how can new surgeons safely hone their skills? The answer lies in emerging technology with roots in the world of gaming that seamlessly combines the physical and digital worlds.

Extended Reality a Boon to Medicine

The term virtual reality (VR) was coined back in 1982, referring to a purely digital world. Users see and interact with an environment that is entirely computer generated. By contrast, augmented reality (AR) is a mix of the real world with the virtual, combining elements of both with the goal of enhancing how the real world is experienced. The complete spectrum from VR to AR is referred to as mixed reality (MR) or extended reality (XR). XR merges the real and digital worlds and allows them to interact in real time.

Fueled by the availability of low-cost hardware, such as the Oculus Rift and the Microsoft HoloLens, the field of XR has seen explosive growth over the past several years as developers take advantage of software libraries like Unity MARS to create AR and VR systems for a spectrum of applications ranging from consumer games to aerospace applications.

In the healthcare realm, XR simulators that combine an intuitive interface with realistic graphics and haptic feedback can provide a rich, immersive training environment, allowing surgeons to safely practice everything from using new equipment to executing unfamiliar surgical techniques.

The Future of Surgery

While the applications of these technologies are nearly limitless, several key areas have started gaining momentum. Specifically, XR offers surgeons and other medical professionals the chance to deliver a better patient experience by improving every phase of the surgical process, from preop planning to the surgery itself to postop care.

Preoperative surgical planning. Currently, much preoperative surgical planning is done using information gathered by computed tomography (CT) or magnetic resonance imaging (MRI) techniques, which have been a part of the surgeon’s toolkit since the 1970s but are often limited in use. Not only must care be taken with CT techniques to limit the amount of radiation to which a patient is exposed, the data is often not illustrative enough to truly inform the surgeon. That’s because while sets of sequential 2D images are acquired and assembled into a single 3D dataset used to visualize the patient’s anatomy, CT techniques typically only allow a surgeon to view a single plane or slice of the data at a time. In addition, many CT techniques only allow for limited interactivity (see Figure 1).

As a result, surgeons must view CT data during preoperative planning sessions and form their own mental model of what lies under the patient’s skin rather than having that data easily accessible during surgery. Certainly, the ability to create this mental model is a critical skill and improves with practice. Yet even the most experienced practitioners feel the burden of this additional cognitive load surrounding what is already a complex set of tasks.

This is where VR, in particular, can help. Preoperative planning can be significantly enhanced by taking the data acquired through traditional CT, MRI, and ultrasound scans and visualizing it using VR.

While 3D modeling has been used for over a decade as a way to better visualize a patient’s anatomy, it is only in recent years that computational power and low-cost hardware have allowed VR to make inroads as a preoperative tool. There have been a number of studies conducted to prove the effectiveness of using VR to aid in preoperative planning. For instance, a study on spinal surgery conducted by De Salvatore compared 60 patients, where half had surgery using traditional preoperative planning methods and the other half using a technique where CT scans were composited into 3D models and then viewed using VR hardware. The results revealed significant reductions in both the length of the operations and in the amount of bleeding a patient experienced during surgery when VR was used in the planning process.

Intraoperative surgical assistance. Moving beyond VR, AR has also emerged as a promising technology for in-theater surgical use. In this scenario, use of AR typically involves blending 3D imagery acquired preoperatively with reference points marked on the patient’s body. This process allows for alignment of computer-generated imagery with the patient’s body when viewed through a head-mounted display (HMD) worn by the surgeon.

With this setup, the surgeon does not have to constantly look away at a remote monitor to review preoperative data — as is commonly done today — and instead can retain focus on the patient. Additionally, AR techniques can be used during surgery to provide real-time insight into the placement of tools relative to the area of interest.

Coupling these techniques with multimodal input methods such as voice control and gestural interactions further allows a surgeon to reference preoperative data in an interactive manner without relying on assistants or touching outside equipment. For instance, cardiovascular surgeons demonstrated a system using transesophageal echocardiography to gather data and render a 3D volume onto an HMD during open-heart surgery. The system could be controlled (e.g., zoom, positioning, etc.) in a touchless manner. ³

The discipline of image guided surgery (IGS) takes this idea ever further by incorporating instruments that are tracked during the procedure. IGS also augments the preoperative images with real-time data using such techniques as interventional MRI or ultrasound to provide the surgeon with an accurate anatomical view. These applications have been used successfully in neurosurgery for several years and are now gaining acceptance in other fields, such as sinus surgery.

Another option for simulating surgical training utilizes a digital twin, a virtual representation of an object that spans its life cycle; is updated from real-time data; and uses simulation, machine learning, and reasoning to help with decision-making. According to Unity Technologies — which has developed the largest platform to power games and virtual experiences and offers tools and frameworks to build VR and AR applications — using a digital twin to simulate training accelerates the learning curve of inexperienced new surgeons and can successfully transfer the burden of the surgical training learning curve from patients to technology.

“A mixed-reality simulation allows surgical students or professionals to practice and prepare with precision for a surgery they may conduct on a real patient,” says Unity Senior Strategist Greg Melling. “A Unity real-time 3D digital twin allows for patient data and procedure guides to be augmented into the surgical environment lowering patent risk while reducing procedure time.”

Postoperative care and beyond. XR tech can also be used to enhance postop care. For instance, supplementing recovery treatments with AR could allow patients to access quantitative data about how quickly they’re improving during physical therapy, knowledge that can further speed up their recovery. Other use cases for XR include everything from enhancing the telemedicine experience to providing mental health patients with aversion therapy to cope with phobias and other conditions. ⁴

The Takeaway

XR applications, which facilitate training that is hyper-realistic and safe, are perfectly suited to healthcare professionals looking to solve complex problems or mitigate dangerous circumstances without putting patients’ lives at risk. As the healthcare industry is constantly looking for new ways to improve patient outcomes and drive down costs, expect to see more XR tools make their way into preoperative planning, surgery, and postoperative recovery — and elsewhere. The potential for this technology is expansive.

Today, forward-looking companies are already drawing upon their expertise in XR, medical device development and intuitive interface design to deliver creative solutions aimed at revolutionizing how doctors learn to perform surgery, and in turn enhance patient care. Look for even more applications of XR in the years to come.

References

1. Johns Hopkins malpractice study: Surgical ‘never events’ occur at least 4,000 times per year.
2. A. Duclos, et al., “Influence of experience on performance of individual surgeons in thyroid surgery: prospective cross sectional multicentre study,” BMJ, 2012; 344: d8041.
3. J.D. Kasprzak, “First-in-man experience with real-time holographic mixed reality display of three-dimensional echocardiography during structural intervention: balloon mitral commissurotomy,” European Heart Journal, Volume 41, Issue 6, 7 February 2020, p 801
4. Y. Choy, et al., “Treatment of specific phobia in adults,” Clinical Psychology Review, Vol. 27 , Issue 3 , April 2007 , pp 266–286.

This article was written by Jeff LeBlanc, Director of Solutions Engineering, and Stephanie Van Ness, Assoc. Director of Marketing and Chief Storyteller, at Integrated Computer Solutions, Waltham, MA. Contact: This email address is being protected from spambots. You need JavaScript enabled to view it. or This email address is being protected from spambots. You need JavaScript enabled to view it.. For more information, visit here .

---