Imperial College London
London, UK
www.imperial.ac.uk

In a series of procedures carried out by a team at Imperial College London at St Mary’s Hospital, researchers have shown for the first time how surgeons can use Microsoft HoloLens headsets while operating on patients undergoing reconstructive lower limb surgery.

AR models reveal the patient’s bones and blood vessels. (Credit: Philip Pratt et al. Eur Radiol Exp, 2018)

The HoloLens is a self-contained computer headset that immerses the wearer in ‘mixed reality’, enabling them to interact with ‘holograms’ — computergenerated objects made visible through the visor. In the UK, headsets are currently only available for developers.

The Imperial team used the technology to overlay images of CT scans — including the position of bones and key blood vessels — onto each patient’s leg, in effect enabling the surgeon to see through’ the limb during surgery.

According to the team trialling the technology, the approach can help surgeons locate and reconnect key blood vessels during reconstructive surgery, which could improve outcomes for patients.

“We are one of the first groups in the world to use the HoloLens successfully in the operating theatre,” says Dr. Philip Pratt, a research fellow in the Department of Surgery & Cancer and lead author of the study, published in European Radiology Experimental. “Through this initial series of patient cases we have shown that the technology is practical, and that it can provide a benefit to the surgical team. With the HoloLens, you look at the leg and essentially see inside of it. You see the bones, the course of the blood vessels, and can identify exactly where the targets are located.”

Repairing Damage

Following a car accident or severe trauma, patients may have tissue damage or open wounds that require reconstructive surgery using fasciocutaneous flaps. These flaps of tissue, which are taken from elsewhere on the body and include the skin and blood vessels, are used to cover the wound and enable it to close and heal properly. A vital step in the process is connecting the blood vessels of the new tissue with those at the site of the wound, so oxygenated blood can reach the new tissue and keep it alive.

“Augmented reality offers a new way to find these blood vessels under the skin accurately and quickly by overlaying scan images onto the patient during the operation,” explains Dr Pratt.

Making the Model

The device can be controlled by different signals, including electromyography and voice control. (Credit: EPFL)

In the procedures used to trial the technology, five patients requiring reconstructive surgery on their legs underwent CT scans to map the structure of the limb, including the position of bones and the location and course of blood vessels. Images from the scans were then segmented into bone, muscle, fatty tissue, and blood vessels by Dr. Dimitri Amiras, a consultant radiologist at Imperial College Healthcare NHS Trust (ICHNT), and loaded into intermediary software to create 3D models of the leg.

These models were then fed into specially designed software that renders the images for the HoloLens headset, which in turn overlays the model onto what the surgeon can see in the operating theatre.

Clinical staff are able to manipulate these AR images through hand gestures to make any fine adjustments and correctly line up the model with surgical landmarks on the patient’s limbs, such as the knee joint or ankle bone.

The Surgeon’s View

“The application of AR technology in the operating theater has some really exciting possibilities,” says Jon Simmons, a plastic and reconstructive surgeon at ICHNT, led the team who carried out the procedures using the HoloLens headset and augmented reality models. “It could help to simplify and improve the accuracy of some elements of reconstructive procedures.

Limitations with the technology could include errors during the modelling stages as well as the potential for the overlaid model to be misaligned. In addition, the case studies so far have been based on the leg, which has a number of clearly visible surgical landmarks, such as the ankle or knee. Areas without these rigid landmarks, such as abdomen, may be more complicated with a greater potential for movement of blood vessels.

However, the researchers are confident that, once refined, the approach could be applied to other areas of reconstructive surgery requiring tissue flaps, such as breast reconstruction following mastectomy. The next steps include trialling the technology in a larger set of patients, with procedures carried out by teams at multiple centers.