Not playing games

This summer, while millions around the world are hunting for virtual creatures known as “Pokémon,” Nhu Nguyen is engrossed in a different application for augmented reality.

As Nguyen, a summer student in the lab of Sunnybrook Research Institute senior scientist Dr. Victor Yang, explains, augmented reality allows users to interact with digital content in their physical environment. What she is working on is the stuff of science fiction: a way of peering into the body in the operating room without any incisions.

Nguyen is developing a computer program to be used with augmented reality glasses with the aim of enhancing surgical navigation during spinal procedures. She is designing software that will project a 3-D hologram of a person’s anatomy onto the glasses. The hologram is based on medical imaging done beforehand.

“It’s like X-ray vision, almost. You can see through the human body, in a sense, because the hologram or image that’s being projected is a reconstruction from the [patient’s] CT scans. By the end of the summer I’m hoping I can make it appear so that my hologram is spatially in the same place as the spine,” says Nguyen, who begins her master of electrical engineering at Ryerson University this fall.

Yang, who is a neurosurgeon at Sunnybrook, challenged Nguyen with addressing a problem he encounters during operations when he implants devices such as pedicle screws and rods to stabilize the backbone. Current surgical navigation technology, which improves accuracy of the placement of implants, requires surgeons to use a nearby computer system that displays the patient’s CT scans and “maps” surgical tools to the CT images. The system requires a second user to adjust the screen and flip between images. The risk is that in turning to look at the screen there may be a loss of spatial awareness at the surgical site, which could be hazardous should navigation be inaccurate.

“The surgeon is always turning his head to check where his tool is; he’s looking side to side. With this there’s no looking back and forth. Everything is in front of you. You know where everything is at all times,” she says.

To generate the hologram, Nguyen uses software that assembles CT scans, which are simply cross-sectional X-ray images of the spine into a 3-D model. She then imports the 3-D model into another computer program—gaming software, to be precise—for which she is developing algorithms, so that the model can be projected wherever the user is looking. The software would work with the augmented reality headset’s feature-detection technology, which “maps” an environment based on objects in the real world. The aim is to make the hologram to scale and appear where the patient’s spine would be in real life.

Nguyen is also writing software that will let the user manipulate the hologram using finger gestures. The technology with which she is working has a camera at the front of the glasses that detects hand movements such as the “clicking” of a computer mouse, which would enable surgeons to handle the model. She demonstrates her work by pulling up a hologram of a spine, which is solid and lifelike, and “placing” it on a table in the meeting room in which we meet.

Nguyen has made good progress on the project, in part because she was able to purchase the augmented reality technology, which is made by Microsoft, much sooner than expected. The product is not available for sale in stores. It is being sold through Microsoft’s website to developers who must submit an application explaining how their usage will “contribute to holographic computing.” Accepted applicants are then invited to purchase the technology. The process of vetting applications can take months, but luckily she was able to get her hands on the product in a few weeks.

Even during her down time, Nguyen loves researching how computer vision, which uses algorithms to mimic and even exceed human vision, can be applied in the operating room. The project doesn’t feel like work, she says. “It’s almost like a hobby. It’s just really fun, and I enjoy myself.”

Nhu Nguyen received a D+H Summer Studentship Award.