3D4Medical’s pioneering videos are almost literally mind blowing, thanks to an “Explode” feature that moves the segments of an organ apart, allowing each to be studied in isolation. The Irish Medtech company is ahead of its time, leading the pack in the use of mixed and Augmented Reality (AR) in medical education, over the past three years.
Timely Research & Development and Innovation investments helped Medtech visionary, CEO John Moore, to push towards the bleeding edge of digital health technologies. The journey into Augmented Reality began with “Project Esper: Mixed Reality Anatomy Learning”, a video 3D4Medical made three years ago, that showcased capabilities the company had then developed, as well as where it aimed to go.
Watch Project Esper to see this remarkable technology for yourself.
The video’s YouTube comments speak for themselves.
Irish Medtech Innovators
So how does this apparent magic actually work? John explains,
Our application of augmented and mixed reality is based on 3D models that we invested heavily in to ensure that they’re medically accurate to the point of photorealism.
John describes the combined sense of reality and unreality that is a hallmark of the 3D4Medical experience and has wowed such reputable international partners as the Mayo Clinic.
A user can take a heart, for example, and place it on an everyday surface like a kitchen table and see it through an iPad. Of course, the heart isn’t actually there, but the room is. So you can view a photorealistic beating organ in your own home or office.
When you watch a 3D4Medical video, that attention to detail stuns just as much as the potential of the technology. It feels futuristic to examine the body’s hidden vital pieces with such precision. The organ you see in their latest video, the Complete Heart, is the most advanced 3D realistic heart which has ever been developed. It took the company several years with an extensive team and resources to develop the technology necessary. The heart model alone took almost a year to develop with 100 moving parts that allow it to look completely photorealistic and in such a natural manner.
3D4Medical operate within an ecosystem of Irish Medtech pioneers whose innovation have taken them global. As John describes,
Irish Medtech companies are now seen as among the best in the world. People know we can compete with anyone in Silicon Valley. We work closely with all the US tech giants in the States and have a particularly close relationship with Apple. We work in their lab, with expertise passing both ways.
For Medtech pioneers like 3D4Medical, investment in innovation is essential:
The funding for Research & Development that we received from Enterprise Ireland allowed us to reach another level that we may not have attained otherwise. Innovation pushed us onto the world stage, where we became recognised as international leaders in medical applications.
Saving costs and creating efficiencies in Medtech
For medical professionals and students, the technology’s immediate applications are highly practical. Innovations in augmented and mixed reality create efficiencies that revolutionise medical learning. The technology extends the lifespan of events that occur in the real world, while fundamental parts of a medical education may no longer need to occur physically at all. John explains,
With the technology we have today, users can already virtually cut and dissect organs and walk around them. Professors can record a virtual or a physical dissection and push it out to students, who can view the recordings in augmented reality.
In pedagogical terms, these applications give students huge control over how they absorb and learn from dissections.
Students aren’t just sitting passively in a theatre or lecture room. They can walk around at the same time and see exactly what is happening from every angle they want to.
But the use of Augmented Reality in medical learning doesn’t just benefit students and professors. There are huge potential cost savings for institutions and professional bodies.
Augmented reality potentially displaces cadaveric dissection, which can be expensive. In Ireland, dissection is not as expensive as it is in some countries because many people donate their bodies to science after death. In countries with lower levels of donation, managing dissection can be expensive. Augmented reality offers a clean way to learn and economies and efficiencies when compared with traditional methods.
The question of realism intrudes here also. How ‘realistic’ is cadaveric dissection? How ‘realistic’ is the virtual version offered by Medtech?
When you examine a cadaver, the body is, of course, dead. Students and doctors dissect something that is already dead. With a technology like our Complete Heart, dissection occurs while the organ is actually beating. Our patented technology allows users to cut the organ while it is beating. They can examine something like endocarditis, growths on the heart’s valves, and see how they affect it in real time. They can slow down the body’s processes to examine the impact a problem with a valve has on blood flow. Everything can be performed in real time, as if the model was alive, compared to dissecting something dead, in which case there is no blood flow or movement.
Waiting for the hardware
When you think about medical learning, one of the first things to spring to mind are heavy, expensive books. The technology that 3D4Medical pioneers has been embraced by publishers who recognise collaboration may spare their obsolescence. The future of augmented and mixed reality also departs from the iPad as viewer. Realising the full potential of 3D4Medical’s innovations is reliant on hardware keeping pace. John explains,
People like us software developers depend on innovations in hardware. Right now, we are close partners with Apple. The technology they deliver with the iPhone and iPad is fantastic. But we won’t have surgeons doing operations while looking through an iPad either. The next big stage in development lies in the use of mixed reality glasses with cameras. Users will wear what look like normal shades but when they look through them artificial intelligence explains what they are seeing.
3D4Medical’s applications will soon be integrated with Microsoft’s mixed reality smartglasses, HoloLens:
When professors record a dissection or an operation and push it out, students use a HoloLens headset to view it there and then. There could be several students wearing headsets at the same time, in the same room, walking around the dissection or operation, viewing it from whatever angle they want, and understanding it better. When students play back recordings, HoloLens allows them to hear the professor’s voice as they describe what they are doing on the model at the same time. We will be releasing an app which works like that in early 2018.
Combining Augmented Reality and Artificial Reality is the future of Medtech
John may be a visionary but his feet are firmly on the ground. He has an aversion to unsubstantiated hype about the technology’s future. His focus circles back to end users, the practical wins for students, professionals and the ultimate end users, patients. The question he returns to repeatedly is:
What does augmented reality really offer over alternatives? There has been a lot of hype. Too many applications look cool but don’t offer much. There are a lot of gimmicks. Innovation must offer value and improve on what it displaces.
For John, much of the technology’s potential lies, not just in the use of augmented reality but in combining it with Artificial Intelligence (AI).
The future lies in the combination of augmented reality and AI. When a surgeon wearing mixed reality glasses cuts something or looks at an open wound, smartglasses will recognise what they are looking at – that’s an aorta, that’s a vein, that’s a nerve, don’t cut it, and guide the user through the operation, even as a novice. Intelligent glasses will superimpose labels and signposts over what the eye sees to tell the viewer what they are looking at.
While the phrase Artificial Intelligence can seem esoteric, the benefits are, again, immediate. The value for a 3D4medical partner like the Mayo Clinic lies in the elimination of the messiness and challenges of working with physical bodies. John explains,
Some operations release a lot of blood, or a lot of spinal fluid. That creates risk, if a surgeon can’t see the anatomy clearly. They still have to perform the operation, pretty much blinded. There you see the value of being able to superimpose labels or signposts on to the anatomy which is based on a 3D scan of the patient’s body made before the operation. Over the blood or spinal fluid clouding the surgeon’s view will be a photorealistic scan of the body part in question. The risk of cutting a vital nerve or the wrong tissue or leaving something behind is then greatly reduced.
The winners of those improvements are, of course, all of us. Our operations will be safer when surgeons have been trained and assisted to see the body at a level of depth and clarity the human eye alone could never hope to achieve.