Real-time tomographic holography for augmented reality
Abstract
The concept and instantiation of real-time tomographic holography (RTTH) for augmented reality is presented. RTTH enables natural hand–eye coordination to guide invasive medical procedures without requiring tracking or a head-mounted device. It places a real-time virtual image of an object’s cross section into its actual location, without noticeable viewpoint dependence (e.g., parallax error). The virtual image is viewed through a flat narrowband holographic optical element (HOE) with optical power that generates an in-situ virtual image (within 1m of the HOE) from a small spatial light modulator display without obscuring a direct view of the physical world. Rigidly fixed upon a medical ultrasound probe, an RTTH device could show the scan in its actual location inside the patient, even as the probe was moved relative to the patient.
------
Most medical imaging augmented-reality systems depend on tracking of the viewer relative to either the world and/or to some image-scanning device [1]. To avoid the undesirable technical and ergonomic requirements imposed by use of a tracking system [2], some researchers have previously mechanically mounted a 2D display and a half-silvered mirror in the proper positions on an ultrasound probe such that by the law of reflection the mirror produces a virtual image of the 2D display that is in situ, i.e. precisely located (registered) in 3D space with the slice of tissue scanned by the ultrasound probe, regardless of viewpoint or probe position [3, 4]. An entire 3D volume can be interrogated tomographically (i.e. in a series of slices) by moving the ultrasound probe to show multiple ultrasound slices in their correct 3D locations; the correct ultrasound data will be displayed at each point in the entire 3D volume inside the patient. This see-through autostereoscopic augmented-reality display technique has been termed Real-Time Tomographic Reflection (RTTR) [4].
Here we present an extension of the RTTR concept, which we first proposed but had not yet implemented in [5]. By replacing the semi-transparent mirror with a narrow-band holographic optical element (HOE), as shown in Figure 1, we are able to maintain all of RTTR's above benefits without our device geometry being constrained by the law of reflection and without the substantial attenuating effects of a semi-transparent mirror. Our approach, analogously termed real-time tomographic holography (RTTH), can produce a 2D virtual image that is always correctly positioned and larger than the source spatial light modulator (SLM) display, with a less obscured view of the patient and with substantial flexibility in the position and orientation of the HOE and the (monochromatically illuminated) SLM. These benefits derive directly from the flexibility of an HOE. A flat-surface (e.g. volume phase) HOE appears transparent if it has sufficiently narrow bandwidth (Bragg window). An HOE can act as a focusing optic and simultaneously as a diffraction grating.
See the full article here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3022476/
Video from Erick Moen: Stetten has been working on stuff like this for a while. It only works with monochromatic light, but it is widely adopted in the medical field. My brother is a PA and told me they use it in some hospitals to find veins for blood draws, etc. I found a video of it: https://www.youtube.com/watch?time_continue=114&v=ERle45xg9l8