PhilNote: It creates a digital twin, hides the physical object from view, and then manipulates the digital twin. It allows you to see the physical object just before you interact with it. It is a mystery how this would be used.
... Researchers at Meta Reality Labs Research and University of Duisburg-Essen devised a method for making real objects virtual so they can be interacted with in real-time. By not only digitizing the objects, but also seamlessly erasing their real counterparts, the method creates a mind-bending blend between the real and virtual worlds. The video below with examples explains it better than I possibly could with text alone:
https://www.youtube.com/watch?v=-GwLu82QLCw&t=281s
The researchers—Mohamed Kari, Reinhard Schütte, and Raj Sodhi—call this concept ‘Scene Responsiveness‘ and published their work late last year at the ACM Symposium on User Interface Software and Technology. ...
See the full story here: https://www.roadtovr.com/meta-reality-lab-ar-research-scene-responsiveness/