Facebook teams with researchers to transfer your smiles and frowns into virtual reality.
The system tracks the motion of a person’s mouth using a 3-D camera attached to the headset with a short boom. Movements of the upper part of the face are measured using strain gauges added to the foam padding that fits the headset to the face. After the two data sources are combined, an accurate 3-D representation of the user’s facial movements can be used to animate a virtual character, whether that’s an ersatz person or something other than human (see video).
That could make inhabiting and interacting in virtual worlds more compelling, says Hao Li, an assistant professor at the University of Southern California who led the project. “To get a virtual social environment, you want to convey this behavior to other people,” he says. “This is the first facial tracking that has been demonstrated through a head-mounted display.” Li was named one of MIT Technology Review’s Young Innovators in 2013.
Li recently made a system that tackles the more challenging task of making a realistic 3-D re-creation of a person’s hairstyle. His software can do it using only a single photo. That project and the face-tracking research will both be presented at the Siggraph computer graphics conferencein Los Angeles this August.