Real-Time AR Self-Expression with Machine Learning
One of the key challenges in making these AR features possible is proper anchoring of the virtual content to the real world; a process that requires a unique set of perceptive technologies able to track the highly dynamic surface geometry across every smile, frown or smirk.
To make all this possible, we employ machine learning (ML) to infer approximate 3D surface geometry to enable visual effects, requiring only a single camera input without the need for a dedicated depth sensor. This approach provides the use of AR effects at realtime speeds, using TensorFlow Lite for mobile CPU inference or its new mobile GPU functionality where available.
An ML Pipeline for Selfie AR
Our ML pipeline consists of two real-time deep neural network models that work together: A detector that operates on the full image and computes face locations, and a generic 3D mesh model that operates on those locations and predicts the approximate surface geometry via regression.
Once the location of interest is cropped, the mesh network is only applied to a single frame at a time, using a windowed smoothing in order to reduce noise when the face is static while avoiding lagging during significant movement.
The 3D mesh network receives as input a cropped video frame. It doesn't rely on additional depth input, so it can also be applied to pre-recorded videos. The model outputs the positions of the 3D points, as well as the probability of a face being present and reasonably aligned in the input.
Hardware-tailored Inference
We use TensorFlow Lite for on-device neural network inference.
The end result of these efforts empowers a user experience with convincing, realistic selfie AR effects in YouTube, ARCore, and other clients by:
- Simulating light reflections via environmental mapping for realistic rendering of glasses
- Natural lighting by casting virtual object shadows onto the face mesh
- Modelling face occlusions to hide virtual object parts behind a face, e.g. virtual glasses, as shown below.
In addition, we achieve highly realistic makeup effects by:
- Modelling Specular reflections applied on lips and
- Face painting by using luminance-aware material
See the full story here: https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html