New Technology Uses Mouth Gestures to Interact in Virtual Reality
The proliferation of affordable virtual reality head-mounted displays provides users with realistic immersive visual experiences. However, head-mounted displays occlude the upper half of a user's face and prevent facial action recognition from the entire face. To combat this issue, Binghamton University Professor of Computer Science Lijun Yin and his team created a new framework that interprets mouth gestures as a medium for interaction within virtual reality in real-time.
Yin's team tested the application on a group of graduate students. Once a user put on a head-mounted display, they were presented with a simplistic game; the objective of the game was to guide the player's avatar around a forest and eat as many cakes as possible. Players had to select their movement direction using head rotation, move using mouth gestures and could only eat the cake by smiling. The system was able to describe and classify the user's mouth movements, and it achieved high correct recognition rates.
The paper, "Partially occluded facial action recognition and interaction in virtual reality applications," was presented at the 2017 IEEE International Conference on Multimedia and Expo.
See the full story https://www.rdmag.com/news/2017/10/new-technology-uses-mouth-gestures-interact-virtual-reality