An AI program can analyze movement in videos and generate its own sound effects, similar to a Foley artist. The AI, dubbed AutoFoley, tricked a majority of people into believing its sound effects were real.
More:
A study describing AutoFoley was published in IEEE Transactions on Multimedia.
Professor Jeff Prevost co-created AutoFoley with PhD student Sanchita Ghose. They developed two machine learning models, one that extracts image features from video frames and one that analyzes the temporal relationship of objects in different frames to predict what action is occurring. The sound is synthesized to match each model's prediction of motion.
They used AutoFoley to generate sound for 1,000 short movie clips of action, like a horse galloping and fire.
In a survey of 57 college students, 73% of students surveyed chose the first model's AutoFoley clip as having the original sound over the true sound clip.