Machine learning algorithms tend to be specialized — they excel at singular, highly repetitive tasks. (Think generating synthetic scans of brain tumors.) But a new paper published by researchers at the Massachusetts Institute of Technology’s Computer Science and Artificial Intelligence Lab (CSAIL) describes something of an AI polymath: a model that’s equally skilled at both speech andobject recognition.
“We wanted to do speech recognition in a way that’s more natural, leveraging additional signals and information that humans have the benefit of using, but that machine learning algorithms don’t typically have access to,” David Harwath, a researcher at CSAIL and a coauthor on the paper, told MIT News. “We got the idea of training a model in a manner similar to walking a child through the world and narrating what you’re seeing.”
To that end, their system learned to identify objects in an image by associating the words it heard in speech samples with regions in the picture. All the more impressive, it didn’t once fall back on transcription or annotations — it trained solely on pairs of images and audio captions.