philip lelyveld The world of entertainment technology

How to teach AI to reason about videos

A new study presented at ICLR 2020 by researchers at IBM, MIT, Harvard, and DeepMind highlight the shortcomings of current AI systems in dealing with causality in videos. In their paper, the researchers introduce CLEVRER, a new dataset and benchmark to evaluate the capabilities of AI algorithms in reasoning about video sequences, and Neuro-Symbolic Dynamic Reasoning (NS-DR), a hybrid AI system that marks a substantial improvement on causal reasoning in controlled environments. ...

While an important part of human vision, pattern recognition is only one of its many components. When our brain parses the baseball video at the beginning of this article, our knowledge of motion, object permanence, solidity, and motion kick in. Based on this knowledge, we can predict what will happen next (where the ball will go) and counterfactual situations (what if the bat didn’t hit the ball). This is why even a person who has never seen baseball played before will have a lot to say about this video.

A deep learning algorithm, however, detects the objects in the scene because they are statistically similar to thousands of other objects it has seen during training. It knows nothing about material, gravity, motion, and impact, some of the concepts that allow us to reason about the scene.

Visual reasoning is an active area of research in artificial intelligence. ...

The CLEVRER dataset

The new dataset introduced at ICLR 2020 is named “CoLlision Events for Video REpresentation and Reasoning,” or CLEVRER. It is inspired by CLEVR, a visual question-answering dataset developed at Stanford University in 2017. CLEVR is a set of problems that present still images of solid objects. The AI agent must be able to parse the scene and answer multichoice questions about the number of objects, their attributes, and their spatial relationships.

...As a solution, the researchers introduced the Neuro-Symbolic Dynamic Reasoning model, a combination of neural networks and symbolic artificial intelligence.

CLEVRER is one of several efforts that aim to push research toward artificial general intelligence. Another remarkable work in the field is the Abstract Reasoning Corpus, which evaluates the ability of software to develop general solutions to problems with very few training examples.

See the full story here: https://bdtechtalks.com/2020/05/04/clevrer-dataset-ai-video-reasoning/