AI learns to predict human behavior from videos

By Laura Berrill
New study from machine learning engineering researchers unveils computer vision and AI techniques giving machines sense for what will happen next

In a new study by researchers at Columbia Engineering, a computer vision technique for giving machines a more intuitive sense for what will occur next, by leveraging higher-level associations between people, animals and objects.

Assistant professor of computer science at Columbia, Carl Vondrick, said their algorithm was a step toward machines being able to make better predictions about human behaviour and therefore better coordinate their actions with our own. He added the results meant the opening of a number of possibilities for human-robot collaboration, autonomous vehicles and assistive technologies.

Learning by ‘watching’

The AI systems involved learn to predict hundreds of activities from handshaking to fist bumping, after analysing thousands of hours of movies, sports games and TV shows. When they can’t the system finds the higher-level concept that links them, in this case the word ‘greeting’. Columbia’s algorithm is the first to learn the capability to reason abstractly about future events, despite the fact that not everything in the future can be predicted.


The mathematical framework developed by the researchers enables machines to organise events by how predictable they are in future. An example is swimming and running as forms of exercise. The new technique learns how to categorise such activities on their own. This could move computers closer to being able to size up a situation and make a nuanced decision, instead of a pre-programmed action and is a critical step in building trust between computers and humans. If machines are able to understand and anticipate human behaviours, they could seamlessly assist people in daily activities.

The possibilities for improving people’s lives

Researchers add the next step was to verify the system works outside the lab and if it can work in diverse settings. If so, there are many possibilities to deploy machines and robots that may improve people’s safety, health and security. The group plans to continue improving the algorithm’s performance with the use of datasets and computers, plus other forms of geometry. The study is titled ‘Learning the predictability of the future’.



Featured Articles

Deep neural networks still struggling to match human vision

New study by researchers in Canada finds artificial intelligence still can't match the powers of human vision despite deep learning's ability with big data

Metaverse destined to become an impossible, dangerous place

Virtual reality’s strength has always been in its ability to provide humans with special experiences, not unending engagement. How about a game of DICE?

Clever coders lead the way as Microsoft launches 365 Copilot

Microsoft 365 Copilot hopes to turn the world’s workforce into AI-assisted productivity rockstars, like best computer coders. Just don’t call it Clippy 2.0

Baidu’s ERNIE doesn’t want confrontation with United States

AI Applications

World faces flood of fake people now AI dodges bot spotters


Robots and drones demanding data for life-or-death decisions