AI learns to predict human behavior from videos

By Laura Berrill
New study from machine learning engineering researchers unveils computer vision and AI techniques giving machines sense for what will happen next

In a new study by researchers at Columbia Engineering, a computer vision technique for giving machines a more intuitive sense for what will occur next, by leveraging higher-level associations between people, animals and objects.

Assistant professor of computer science at Columbia, Carl Vondrick, said their algorithm was a step toward machines being able to make better predictions about human behaviour and therefore better coordinate their actions with our own. He added the results meant the opening of a number of possibilities for human-robot collaboration, autonomous vehicles and assistive technologies.

Learning by ‘watching’

The AI systems involved learn to predict hundreds of activities from handshaking to fist bumping, after analysing thousands of hours of movies, sports games and TV shows. When they can’t the system finds the higher-level concept that links them, in this case the word ‘greeting’. Columbia’s algorithm is the first to learn the capability to reason abstractly about future events, despite the fact that not everything in the future can be predicted.


The mathematical framework developed by the researchers enables machines to organise events by how predictable they are in future. An example is swimming and running as forms of exercise. The new technique learns how to categorise such activities on their own. This could move computers closer to being able to size up a situation and make a nuanced decision, instead of a pre-programmed action and is a critical step in building trust between computers and humans. If machines are able to understand and anticipate human behaviours, they could seamlessly assist people in daily activities.

The possibilities for improving people’s lives

Researchers add the next step was to verify the system works outside the lab and if it can work in diverse settings. If so, there are many possibilities to deploy machines and robots that may improve people’s safety, health and security. The group plans to continue improving the algorithm’s performance with the use of datasets and computers, plus other forms of geometry. The study is titled ‘Learning the predictability of the future’.



Featured Articles

Virgin Atlantic accelerates AI transformation with Amperity

Leading enterprise customer data platform will help Virgin Atlantic leverage a data-driven strategy to deliver highly personalised customer experiences

Sustainability LIVE: Event for AI leaders

Featuring experts from companies including Microsoft, Google, AWS, Meta and Tech Mahindra, Sustainability LIVE offers a number of sessions for tech leaders

VMware and NVIDIA AI Foundation unlocks business potential

VMware and NVIDIA have partnered in a private AI foundation with a wide range of offerings, to aid businesses to better adopt and customise AI models

TimeAI Summit Oct 2023 to unite tech giants and visionaries


MIT suggest generative AI is democratising AI access

Machine Learning

How AI could help airlines mitigate contrail climate impact

Machine Learning