AI learns to predict human behavior from videos

By Laura Berrill
New study from machine learning engineering researchers unveils computer vision and AI techniques giving machines sense for what will happen next

In a new study by researchers at Columbia Engineering, a computer vision technique for giving machines a more intuitive sense for what will occur next, by leveraging higher-level associations between people, animals and objects.

Assistant professor of computer science at Columbia, Carl Vondrick, said their algorithm was a step toward machines being able to make better predictions about human behaviour and therefore better coordinate their actions with our own. He added the results meant the opening of a number of possibilities for human-robot collaboration, autonomous vehicles and assistive technologies.

Learning by ‘watching’

The AI systems involved learn to predict hundreds of activities from handshaking to fist bumping, after analysing thousands of hours of movies, sports games and TV shows. When they can’t the system finds the higher-level concept that links them, in this case the word ‘greeting’. Columbia’s algorithm is the first to learn the capability to reason abstractly about future events, despite the fact that not everything in the future can be predicted.


The mathematical framework developed by the researchers enables machines to organise events by how predictable they are in future. An example is swimming and running as forms of exercise. The new technique learns how to categorise such activities on their own. This could move computers closer to being able to size up a situation and make a nuanced decision, instead of a pre-programmed action and is a critical step in building trust between computers and humans. If machines are able to understand and anticipate human behaviours, they could seamlessly assist people in daily activities.

The possibilities for improving people’s lives

Researchers add the next step was to verify the system works outside the lab and if it can work in diverse settings. If so, there are many possibilities to deploy machines and robots that may improve people’s safety, health and security. The group plans to continue improving the algorithm’s performance with the use of datasets and computers, plus other forms of geometry. The study is titled ‘Learning the predictability of the future’.



Featured Articles

Should Tech Leaders be Concerned About the Power of AI?

With insights from Blackstone CEO Steve Schwarzman, we consider if tech leaders are right to be anxious about AI innovation and if regulation is necessary

Andrew Ng Joins Amazon Board to Support Enterprise AI

In the wake of Andrew Ng being appointed Amazon's Board of Directors, we consider his career from education towards artificial general intelligence (AGI)

GPT-4 Turbo: OpenAI Enhances ChatGPT AI Model for Developers

OpenAI announces updates for its GPT-4 Turbo model to improve efficiencies for AI developers and to remain competitive in a changing business landscape

Meta Launches AI Tools to Protect Against Online Image Abuse

AI Applications

Microsoft in Japan: Investing in AI Skills to Boost Future

Cloud & Infrastructure

Microsoft to Open New Hub to Advance State-of-the-Art AI

AI Strategy