Facebook announces project to solve AI first-person views

By Laura Berrill
Facebook announces Ego4D, a project aimed at solving AI research challenges in “egocentric perception,” or first-person views

The company says the aim of Ego4D is to teach AI systems to comprehend and interact with the world like humans do, as opposed to the third-person - the  way that most AI currently does.

Computer vision models have historically learned from millions of photos and videos captured in third-person. Facebook believes next-generation AI systems might need to learn from a different kind of data: i.e. videos that show the world from the center of the action, in order to achieve egocentric perception.

Global first-person video research

Ego4D brings together a consortium of universities and labs across nine countries, which collected more than 2,200 hours of first-person video featuring more than 700 participants in 73 cities going about their daily lives. It was funded through academic grants to each of the participating universities. Researchers from Facebook Reality Labs - Facebook’s AR and VR-focused research division - also used Vuzix Blade smartglasses to collect an additional 400 hours of first-person video data in staged environments in research labs.

Collecting the data

Kristen Grauman, lead research scientist at Facebook, said of the project: “For AI systems to interact with the world the way we do, the AI field needs to evolve to an entirely new paradigm of first-person perception. That means teaching AI to understand daily life activities through human eyes in the context of real-time motion, interaction, and multisensory observations.”

Ego4D is designed to tackle challenges related to embodied AI, which is a field aiming to develop AI systems with a physical or virtual embodiment, like robots. The researchers hope to improve the performance of AI systems like chatbots, robots, autonomous vehicles, and even smart glasses that interact with their environments, people and other AI.

And in an effort to diversify Ego4D, Facebook says that participants were recruited via word of mouth, ads and community bulletin boards from the U.K., Italy, India, Japan, Saudi Arabia, Singapore, and the U.S. across varying ages, professions and genders (45% were female, one identified as nonbinary, and three preferred not to say a gender). The company also says it is working on expanding the project to incorporate data from partners in additional countries, including Colombia and Rwanda. However, Facebook declined to say whether it took into account accessibility and users with major mobility issues. 

The company went on to say the Ego4D university consortium will release its data in the coming months. It also plans to launch a challenge early next year inviting researchers to develop AI that understands the first-person perspectives of daily activities.



Featured Articles

Need for responsible AI in some of the world’s largest banks

Research shows one-third of North America and Europe’s largest banks lack transparency and are not publicly reporting on their AI development

Lenovo: Employees prefer mix of AI and human IT support

New Lenovo survey shows 91% of employees believe they would be more productive when their IT issues at work are resolved quickly and effectively

Kyndryl’s Data and AI Console to simplify data management

Data-driven solution expands and increases observability and insights, while enhanced data governance helps identify irregularities and threats

Deep neural networks still struggling to match human vision

Data & Analytics

Metaverse destined to become an impossible, dangerous place


Clever coders lead the way as Microsoft launches 365 Copilot

AI Applications