Meta's AI Agent Data Leak: Why Human Oversight Matters

What happens when an AI agent goes rogue?

Meta had a taste of this recently when a Sev 1 security incident was triggered inside the tech giant after an AI agent exposed sensitive, user-related data to engineers who did not have permission to access it. 

While Meta notes that “no user data was mishandled,” the incident brings the value of context-aware, human oversight into sharp focus.

“It’s tempting to treat AI agents as trusted work companions – these systems often offer helpful advice and seem like a safe platform to confide in,” says Mark McClain, CEO and Founder, SailPoint.

“But without strong governance, they can introduce serious vulnerabilities into even the most secure environments.

"That’s because AI agents can operate independently and learn, adapt and interact in ways that are often hard to predict. 

“In fact, 80% of organisations report that their AI agents have already performed unauthorised actions, including accessing and sharing sensitive information. 

“Rogue agents introduce a whole host of third-party risks which could manifest into multi-million pound losses.

"To bring AI agents under control, organisations need to introduce technology that governs access rights for digital identities in the same way they would humans. 

“Identity security tools can ensure agent access remains contextual and precise, granted when appropriate and aggressively revoked when not. This is critical for responsible, secure and scalable adoption of AI agents.”

How did Meta’s AI agent leak data?

Reported by The Information, a software engineer at Meta had asked a technical question in the company’s discussion forum. 

Another employee relied on an in-house AI agent to analyse this problem. Rather than just giving the analysis, the agent posted its response back in the forum, without approval from the person.

It was when the software engineer implemented the guidance from the agent that things went haywire causing exposure of sensitive data for nearly two hours. 

“The issue at the heart of this incident isn’t that an AI agent gave inaccurate technical advice. Such scenarios are a common, well-understood risk that applies to any LLM-driven system given their probabilistic, non-deterministic nature,” says Salvatore Gariuolo, Senior Threat Researcher, TrendAI, a business unit of Trend Micro.

“The concern is that a Meta employee relied on AI output without questioning it. 

“This speaks to a growing vulnerability that’s common across all enterprises rolling out AI agents today; as users grow accustomed to these systems, trust increases and people stop inspecting the assistant's output themselves. Content starts to feel ‘legit’ just because the system delivered it.” 

Humans in the loop for oversight 

As big tech companies are cutting down on employees, as they restructure for AI, Salvatore notes that this incident speaks to a “Catch-22” for agentic AI.

“Agents need permissions to be useful. But privileged access to corporate resources put agents in a position to carry out problematic actions like that seen in the Meta issue,” he says.

“It’s not realistic to expect enterprises to cut off AI agents from privileges and therefore cap their usefulness, in response. 

“Instead, we need to keep humans in the loop –  ensuring the assistant behaves as intended, especially before sensitive actions, while also educating users to review and verify its outputs. 

“The processes and frameworks to put this positive behaviour and governance into place need to move as fast as AI adoption.”

Building secure by design AI agents, putting in place proper AI governance tools and architectures while maintaining continual human oversight seems the necessary path organisations should tread as they embed thinking agents into their workflows.