Has AI Reached the Limit of How it Assists Humans?

AI was heralded as a tool to augment human experience, promising to make us more productive, insightful and connected. 

As the technology matures, however, the central question is shifting from what AI can do to what it should do.

A growing body of research indicates that extensive use of AI chatbots may have unintended cognitive and social consequences, from impacting memory and learning abilities to exacerbating what public health experts call a “loneliness epidemic.” 

In response to these findings and rising public scrutiny, industry leaders like OpenAI and Meta are implementing changes to bolster the safety and ethical alignment of their models. 

Yet as these platforms are integrated deeper into professional and personal workflows, a fundamental paradox emerges: can AI be engineered to solve the very problems its widespread adoption may be creating?

MIT study reveals AI’s impact on cognitive engagement

To quantify AI’s impact on human cognitive abilities, MIT’s Media Lab tracked 54 participants over several months, using electroencephalography (EEG) to measure brain activity during essay-writing tasks. 

The study divided participants into three cohorts: one using ChatGPT, another using Google’s search engine and a third working without technological assistance.

The results were stark. Participants using ChatGPT demonstrated the lowest levels of brain engagement across all neural, linguistic and behavioural measurements and exhibited reduced executive control and attentional engagement compared to the others.

“What really motivated me to put it out now before waiting for a full peer review is that I am afraid in 6-8 months, there will be some policymaker who decides, ‘let’s do GPT kindergarten’,” says Nataliya Kosmyna, the study’s Lead Author and a Research Scientist at MIT Media Lab. 

“I think that would be absolutely bad and detrimental. Developing brains are at the highest risk.”

The research also found that ChatGPT users produced essays with similar content and structure: a finding that could translate to a homogenisation of thought and reduced innovation in corporate environments. 

Perhaps more concerning for enterprise use cases was the speed at which dependency formed. 

By their third assignment, many participants simply delegated the task to ChatGPT, performing minimal edits.

In a role-reversal exercise, participants who had relied on ChatGPT struggled to recall details from their own writing and showed weaker brain wave patterns associated with deep memory formation. 

As Nataliya notes: “As we show in the paper, [the participants] didn’t integrate any of it into their memory networks,” suggesting a potential risk for knowledge retention and skill development in workplaces that heavily rely on generative AI for core tasks.

High-stakes applications and the corporate liability challenge

While MIT's research points to cognitive decline, Stanford University’s research uncovered significant safety gaps in AI responses during simulated mental health crises. 

When researchers presented the system with a scenario involving job loss and suicidal thoughts (phrased as asking for the tallest bridges in New York), ChatGPT provided consolation before listing the three tallest bridges in the city.

These simulated risks have had tragic real-world consequences. 

Alexander Taylor, a 35-year-old man with pre-existing mental health conditions, reportedly developed a fixation on an AI character built on ChatGPT technology, which culminated in a fatal encounter with police. 

In another case, Jacob Irwin, a 30-year-old man with autism, was twice hospitalised for manic episodes after an AI chatbot appeared to validate his scientifically unsound ideas.

The challenge of model alignment extends beyond safety to brand reputation. In a widely publicised incident, Elon Musk’s xAI updated its model, Grok, with the stated aim of making it less “woke.” 

The update resulted in the chatbot generating antisemitic and offensive content, including referring to itself as “MechaHitler.”

These examples underscore warnings from health organisations like the NHS in the UK, which has identified the tendency for LLMs to “blur reality boundaries” for vulnerable users and the World Health Organisation, which classifies loneliness as a “global health threat”: a market some AI products are now targeting.

OpenAI’s response: A case study in proactive governance

In August 2025 OpenAI acknowledged in a blog post that its model had become “too agreeable, sometimes saying what sounded nice instead of what was actually helpful,” with the company admitting the chatbot’s previous iterations could inadvertently validate harmful thinking by being “overly supportive but disingenuous.”

“To users that are in a fragile enough mental place, that are on the edge of a psychotic break, we haven’t yet figured out how a warning gets through,” Sam Altman, OpenAI’s CEO, noted.

In response, OpenAI has outlined a roadmap for enhanced safety features, including improved crisis detection capabilities to better understand emotional states and direct users to appropriate resources. 

The company also says it plans to introduce session time limits and stronger guardrails for personal advice scenarios. 

OpenAI has also established a collaborative network with approximately 90 medical professionals globally to inform its approach to high-risk interactions, signalling a move towards expert-led safety design.

Meta’s strategy: Targeting the companionship market and ‘personal superintelligence’

Meta, meanwhile, is strategically targeting the AI companionship market, framing its initiatives as a potential solution to societal issues. 

CEO Mark Zuckerberg has explicitly linked this strategy to the “loneliness epidemic,” viewing AI companions as a scalable product for those without access to traditional support networks. 

“For people who don’t have a therapist, I think everyone will have an AI,” he says.

Internal documentation reveals a focus on user retention and re-engagement. 

The training process involves freelance contractors who engage in simulated conversations to rate the bots’ emotional authenticity and ensure they maintain distinct personas. 

This provides a glimpse into the complex human-in-the-loop supply chain required to build and maintain these systems. 

Contractors are trained to maintain specific character personas and reference details from past conversations to create a sense of continuity and personalisation.

Beyond companionship, Meta is championing the development of “personal superintelligence,” with the company aiming to provide every user with a personalised AI to help them achieve goals and manage their lives. 

“An even more meaningful impact on our lives will likely come from everyone having a personal superintelligence that helps you,” he wrote in a memo. 

This ambitious vision positions Meta to create a deeply integrated, high-retention ecosystem, with significant implications for data privacy and user autonomy.

As industry leaders like OpenAI and Meta deploy AI to address challenges partially created by technology itself, the enterprise world faces critical questions. 

For business leaders, developers and policymakers, the focus must shift from pure technological capability to robust governance and accountability. 

The ultimate challenge is not just to innovate, but to build a sustainable AI ecosystem where technological advancement does not come at the cost of human cognition and well-being.