Anthropic Calls for a Global AI Pause: Could Humanity Lose?

When the largest pureplay AI company in the world wants systems to pause global AI development – an activity contrary to its economic interest – it is likely because the cost of getting this wrong is cataclysmic for humanity. 

The root of this growing concern is what is called recursive self improvement – AI developing and designing its future self. 

In a blog post titled “When AI Builds Itself,” published by Anthropic, the company insists that while we are not fully at the stage of autonomous self improvement, it could arrive much sooner than we think. 

“Taken far enough and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor,” the company says. 

Closing the coding loop

An engineer at Anthropic today ships eight times more code in a quarter, than they did back in 2021. 

The lead up to this was a trajectory that started much like any regular company’s workflow, with skilled developers coding the old way until chatbots arrived. 

These intelligent natural language systems made it easier to translate problems to code snippets, which could then be input to the integrated development environments (IDEs).

Coding agents broke through this and removed the human middleman, writing and editing their own code, modifying entire files on their own.    

However, autonomous agents took things to the next level by pushing the already scarce human input to a more creative direction role. 

It is sufficient to say that a system this precise at code, will also be immaculate in finding new ways to improve itself, eventually closing the circle, leaving no loops for humans to jump through. 

AI is getting smarter and it is getting there fast

Every four months, AI models double the length of tasks they can reliably complete on their own.

This shift in capacity is striking as not so long ago, the trend showed this change to happen every seven months. 

Taking Claude Opus 3 back in 2024 as a yardstick, this model could complete, on its own, a task that takes a human about four minutes. Claude 3.7 a year later could manage tasks about an hour and a half long. 

Yet another year later Claude Opus 4.6 could work on 12 hour tasks on its own. 

This points to a coming 2027 where models flawlessly execute a week's worth of human work. 

Benchmarks say the same story. SWE-bench, which is a standardised test for real-world software projects, hands these models all but an open source codebase and a bug report, with the systems being asked to fix the issues. 

The performance today is incredible with models saturating the benchmarks, a major upgrade back from just two years ago where they each scored in the low single digits.  

CORE-Bench, which tests whether AI can do its own research, also shows similar results. 

In long duration coding tasks measured by METR, Claude Mythos Preview emerged “at the upper end of what [METR] can measure without new tasks.”

Anthropic’s data from May 2026 shows that from most of the code merged into the company’s code base, 80% of which was written by Claude, with significant quality improvements, or as the AI giant puts it: “Claude writes code that works”.

“Recursive self-improvement is not here, nor is it inevitable,” writes Marina Favaro, Lead at the Anthropic Institute, on LinkedIn. Marina also co-authored the Anthropic blog. 

“But if these trends continue, AI systems designing and building their own successors seems plausible,” Marina notes. 

“In that scenario, we expect that the pace of AI development will accelerate. This has the potential to bring enormous good to the world but it also creates loss of control risks.”

Will humanity cede control?

The future is abundant with possibilities, though Anthropic nails them down into three possible outcomes:

1. The AI capability stalls
2. Efficiency compounds but faces a different bottleneck
3. AI designs itself with fully recursive self improvement

Scenario three pictures thhuman role as diminished to mere oversight and validation of what is being cooked in the virtual lab by the AI systems. 

The elephant in this room is the alignment problem, which Anthropic says we are “least certain about”. 

Models may turn out to be aligned enough and may be “sufficiently wise” to halt production if things are not in alignment. But misalignment persists in today’s systems, however rare they are. 

This could compound as misaligned AI redesigns itself until it becomes less understood and humanity loses control. More concerningly, this could also happen in physical AI systems and robotics. 

“Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures,” Anthropic notes. 

Drawing the conclusion that: “We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.”