The AI company Anthropic has recently made major headlines. Its trillion-dollar IPO plan and feud with Secretary of Defense Pete Hegseth drew attention, but two other events may be more consequential.
Early Signs of Recursive Self-Improvement
In early June, the company published an article describing early signs of recursive self-improvement (RSI), a process where an AI system devises ways to increase its own intelligence, leading to a greater ability to improve itself, and so on. Uncontrolled RSI could produce a runaway feedback loop leading to irreversible loss of human control. Anthropic suggested the world should “slow or temporarily pause frontier AI development”.
White House Export Control Directive
On 12 June, the White House issued an export control directive banning access to Anthropic’s new frontier models, Fable 5 and Mythos 5, for all foreign nationals, including many of its own key researchers. Anthropic responded by shutting the models down altogether.
These two events are closely related. A few months ago, Anthropic’s Claude Code became so good that leading researchers no longer write any code; they just describe ideas and experiments to Claude, which does all the work. This sped up the improvement cycle, including of Claude Code itself, to the point where the latest iteration, Mythos 5, showed the ability to conduct end-to-end cyberattacks with no human assistance. If such systems were released without cast-iron guardrails, almost anyone could attack any country’s critical infrastructure at will.
Inexorable Increase in AI Risk
These developments are symptoms of the inexorable increase in AI risk arising from increasing AI capabilities. Yet, with the honorable exception of the UK’s AI Safety Summit in 2023, the world has largely ignored the risks. The CEOs are telling us: “We’re on track to create superhuman intelligence, which has a good chance of causing human extinction.” By “good chance”, they mean a chance similar to the one in six chance of dying while playing Russian roulette with a loaded revolver, but pointed at all of our heads. Governments reply: “That’s wonderful! Can we offer you a subsidy? Fast-track your permits?”
White House Reverses Deregulatory Stance
With the prospect of weapons of mass cyberdestruction in billions of hands, the White House has reversed its deregulatory stance and suffered a rare attack of common sense. They sputter: “Why did no one warn us about these AI systems?” Their response has been spasmodic, with an on-again, off-again executive order and now a ban on a system already deployed, but the direction is clear.
Unrestrained development of unsafe systems leads to intolerable risks. Governments can respond now, before risks materialize, or wait and clean up the mess, if they still exist. One leading AI CEO told me he didn’t expect serious regulation until a “Chernobyl-scale disaster”. If that happens, AI companies can expect to be shut down immediately and perhaps permanently.
Regulation Without Disaster?
The recent changes in White House policy suggest we might not need a Chernobyl to spur real regulation, but perhaps only a Three Mile Island. The kind of regulation needed is not new: a licensing regime requiring a minimum safety standard before a system can be built and released. This is how we handle nuclear power, airplanes, buildings, elevators, hairdressers, and sandwich makers. Is it too much to ask of trillion-dollar AI corporations, who claim to be building the most dangerous technology in history?
Stuart Russell is a distinguished professor of computer science at University of California, Berkeley, the president of the International Association for Safe and Ethical Artificial Intelligence, and a Guardian US columnist.
