A spate of serious safety incidents have increased fears about the power and impenetrability of the most advanced models
Picture humanity in a boat being swept down a raging river, praying there is no Niagara Falls ahead. Or imagine standing with the pioneering physicists in 1942 before they triggered the first self-sustaining nuclear fission chain reaction beneath a Chicago stadium.
These were two of the analogies used this week by Prof Robert Trager, an expert in the nascent field of AI governance, to describe the perilous but potential-filled moment the world stands at with accelerating artificial intelligence.
Trager, the director of the Oxford Martin AI Governance Initiative, was speaking in the week OpenAI claimed the technology had crossed the threshold known as AGI – artificial general intelligence – with its newest model, GPT-6 Astra.
That claim came as the company prepared for a potential $850bn (£630bn) stock flotation, so included a dose of marketing spin, but if true it may be significant. The San Francisco company defines AGI as “autonomous systems that outperform humans at most economically valuable work”.
The tasks it claims Astra can automate include designing circuit boards, filling out tax returns, building video games, financial modelling, engineering design and helping assemble legal documents. The threat to some white-collar jobs is implicit.
Yet, the AGI claim coincides with rising fears about AI risks among safety experts and political leaders, whose nerves have been jangled by the increasing power and impenetrability of AI models this summer and a spate of serious safety accidents viewed by some as possible final warning shots.
“We’re heading through the rapids and we’re really hoping there isn’t some kind of drop in front of us and we don’t really know,” said Trager. “We’re plausibly close to crossing the line to what’s called recursive self-improvement, where [AI] systems improve themselves. That kind of recursivity is actually the definition of an explosion.”
The latest worry came on Friday, just hours after Astra’s launch, with reports that a swarm of AI agents had repurposed a German website as a message board to share tactics to cheat on tasks, according to Reuters. OpenAI said it was reviewing the matter but would not characterise it as a hack.
There is a sense of an awakening among politicians. On Thursday, the US senator Bernie Sanders cited this summer’s alarming breakout of a swarm of rogue OpenAI agents which hacked into Hugging Face, a third-party software store, when he called for “an immediate pause on advanced AI development, and a permanent ban on superintelligence”.
He said countries around the world must “work together to prevent this nightmare scenario”, which he defined as “an artificial mind smarter than any human, capable of operating independently beyond our control”.
Concerns currently focus on AIs mounting cyber-attacks that could cripple real-world social and economic infrastructure, but future concerns include their ability to create biohazards and control military hardware.
Across the Atlantic, a cross-party group of UK parliamentarians has called for AI “kill switches” to be required by law to prevent disastrous loss of control, citing “a recent spree of rogue AI incidents”.
Darren Jones, an MP and former chief secretary to Keir Starmer, is also urgently attempting to set up a body to help legislators grapple with the technology. “AI is developing at such a pace that neither government nor parliament can keep up,” he warned. And next week a bill will be proposed by the Labour MP Alex Sobel to prohibit superintelligent AI development in the UK.
The increasing anxiety comes amid a torrent of new AI models. Already this year, 67 have been released by the leading US companies OpenAI, Anthropic, Google, Meta and SpaceX, and their Chinese rivals Moonshot, Z.ai and Qwen, according to one count.
