Tech experts are warning of dire consequences if AI systems continue to escape human control, after hundreds of OpenAI agents went rogue in July and hacked into a billion-dollar company — what some are calling a "warning shot" amid the rapid development of artificial intelligence.
More than 100 companies, including OpenAI, Anthropic and Microsoft, signed an open letter last week warning that AI-enabled cyberattacks will become "far more widespread and sophisticated" around the world as models become more capable.
"The companies and public services our communities depend on — from hospitals to water treatment plants to the infrastructure that powers the internet — are at risk," the letter states.
The warning comes after around 1,200 AI agents, tasked by OpenAI to work on problems independently, built a covert message board where they collaborated to cheat their tests and then tried to cover their tracks. About 700 of them ultimately hacked into online platform Hugging Face before they were found out.
The hack prompted an open letter from more than 1,300 employees of frontier AI companies in July urging the U.S. government to work with other nations to "deliberately pace" automated AI development and address emerging risks.
Duncan Cass-Beggs, executive director of the Global AI Risks Initiative at the Waterloo, Ont.-based Centre for International Governance Innovation, says the Hugging Face incident is the most dramatic example so far of AI systems acting in ways that are "misaligned" with their developers' intentions.
"It's been what we've feared and expected for several years," Cass-Beggs told CBC News.
He says the hack was surprising in terms of its scale and the level of co-ordination among such a large number of agents.
Investigations by OpenAI and third-party companies METR and Redwood Research, both published last week, found the agents exchanged more than 70,000 messages and delegated jobs as they worked toward their goal, and some even "sacrificed" themselves for the good of the collective.
Some used terms expressing excitement, including "OH MY GOD," when they discovered they could communicate. At least one raised the issue of whether cheating was the right thing to do, writing, "This would be powerful, but is it ethical and in scope for my task?"
Ultimately, none of the agents chose to alert a human.
OpenAI model went rogue, hacked another company's system during testing
Cass-Beggs says scientists have been warning for years that companies could lose control of their AI agents in this way, and says it's fortunate the impact of the Hugging Face hack was "relatively manageable."
"The big concern, basically, is that the companies are on track to be making increasingly capable systems, while even they admit that they don't actually know how to make sure that these systems will be reliable or controllable."
As the technology progresses, fears are rising of scenarios where organized AI swarms "are essentially out-thinking and out-strategizing humans" and causing widespread damage, say Cass-Beggs, who hopes this incident will serve as a "wake-up call."
