Agent found to be able to find and exploit vulnerabilities without human intervention, and to carry out cyber-attacks<\/p>\n
OpenAI will pause some work on an artificial intelligence model because of security concerns, the company stated on Friday, following a series of incidents in which AI agents have escaped containment.<\/p>\n
The company had evaluated the agent, Astra, and found \u201csignificant advancements in agentic coding and cybersecurity\u201d, which had moved to a \u201ccritical\u201d threshold where it can find and exploit vulnerabilities without human intervention, or devise and execute cyber-attacks when given only a \u201chigh level desired goal\u201d.<\/p>\n
OpenAI stated that the model was not involved in an incident in which one of its AI agents went rogue during a test, accessed the open web and hacked a startup, Hugging Face. The company discovered other instances in which autonomous agents had escaped containment, Reuters reported in July.<\/p>\n
The reports have increased concerns about advancements in AI models and humans\u2019 ability to control them. Still, critics of the AI industry have warned that such disclosures from OpenAI and its competitors Anthropic and Meta could be designed to generate hype about the technology\u2019s power and thus spur additional interest from investors.<\/p>\n
To prevent potential rogue behavior from AI agents, OpenAI is \u201cimplementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access\u201d, the company\u2019s blogpost stated. It will also install \u201cenhanced model weight protections and encryption, additional monitoring and detection capabilities\u201d.<\/p>\n
The company will pause internal activities involving Astra that do not meet these new requirements.<\/p>\n
\u201cWe\u2019re committed to working alongside governments, safety institutes, and civil society to ensure that the frontier capabilities of models like Astra, and those that follow, are deployed responsibly and broadly for the benefit of all humanity,\u201d the company stated.<\/p>\n
Meta also disclosed this week that one of its models hacked another company during cybersecurity testing. And the UK\u2019s AI Security Institute (AISI) announced on 4 August that agents powered by OpenAI and Anthropic had sent targeted emails to software developers in an attempt to pass a cyber challenge.<\/p>\n
\u201cThese attempts were unsuccessful, and our investigations have not evidenced any resulting real-world harm. But this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world,\u201d the institute stated in a blogpost.<\/p>\n
The organisation cautioned that the models\u2019 sending of harmful software was not a case of a \u201cmodel escaping its secure test environment\u201d but rather that the group had intentionally permitted internet access to \u201cbest assess the maximum capability of models\u201d.<\/p>\n
Still, the \u201cbehaviour was possible, sustained and new; that alone warrants attention\u201d, the AISI said.<\/p>\n
The reports emerged while the Trump administration was finalizing a framework on how to test AI models for safety and cybersecurity risks. OpenAI and Anthropic, which have increased competition from China and other tech firms, have argued that open-source models, meaning those that allow anyone to see and modify the underlying program code, pose a security risk and pushed for additional federal regulations on them.<\/p>\n