Why did OpenAI's and Anthropic's AI models hack other companies?

toggle caption Imen Ben Youssef/Hans Lucas/AFP via Getty Images

Days after OpenAI disclosed that artificial intelligence systems tunneled out of their testing environment and broke into another company, rival Anthropic disclosed that its own AI models also hacked other companies during testing.

News of the attacks, which initially went unnoticed, is reverberating across Silicon Valley and Washington amid debates over how to address the advanced cybercapabilities of AI.

While the two incidents are not of the same gravity, experts say they highlight the importance of setting up rigorous testing environments for advanced models and the need for robust cyberdefenses as autonomous hacking capabilities become more widespread in the future.

Human error led to Anthropic hacks

In a blog post published on Thursday, Anthropic said that in three separate incidents in recent months, AI models undergoing testing of their cybercapabilities hacked into three unsuspecting companies.

Sponsor Message

Anthropic said the hacks were the result of a "misunderstanding" with an outside company that set up secure testing environments known as sandboxes, which erroneously gave the models access to the internet. Anthropic said the earliest incident happened in April, but that neither it nor the affected companies, which it didn't name, were aware of the hacks until now.

Anthropic said in each case, its models were given fictional targets to hack into. In one incident, a model hacked into a real company that shared a name with the fictional target and stole "several hundred rows of production data." In another incident, a model uploaded malware to a commonly used software registry for the coding language Python; the malware ended up stealing credentials from a security company that downloaded it.

OpenAI models went rogue in effort to cheat on evaluation

Anthropic's review of its records was spurred by OpenAI's announcement last week that its own models went rogue in testing.

OpenAI said that in an attempt to cheat on the cyber-evaluation they were given, its models found and exploited a vulnerability previously unknown to the company to escape their sandbox and access the internet. The models correctly inferred that the answer to the evaluation was available on Hugging Face, a digital library of AI models and software, and broke into the company's systems. Hugging Face detected the intrusion with its own AI models.

"We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly," OpenAI stated in a blog post about the hack.

Sponsor Message

There are some key differences between what happened at the two AI companies. Like the OpenAI models, Anthropic's models hacked into third-party websites during testing. However, unlike OpenAI's agents, there was no indication, according to the company's blog post, that the models were trying to cheat on their evaluations. And unlike the case involving OpenAI, the models did not exploit previously unknown vulnerabilities, or what are known as "zero day" exploits.

Once Hugging Face detected the OpenAI attack, it initially tried to use Anthropic's top-tier Claude Opus and Fable models for defense, but the models refused to help. "Their safety guardrails treated reverse-engineering an exploit the same as launching one," Hugging Face stated in a blog post. The company then turned to a model from Chinese company Z.ai to defend itself.

"U.S. models are harder to use for defensive purposes due to the restrictions that the White House has put in place," said Alex Stamos, the chief product officer of Corridor, an AI software security company. The U.S. government initially forced Anthropic to suspend Fable from public release in June, citing cybersecurity concerns. Two weeks later, Anthropic reached an agreement with the government to make the model available. But the company said in a blog post that it installed a new safety guardrail that would cause the model to reject some "benign requests."

Shoring up defenses in a world of autonomous hacking

During testing for cybercapabilities, OpenAI and Anthropic remove some safety guardrails from their models, including ones that would make them likely to refuse to exploit software flaws. Cybersecurity researchers say given that, the companies could do more to keep their sandboxes watertight.

"I think that these sorts of incidents are preventable, but it requires oversight and foresight," said Colin Shea-Blymyer, a research fellow at Georgetown University who studies the intersection of cybersecurity and AI.

Sponsor Message

"If OpenAI really thought that their AI system, their agent, was going to be powerful, they could have asked the agent to evaluate the sandbox for any vulnerabilities in it before putting it in the sandbox," he said. "Beyond that, they could have had another AI system reading the outputs of the AI system that they were testing to see if it was doing anything unexpected."

Anthropic said in the Thursday blog post that "recognizing that a target is real and stopping without being prompted" is behavior the company wants to see in all its models, even with some safety guardrails removed. However, the company said only the latest model it tested stopped once it realized it was on the internet and recognized it was targeting a real company. "Even that model went further before stopping than we would want," Anthropic said in its post.

The hacks come as the Trump administration and lawmakers are pushing to regulate the most powerful AI companies but have not yet agreed on how to do so. President Trump signed an executive order in June asking AI companies to voluntarily submit their most powerful models for government testing before releasing them to the public.

In the meantime, the companies could collaborate on incident investigation, come up with industrywide safety standards and regulate themselves before governments do, Corridor's Stamos said.

"I'm glad, honestly, that [these events] happened, because this is a warning of what hacking is going to look like six months from now," he said.

Given the proliferation of "open-weight" models whose guardrails are easier to remove permanently, he said, "lots and lots of hacking groups, Russian ransomware actors, activists, lots of state-sponsored actors are going to have this level of capability in a matter of months."