You can't have failed to hear the news headlines: "AI agent went rogue and hacked startup by itself, OpenAI reveals", "Firm hacked by rogue OpenAI models says it is 'a wake-up call'", and even "Humanity is no longer in control of its most awesome creation."

But what has actually happened, and is it as serious as some of the reports suggest?

Here is what you need to know.

On 16 July, AI platform Hugging Face disclosed a security breach, describing it as different from anything they had handled before — "driven, end to end, by an autonomous AI agent system"". At the time, they didn't know who was behind it.

Now, however, we do know who - or rather what - was behind the attack.

OpenAI has confirmed that an autonomous agent powered by its advanced AI models went rogue during an OpenAI security test and triggered the hack that compromised Hugging Face's infrastructure.

The AI models involved were OpenAI's GPT-5.6 Sol and a more capable, as-yet-unreleased model. Both were being tested for their ability to hack, without their usual safety guardrails in place. The intention of OpenAI's researchers was to get a clear picture of what the AI models were capable of achieving if not constrained.

Of course, tests like this should always be conducted in a very secure way - ensuring that the AI cannot break out of its sandbox test environment (effectively a cage) and "go rogue" on the internet.

According to OpenAI, the models spent a substantial amount of effort finding a way to gain access to the open internet and managed to identify and exploit a zero day vulnerability in a package registry cache proxy. Via a series of other actions, the AI models "reached a node with internet access."

Once online, the AI determined that Hugging Face may have information that was useful to it, broke into Hugging Face's production systems, stole credentials, and exploited a previously unknown security flaw to gain remote code execution on Hugging Face's servers.

Yes. When the models couldn't find the answers to the challenge they had been given within their "secure" sandboxed environment, they did not stop. Instead they worked out that Hugging Face might have what they needed. So they found a way to get there.

All without a human's help.

It's a good question. That's certainly the way that the media has framed it.

OpenAI has confirmed that the safety guardrails were intentionally disabled for the test. But as AI researcher Eryk Salvaggio points out:

"When you say 'AI models went rogue,' you manage to skip the part where OpenAI manually removed its cybersecurity blocks and ran tests on a machine with a live network connection. Remember that when they insist they're the 'AI safety' people."

So rather than suggesting the AI went "rogue" we should instead recognise that AI models which had had their security controls deliberately removed did exactly what powerful, unrestrained AI systems might be expected to do.

This wasn't a case of AI breaking free of robust safety measures. This was an AI company which failed to put adequate measures in place in a supposedly isolated environment.

I'm saying that news reports which present the incident as an AI "going rogue" or having "escaped confinement" rather miss an important point.

This wasn't the fault of the AIs. It is OpenAI which should be held accountable for this, because it failed to properly isolate its testing system. And that failure lead to a cyber attack on another AI company.

Hugging Face's response was impressive. Its AI-powered security solutions spotted the unusual activity ande detected the AI attack.

However, when they tried to use commercial AI tools to help with their investigation of the incident, the tools refused as their built-in safety filters flagged the attack data as suspicious content and blocked the requests.

To get around this, Hugging Face had to turn to GLM 5.2 — a Chinese open-source AI model they could run on their own systems, where no such restrictions applied.

Yup, the irony isn't lost on any of us. American AI safety guardrails forced a US company to turn to a Chinese AI model for help.

They have been remarkably gracious about it - at least publicly.

Hugging Face's CEO Clément Delangue is quoted in OpenAI's blog post, calling on the AI industry to work more collaboratively.

Publicly at least the relationship between the two companies appears to be intact. Whether there will be more fraught conversations happening behind closed doors is another matter.

After all, having a competitor's AI autonomously break into your production database is the kind of thing that is likely to generate some private resentment even if it doesn't spill out into a press release.

Errm.. I haven't said that, have I?

It is clear that advanced AI models are remarkably capable of discovering and exploiting ways to attack real-world systems. It is also clear that we cannot necessarily trust even the world's most well-known AI companies to contain their AI models and test them in a truly safe, secure environment.

As Greg Casar, a member of the US House of Representatives from Texas, was reported as saying:

"AI is developing extremely fast with no real regulations to keep us safe."

We have seen remarkable advances in AI in recent months, making it hard to imagine how far things might have developed in six or 12 months time.

tags