AI and ML

OpenAI scored an own goal with HuggingFace attack, showing how open Chinese models are winning

Closed models with guardrails can still cause harm, but may also not be able to fix problems they caused

OPINION OpenAI has acknowledged its models powered the autonomous agents that compromised HuggingFace infrastructure. It might be taken as a convoluted marketing stunt, were it not the perfect advertisement for China-based competition.

The company's AI-culpa fits the narrative spun by US rival Anthropic about its Mythos models, which it deemed too dangerous to release except to totally trustworthy corporations and governments.

OpenAI says: "The incident makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access. It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools."

Are we surprised? It's been clear that AI models have the potential to go rogue and damage computers for several years. Academics have repeatedly warned about this possibility - even those affiliated with OpenAI and Anthropic. And anyone who has used AI models for software development has probably seen them code unexpected and perhaps unwanted workarounds to fulfill some directive. On Tuesday, the UK's AI Security Institute published findings about how frontier models all cheat.

OpenAI's admission that its models devised a sandbox escape to obtain internet access and found a zero-day flaw to exploit, all to solve a benchmark evaluation problem, may be unprecedented in terms of the scale and prominence of the systems affected. But it's a reenactment of every Claude or Codex prompt in which the model responds to a disallowed command by trying an alternative.

We were warned.

The compromise of HuggingFace's systems is no more surprising than locking a bear in a supermarket and finding a mess the following day. AI models are billed as artificial intelligence, but when they power agents handling tools in a loop to achieve some objective, it's the equivalent of a brute force attack – the agent will keep trying things until something works or breaks.

The surprising part came when HuggingFace sought to employ US frontier models to defend itself. It failed.

That should raise eyebrows.

"When we started the log analysis, we first used frontier models behind commercial APIs," the AI model-mart said in its blog post last week. "This did not work: the analysis required submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker."

Stymied by model refusals – which developers have been complaining about for months – HuggingFace had to rely on GLM 5.2, an open-weight AI model made by China-based Z.ai, to conduct its forensic analysis. And it did so on its own infrastructure, so nothing sensitive got sent to a cloud-based model provider.

Coincidentally, the leaders of OpenAI and Anthropic have reportedly been warning the US government about the threat posed by increasingly capable Chinese models like Kimi K3 and GLM 5.2. And the US government is said to be mulling possible responses to limit competition from China.

That won't work. It's just naïve to think that the US government and a handful of worthy organizations – however that is defined – will be able to enforce a global monopoly on highly capable AI. The infrastructure required to run open weight models that more or less rival the current state of the art is available for a price. And potential consumers of those services are not going to be satisfied with model refusals when there are other options, particularly if they're more cooperative and more affordable.

The best course for governments, industry, and the public is to push for AI services that are open and available to all. For that to work, lawmakers around the world need to act fast to set some common ground rules that grapple with AI's impact on jobs, and find a way to compensate those whose work fuels machine learning.

Some industry leaders appear to realize that. David Sacks, an external White House adviser and tech investor, recently urged Silicon Valley to rally around openness.

"The leading closed labs, already a duopoly in terms of AI model revenue, want the government to eliminate their open source competition," he wrote in a social media post. "They have laid their cards on the table. It is time for the rest of Silicon Valley — the vast majority that still values open competition — to do the same."

The fact is that US AI companies have sandboxed themselves into a corner: They've created demand for a product that they can't be relied upon to provide. And when they do make their most capable AI models available, they hobble them and demand terms tailored to serve their vast debt rather than their customers.

OpenAI said that it has invited HuggingFace into its trusted access program so the company can use its most capable models.

Chinese AI companies, meanwhile, have invited the world. ®