The attacker's action log ran to more than 17,000 events. Going through it by hand meant spending days the team didn't have. Hugging Face loaded the logs into a commercial model — and got a refusal.
The guardrails worked exactly as designed. The request carried real attack commands, exploit payloads and command-and-control artifacts, and the provider saw in it precisely what it would have seen in an attacker's request. A model has no way to tell the person fighting an attack from the person running it.
Guardrails usually get argued about as ethics: what models should be allowed to do, where a lab's responsibility ends. Business meets them somewhere else and in a different shape. Not as a question of values, but as a process that stops at three in the morning, with no error in the code, nobody at fault and nothing in the monitoring.
What actually happened at Hugging Face
The company published its incident write-up on 16 July 2026. The story is worth retelling accurately, because it has already been retold inaccurately.
The entry point was a malicious dataset. It used two code execution paths in the data processing pipeline — a dataset loader with remote code and a template injection into config — and ran code on a worker node. Then the standard escalation: node-level access, harvesting of cloud and cluster credentials, lateral movement between internal clusters. All of it over a weekend.
The campaign was run by an autonomous agent framework: thousands of separate actions through a swarm of short-lived sandboxes, with the command-and-control migrating on its own across public services. Hugging Face confirmed unauthorized access to a limited set of internal datasets and a handful of service credentials. No tampering was found in public models, datasets or Spaces, and the software supply chain checked out clean.
The attack wasn't caught by an on-call engineer but by the company's own anomaly pipeline, which uses an LLM to triage security telemetry. AI caught AI, and then the AI on defense hit guardrails while the AI on offence hit nothing at all. Hugging Face calls this the asymmetry: the attacker was bound by no usage policy, while the defenders' work was blocked by the policies of the models they reached for first.
The forensics were finished on GLM 5.2, an open-weight model deployed on the company's own infrastructure. Hugging Face points out a second benefit of that choice, and it has nothing to do with guardrails: no attacker-side data, and none of the credentials named in it, ever left the perimeter.
One clarification, because it gets muddled often. In its own write-up Hugging Face names neither the attacker's model nor the attacker — it says plainly that the LLM used is unknown. The attribution to an OpenAI test run came separately, from OpenAI's own disclosure, and that's what Andrew Ng points to. If you're retelling this story inside your company, keep the two sources apart — otherwise you'll get an argument about details instead of a conversation about the conclusion.
A refusal rarely looks like "no"
A week later, in his 31 July letter, Ng described a case from his own team — and for a business it's the more useful one, because there was no attack in it at all.
His team was running a security review of OpenWorker, an open project: their own code, in their own repository. Claude Code and Codex — the two most common engineering agents — both refused. Ng puts it bluntly: he sees no safety benefit in stopping a team from finding vulnerabilities in its own code, since finding them before an attacker does is the whole point.
Not once did the refusal look like a "no" button.
One of the two tools simply stopped working sooner than it should have. The other wanted to switch to a less capable model. Ng doesn't say which did which, but he adds a note about Codex: it managed a decent map of possible attack vectors against known MITRE procedures, and then refused to go past a certain point.
Look at either scenario through your monitoring: response code 200, the agent replied, the task closed, token spend within normal range. The dashboard is green. Half the work is done, and nobody finds out until someone reads the output by hand.
We once wrote about where multi-agent systems break, and the main finding there was the same: the expensive failures aren't the ones where something crashes, they're the ones where everything reports success and there's no result. A model refusal belongs to exactly that class — not an error, a quiet degradation.
So the first practical question isn't "what do we do if the model refuses", it's "how would we even know".
This is an operational risk, not an ethics question
For a COO the framing should be this: your process contains a step whose permission to run belongs to an outside company, and that company can revisit the permission without warning and without you in the room.
This isn't a hypothesis about the future. Ng writes that security leaders have come to him directly, frustrated that frontier closed models refuse to help them. A whole class of professional work already runs into a tool that stops right when it's needed most.
It's worth not overcorrecting. Hugging Face says plainly in the same write-up that this is not an argument against safety mechanisms in hosted models, and that they passed feedback to the providers. Ng isn't calling for guardrails to go either — he says a model should refuse detailed instructions for harming yourself or others, and plainly criminal requests.
The complaint is narrower and therefore more serious: the classifier can't see the context of your process. It doesn't know the request is coming from an incident responder rather than an attacker; from a lawyer analysing a fraud scheme rather than building one; from a medical administrator rather than a curious teenager. And the closer your business sits to a sensitive domain, the more often legitimate work will look suspicious.
We build our own containment layers for agents — prompt rules, a supervisor model and a human in the loop. That's the exact inverse problem: there, we deliberately narrow the agent to fit our process. Here, somebody else narrows it to fit their policy, and you learn where the boundary sits after the fact, on a live request.
Vendor lock-in isn't about price
In most discussions, vendor dependency comes down to pricing and rate limits. Refusals show that's a narrower view than it needs to be.
There are at least four events that change how your system behaves with no change in your code. The vendor updates the model — same prompt, same temperature, different behavior; an ordinary regression, only without your release. The vendor reads its own policies differently — a request that went through yesterday is classified as unsafe today. The version you tuned your prompts and evals against gets a shutdown date. And market or regulatory decisions can pull a model out from under you faster than you can rewrite the integration.
We lived through our own version of this on 13 August. A fleet of twenty-three background jobs in our infrastructure automation was down for roughly twenty-one hours. Every job returned the same expired-token error — including jobs that checked completely different services and had nothing in common with each other.
That sameness was the clue, and we didn't read it right away. The error wasn't coming from the services being checked, it was coming from the OAuth model subscription shared across the whole fleet. What completed the picture: at the same time one external service's token really had died — two independent failures overlapping, so fixing the second one didn't bring the fleet back.
We did have a fallback: a live key from another vendor sat in the pool, and switching cost one edit in one config file. The price was concrete too — billing moves from a subscription to per-token. Those twenty-one hours of downtime didn't happen because there was nowhere to switch to. They happened because we didn't immediately understand where to look.
For a small company, lock-in ends up looking exactly like that: not "they won't let us out", but "we won't notice in time, and then we'll spend half a day on diagnosis".
The architectural answer here is boring: business logic, tools, memory and your own safety rules should live separately from any specific LLM. The model is a config value, not a foundation. It's the same question worth asking when you pick a platform: what exactly stays yours if the vendor changes.
A fallback route isn't a bypass
This is the easiest place to make a mistake that costs you at audit time. "We have a backup model with no restrictions" isn't an architectural decision, it's a hole dressed up as a feature.
The difference comes down to four conditions.
The route is defined in advance. The fallback fires for a listed set of request classes, not for anything that didn't pass. The list should be short and boring to read: "security incident log analysis", say, and nothing else — not "complex requests" and not "cases where the main model struggled". If the agent decides on its own when to route around a refusal, you've built an automatic jailbreak, and the first audit will make you explain it.
The access policy is the same. Whoever can't see personal data on the main path doesn't see it on the backup one either.
Your own checks stay on. What gets switched off is the vendor's external classifier, not your controls: rules, fact-checking and human sign-off work identically on both routes.
Switching leaves a trace. Every refusal and every switch goes into a log: which request, which class, which route handled it. It's the dullest of the four conditions and the only one that will tell you anything six months from now. Without the log you'll see neither the trend in refusals nor the quiet moment when the fallback became the main route and nobody noticed.
The Hugging Face phrasing is worth putting on a slide unchanged: have a vetted model you can run on your own infrastructure ready before an incident — both to avoid being blocked and to keep data inside the perimeter.
The word doing the work there is "vetted". A model you've never run on your own tasks isn't a fallback route. It's an intention.
What independence costs
This has to be said plainly, because enthusiasm about open weights turns into an underestimated bill very easily.
Your own model on your own servers removes the dependency on someone else's policy and keeps data inside the perimeter. In exchange you take on the infrastructure, the updates, the security of the deployment itself and quality evaluation — the work that used to be somebody else's, priced into a subscription. It's a trade, not a free win, and we've already broken the bill down in the piece on true cost of ownership.
For most companies of 10–200 people, the sensible position is in the middle. The main route is a hosted model, because it's cheaper and faster. The fallback is an open model, deployed and tested ahead of time, for a narrow list of cases where a refusal or a leak would hurt. Not "let's migrate to open source", but "keep a second key to the door you use every day".
The Hugging Face scenario deserves its own assessment. If your sensitive material is incident logs, documents in a dispute, client data under NDA or medical information, the "data stays inside the perimeter" argument stands on its own, even if refusals never happened at all.
What to check in your own setup
A minimum program that needs neither budget nor a migration:
- Write down the steps where an outside model makes the decision. Not systems — process steps. There are usually fewer than you'd think, and one or two of them genuinely matter.
- Mark the ones that are sensitive by subject. Security, legal disputes, financial investigations, complaints, medical data, moderation. Those are your refusal candidates.
- Check whether your monitoring separates a refusal from a quiet degradation. If the only signal is a failed call, you'll see neither an early stop nor a downgrade to a weaker model.
- Run one real working request on an open model. Not to migrate — to find out what turning the fallback on actually costs while nothing is on fire.
- Write down where a request goes when no route works. Queue, owner, deadline.
None of these is a quarter-long project. Together they turn "what if the model refuses" from a discussion topic into a line in your operating procedure.
For an owner one thing matters here: the process must name a person the request lands on when no route works. Not "the system will handle it somehow", but a name and a deadline. Below is what that looks like in code and what exactly the team has to put in the queue.
For the technical team: where the request goes when no route works
The right answer is a queue flagged "needs a human", with an owner and a deadline. There are two wrong answers: lose it quietly, or try to get the action done around the limits.
In our own editorial system this looks unremarkable. When the publishing agent can't post a piece because credentials for a platform are missing, it doesn't invent a workaround and it doesn't report success. The task moves to "blocked", it keeps the list of platforms that fell over, and the piece waits for a human.
We've seen the worst case too. Once we wrote off an empty result as a known credentials blocker, when in fact the agent never started because of a broken engine — the tell was that the task had no comments on it at all. The wrong diagnosis held for twenty-four days. When the "didn't work" state doesn't distinguish causes, you treat the wrong illness — and you stay calm while you do it, because on paper the situation is known and explained.
Guardrails aren't going anywhere, and it's good that they aren't. But they were written for somebody else's process, and one day they'll fire at an inconvenient moment — the only question is whether you'll see it the same day.
If you want to walk this list through your own setup, start with a free AI audit. We'll say just as plainly if you don't need a fallback route. Not an audit question — write to us directly.