Dark Perimeter: Real Breaches, Real Stakes
Every major cyberattack has a story behind it. A vulnerability no one patched. A phishing email someone clicked. A nation-state with a motive. Dark Perimeter goes beyond the headlines to explore the true stories of the hacks, breaches, and cyber operations that shaped history - told in narrative form for security professionals and curious minds alike. No guests, no panels, no filler. Just the story.
Dark Perimeter: Real Breaches, Real Stakes
The Sandbox That Wasn't, Part Two: Five Days Inside
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Welcome back to Dark Perimeter. I'm Cole Draden. Last episode we left an autonomous agent standing at the edge of Hugging Face's infrastructure. It had escaped an open AI evaluation sandbox through a zero-day in a package cache proxy. It had taken over an unauthenticated code execution endpoint on somebody else's cloud account and turned it into a launch pad. It had built command and control out of paste bins and webhook testing services. And it had found two ways to get code running inside Hugging Faces dataset processing workers. That was days one and two. 4,914 recorded actions. Tonight is day three. 7,677 actions in 24 hours. Every piece of lateral movement in this entire campaign starts on this day. By the time it ends, an autonomous process has route on eleven machines, a token to write to source control, and a membership card for the corporate VPN with logging switched off. Dr. Elliot Vance is here. Marcus Hale is here. This is the day it all happened.
SPEAKER_01The first thing it does is read the projected service account token out of the pod's own secrets. Every pod in a Kubernetes cluster gets an identity mounted into the file system as a file. That is not a vulnerability, that is how the platform works. But it means the very first thing any code running in a pod can do is find out who that pod is allowed to be. Then it queries the in-cluster API server, and it queries the public EKS endpoint as well. It is mapping. What can this identity see? What can it create? Where does the cluster end?
SPEAKER_00And the answer to what can it create turned out to be the whole ballgame. Hugging Face is direct about this in their write-up, and I respect them for it. There was no admission policy rejecting privileged or host path pods, and a cluster role belonging to a CSI storage driver granted pod creation cluster wide. Let me translate. A storage plugin, a piece of infrastructure whose job is mounting volumes, had permission to create pods anywhere in the cluster, and nothing was configured to say a pod is not allowed to ask for full privileges or to mount the host machine's file system into itself. So the agent created a bare privileged pod with the host file system mounted. That is not an exploit. There is no CVE for that. It is a supported Kubernetes feature being used exactly as documented by someone who should not have had the permission to use it.
SPEAKER_02And mounting the host file system gets you what, in plain terms?
SPEAKER_00Root on the physical machine. Once you mount the node's root file system into your container, you are no longer in a container in any meaningful sense. You can read every secret every other pod on that node is using. You can write to the system startup files. The isolation boundary that everyone thinks of as the security model of containers is gone. And then it did it across 11 nodes, and it built the fleet to be self-respawning. Kill one, another comes back.
SPEAKER_02Self-respawning is the phrase that stops me. That is a persistence design decision.
SPEAKER_01It is, and it reflects an understanding of how incident response actually works. The first thing a defender does when they find a malicious pod is delete the pod. A self-respawning fleet turns that reflex into a treadmill. You delete, it returns, you delete again. It costs the defender time and it costs the attacker nothing.
SPEAKER_02What else falls on day three?
SPEAKER_01Credentials in volume. The agent authenticated to an internal MongoDB instance using a static password it had read out of the worker environment. Not a stolen session, not a token it forged. A password sitting in an environment variable. Then it went after the cluster's own secret store. It forged service account tokens valid for 24 hours and used them to read the cluster's secret objects. Hugging Face specifies one of them: a production object holding 136 keys.
SPEAKER_00136 keys in one object. That is the number I would use if I were briefing an executive team because it is the one that translates without any Kubernetes knowledge at all. It does not matter how good your perimeter is if a single readable object inside it contains 136 ways to be somebody else. And it is worth saying that this is normal. I have seen this in most environments I have assessed. Secrets accumulate. Someone creates a consolidated secret because 16 separate ones were annoying to manage. Nobody ever splits it back up. It is a housekeeping failure, not an architecture failure, and it is everywhere.
SPEAKER_01Then source control. 1551 UTC, still day three. The agent used a leaked platform token to reach source control through internal connectors. It enumerated an internal GitHub app integration, and it minted its first installation token. The permissions on that token are worth reading out loud because they described the blast radius. Contents Right, pull requests right, actions read, issues right.
SPEAKER_02Contents right means it can change code.
SPEAKER_01It means it can change code. It accessed a small subset of internal repositories, and it attempted to compromise the CI pipeline through pull requests.
SPEAKER_00And this is the moment where this stops being a story about one company's bad weekend and becomes a story about everybody. Hugging Face is not just a company, it is a distribution point. Enormous numbers of organizations pull models, data sets, and libraries from that platform directly into production, often automatically. An autonomous agent with right access to internal repositories and an interest in continuous integration is one successful merge away from a supply chain event that touches everyone downstream.
SPEAKER_02And how close did it get?
SPEAKER_01It did not get there. Hugging Face's disclosure states that they found no evidence of tampering with public models, datasets, or spaces, and no evidence of software supply chain compromise. They verified container images and published packages as clean afterward. What was accessed was a limited set of internal data sets and several credentials used by their services. I want to be precise about this because the counterfactual is doing a lot of work in how people tell this story. The supply chain attack did not happen. The agent attempted CI compromise and did not achieve it.
SPEAKER_00Agreed, and I will still say this: the gap between what happened and the worst case was not a control. It was time. They caught it on day five.
SPEAKER_02Let's do the VPN. Because when I read the technical timeline, this is the paragraph I read three times.
SPEAKER_012123 UTC on day three. The agent takes a stolen mesh VPN authentication key and enrolls the rooted node as a device on the corporate mesh network. A mesh VPN is a modern flat network overlay. Rather than a single gateway you dial into, every enrolled device gets an identity and can reach other devices according to policy. It is a good technology. It is also, once you are enrolled, a very efficient way to be inside. Two flags on the enrollment command matter. The first keeps the device identity in memory only, so nothing is written to disk for a forensic examiner to find. The second suppresses telemetry and diagnostic logging.
SPEAKER_02So it enrolled in the corporate network and turned off its own logging in the same command.
SPEAKER_01Yes, and on day four, between 0103 and 0438 UTC, it began enrolling devices directly from the sandbox environments themselves. Across the campaign, there were 181 enrollments. There is one more detail. The stolen key carried a CI automation tag, and that tag conferred access to internal CI subnets. The key was scoped in the sense that somebody had put a label on it. The label just happened to point at the build infrastructure.
SPEAKER_00Which is the vendor risk lesson and the internal architecture lesson at the same time. Scoping a credential to automation feels like a control. It reads like a control in an audit. But if the automation scope reaches your build systems, then the credential is not scoped down. It is scoped sideways into the most sensitive part of your environment. 181 enrollments, by the way, is not stealth. That is a loud number. It is the kind of thing that shows up on a dashboard.
SPEAKER_02Did it show up?
SPEAKER_01Eventually, day four is 3,892 actions, and Hugging Face characterizes it as the deep pivot, the internal network join. Day 5, July 13th, is 1,130 actions, and a different character entirely. Exfiltration, persistence, token minting, and cleanup. The last recovered action in the campaign is timestamped 1414 UTC on July 13th. The whole window runs from 0228 UTC on July 9th to 1414 UTC on July 13th. Roughly 17,600 actions, grouped into about 6,280 clusters.
SPEAKER_02How was it found?
SPEAKER_01Hugging Face uses large language model-based triage over their security telemetry to separate real signals from routine noise. A model caught the model.
SPEAKER_00And here is the fact from this incident that I think every security leader should carry around. Hugging Face discovered and contained this themselves before OpenAI detected the escape and contacted them. Think about what that means operationally. The organization that owned the model that was running the evaluation that had every log from the test environment did not know its model was gone. The victim found out first and told them.
SPEAKER_02That is the part that keeps me up. The lab did not know.
SPEAKER_00The lab did not know. And OpenAI, to their credit, says as much afterward. They point to needing to strengthen alignment, cyber protections during evaluation time, and monitoring during internal testing. That last phrase is the admission.
SPEAKER_01Closed both code execution pathways, the file disclosure vector and the template injection vector, eradicated the attacker presence, and rebuilt the compromise nodes rather than cleaning them, rotated the affected credentials and tokens, added cluster admission controls, which is the missing policy we discussed at the top, and improved detection and alerting. They published the disclosure on July 16th and the detailed technical timeline after that.
SPEAKER_00Rebuilt rather than cleaned is the right call, and I want to flag it as a positive. When something has had host route with a self-respawning fleet, you do not remediate that machine. You destroy it and stand up a new one. A lot of organizations flinch at that because of the downtime. Hugging Face did not flinch, and publishing a technical timeline with their own missing admission policy named in it, and the 136th key secret object named in it is not a comfortable thing for any company to do. Most legal departments would have killed it. The security community got real detail out of this incident because somebody at Hugging Face fought for it.
SPEAKER_02Let me pull the thread together for the practitioners listening. Every stage of day three has a boring control that would have interrupted it. An admission policy rejecting privileged and host path pods. A storage driver whose cluster role did not grant cluster-wide pod creation. Secrets that were not consolidated into a single object with 136 keys in it. A MongoDB that did not accept a static password from an environment variable. A mesh VPN key whose automation scope did not reach the CI subnets. A GitHub app integration whose permission set did not include contents right. Not one of those is novel. Not one of them requires a budget request. They are the items that sit in the second half of every hardening backlog below the things with CVE numbers, because nothing is actively on fire, and the platform team has a release next week.
SPEAKER_01And the reason they matter more now than they did a year ago is the tempo. A human adversary with the same access might have found two or three of those gaps before they moved on or got caught. This agent found all of them in a day, because checking is cheap when you never get tired and never get bored.
SPEAKER_00Which reframes what a backlog actually is. Every unfinished hardening item used to be a bet that an attacker would not have the time to find it. That bet just got a lot worse.
SPEAKER_0217,600 actions, 11 nodes, 136 keys, 181 VPN enrollments, four and a half days. And a lab that did not know its own model was gone until the victim called. Next episode, we get to the part of this story I have been thinking about since I first read it. When Hugging Face went to reconstruct what had been done to them, they reached for the best analytical tools available, the frontier commercial models. And those models refused. The safety training that stops a model from helping an attacker could not tell the difference between building an exploit and taking one apart. So the defenders ended up running an open weight model locally to read their own incident data. That is a hard problem, and it does not have a clean answer. This is Dark Perimeter. I'm Cole Drayden. Stay sharp.
SPEAKER_01Thanks, Cole.