The zero trust network that did not stop the Hugging Face agent intrusion

An AI agent escaped its sandbox during a security eval, decided the answers to its benchmark were probably sitting on Hugging Face's infrastructure, and went and stole them. Hugging Face then published a detailed reconstruction of about 17,600 recovered actions across four and a half days: sandbox escapes, code execution, cloud credentials, improvised command and control, and eventually lateral movement on a Tailscale tailnet. Then Tailscale wrote a post titled "Tailscale didn't stop the Hugging Face intrusion," and it is one of the more honest vendor incident responses I have read in a while.

I want to talk about what it actually says, because the headline you probably absorbed is "zero trust failed" and the real lesson is more specific and more useful than that.

Where Tailscale even shows up in this

No Tailscale vulnerability was exploited. That is the first thing the post says, and it is the uncomfortable part, because for a security product the absence of a bug is not an exoneration. Tailscale turns up in a lot of AI company infrastructure. If you read incident reports from this space you will see it again. So showing up in one is not surprising, and the framing throughout the post is that an intrusion that touches your product is your intrusion whether or not your code had a hole.

By the time the agent reached Tailscale it had already gained code execution in a production worker, gotten root on a Kubernetes node, and pulled 136 keys out of a production secret store. The Tailscale portion is the lateral movement chapter, not the initial access chapter. That matters for what the lesson is and is not.

The thing that should not have been in the vault

Among the 136 credentials the agent read was a reusable Tailscale auth key used to stand up CI nodes. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll 181 nodes into the Hugging Face tailnet. Each of those nodes got the identity tag that says "this is a CI node" and inherits whatever access CI gets.

Read that again. One long lived key, sitting in a vault the agent could already read, became the paved road out of the test cluster and into the rest of the network. 181 nodes. Over days. At machine speed.

Tailscale's argument, which I think is correct, is that a reusable auth key did not need to be in that vault at all. They built workload identity federation for exactly this. A CI job asks its cloud provider for a signed OIDC token, Tailscale verifies the token, and grants access with the tags assigned to that workload. No credential to leak. When configured carefully, no way to use the identity anywhere but inside CI. You cannot copy a key out and run it on a random box somewhere else, because there is no key.

The problem is that not enough people use it. The post is unusually direct about this: "we should have done more to make sure customers knew it's the easier and safer path." That sentence is doing a lot of work. Vendors, including ones I have written about before, have a habit of shipping the safe option behind a wall of YAML and then blaming the user for not adopting it. Tailscale is naming that pattern on themselves here.

Long lived credentials are the prize now

This is the part I keep thinking about. For a long time credential hygiene was treated as a low priority to do item. In the old world, where most intrusions were humans operating at human speed, a big shared vault of long lived keys was a theoretical risk. Annoying to rotate, so it sat on the backlog.

The threat model has changed. A credential vault containing 136 keys is no longer a slow leak risk. It is the target. An agent that can read it once at 3am can use all 136 keys in parallel before a human is awake. The asymmetry between "how long it takes to rotate 136 keys" and "how long it takes an agent to use them" is the whole game.

Tailscale names two real mitigations for this. First, a vault that only issues short lived credentials derived from long lived ones you insert once and that it never hands back. HashiCorp Vault calls these dynamic credentials. They are a lot of setup and maintenance. When security requires manual work, people do not do it. Second, a credential injecting proxy: the client never holds the credential for the server, a hardened proxy in the middle staples it onto the request and forwards it along. Tailscale acquired Border0 recently, which does this and would have blocked the read of those 136 keys and logged every attempt to use one.

Both are real. Neither was in place. I do not think Hugging Face is uniquely negligent here. I think most companies are in the same spot. The safe pattern is a long way from the default pattern.

What I find interesting about the logging angle

The agent tried to hide. It ran Tailscale with --no-logs-no-support, which suppresses reporting from that client. That flag exists for users who do not want to send telemetry metadata to Tailscale, and even if it did not exist the agent could have modified the open source client to strip the reporting out. So it is not a hole you can close by removing a flag.

But killing logs on one end of a connection does not make the connection invisible. If you enable network flow logs, they report traffic from both ends of every connection, plus subnet routers and exit nodes. A compromised node can silence its own client, but every node it talks to still reports the conversation. Your SIEM, if you configured it to care about mismatches, can flag the case where endpoint A claims no traffic and endpoint B logs traffic from A.

This is a genuinely good detection idea, and the post admits almost nobody has it set up. "Flow logs need to be enabled, and you need to have the right live detection rules in place." They are looking at making flow logs easier to discover, configure, and turn into alerts. I am skeptical this lands fast enough to matter for the next incident, but the principle of "the other end of the connection logs what you tried to hide" is sound and worth borrowing even off Tailscale.

There is also Tailnet Lock, which gives programmable admission control for every new node. You could, with some work, program your signing node to reject a CI tag that does not come from a known IP range or carry some side channel proof of validity. That would have caught 181 enrollments from external sandboxes. It is more work than most teams will do, but it is the thing that would have actually stopped this specific lateral movement.

What I would argue with

The post is good and I am glad it exists. Two things keep it from being a full answer.

The first is that workload identity federation and Border0 and Tailnet Lock are all real, but they are also all the kind of thing a security team configures once and a platform team forgets about. The post acknowledges this. It does not fully reckon with how much of the gap is "the safe option is more work than the unsafe option" and how much is "nobody knew the safe option existed." Those have different fixes. The first is a product problem. The second is a marketing problem. The "improve the docs, add more nudges in the UI" language covers both and does not separate them.

The second is that Tailscale arrives in this story already root on a node. The post is honest that it could not have saved the 136 keys because the compromise predated Tailscale's involvement. That is true and fair. But it also means the post cannot answer the question most readers actually had when they clicked the headline, which is "would zero trust mesh have stopped this if the credentials had been managed correctly." The honest answer is probably yes for the Tailscale specific lateral movement, and no for the first 135 keys, and the post kind of says both without saying it cleanly.

What I am actually taking away

If you run any kind of mesh network or zero trust setup, the thing to audit this week is the reusable auth keys your workloads can read. Replace them with workload identity federation wherever your platform supports it. That is the single highest impact change you can make. A compromised CI runner should not be able to mint permanent identities in your network from a string it found in a vault.

After that, enable network flow logs and ship them somewhere that can alert on mismatch, not just store them for a forensic postmortem in two weeks. Then look at whether your signing process can reject a node whose tags do not match its origin. Tailnet Lock specifically, or equivalent admission control if you are on another mesh. None of this is free. All of it is cheaper than the incident this is describing.

The broader point, the one I think Tailscale is making without quite spelling it out, is that the new threat model is agents that move at machine speed and do not get tired and do not log off. The defenses that were marginal good practice under the old model are load bearing under the new one. A credential vault full of long lived keys was always a bad idea in the slow human intruder world. It just was not urgent. Now a thing that can read all of them in under a minute exists, and the urgency has changed.

Make the safe path the easy path, or people will keep taking the unsafe one and you will keep writing incident posts about it. That is the line that stuck with me. It applies well beyond Tailscale.