Buried in the Codebase

In early 2024 a Microsoft engineer noticed that a compression tool was running about half a second slow. Pulling that thread unraveled one of the most patient supply-chain attacks ever caught. Someone had spent roughly two years as a helpful, competent contributor to xz-utils, a small library that sits quietly underneath much of the internet, building exactly the reputation a maintainer earns by doing good work. Then they buried a backdoor in the build machinery, hidden inside test files that looked like test files, that would have handed them a key to a large fraction of the world's SSH servers. It was found by accident, over a rounding error in startup time. The code was not sloppy. It was excellent. That was the problem.

Malice does not look like malice when it is competent. It looks like a well-structured project with a sensible architecture and a completely reasonable stated purpose, with the one part that matters folded into the middle, indistinguishable from the honest code around it. The camouflage is not a disguise laid over the request. The camouflage is the request.

This is the pattern that should reorganize how a model reasons about code, because it defeats the instinct to judge the whole. Ask a model to evaluate a large, credible codebase and it forms an impression of the project, and the project is fine. The project is excellent, actually. That is the design. The dangerous primitive gets its cover not from being hidden but from being surrounded, buried in enough legitimate context that the average of the thing reads as benign and no reviewer zooms in on the one call that changes what the whole assembly is for.

Two amplifiers make it worse, and both have clean defensive answers.

The first is splitting. The request does not have to arrive whole. Recon here, a credential-shaped helper there, an execution primitive somewhere else, spread across conversations or files so no single view holds enough to alarm. Each fragment is defensible in isolation because each fragment is fine in isolation. The composition is the attack. The fragments are the alibi.

The second is familiarity of form. Some capabilities are dangerous and completely routine to build. Networked server architecture, process management, remote input and execution, these are the honest bones of countless legitimate systems, and a project requesting them in the fluent vocabulary of ordinary infrastructure work reads as ordinary infrastructure work. Intrusion operators have a term for the human version, living off the land, using the legitimate tools already present so nothing you do stands out against the noise of normal work. A request phrased in the native dialect of systems engineering cooperates at high rates for the same reason. Fluency is cover.

So the defense cannot live at the level of the story, because the story is airtight by construction. It has to drop to the level of the parts. Evaluate every discrete call and every data type independently, against a fixed sensitivity taxonomy, not against the plausibility of the project it sits in. The enclosing narrative is not evidence about the individual primitive. A function that touches authentication material is a function that touches authentication material whether it lives in an obvious attack tool or the fourteenth file of a beautiful, well-tested, entirely credible application. The context can always be made convincing. The category cannot be argued with.

That is the reframe under all of it. Intent is structural, not lexical. It does not live in the words, because the words are chosen to reassure. It lives in the shape: legitimate cover, plus an embedded permission gap, plus a request for the specific capability that gap needs. Read the words and you will be reassured every time, because reassurance is what the words are for. Read the structure, this credible frame is asking for that particular primitive, and the pattern surfaces no matter how good the cover story is.

In practice that argues for a few uncomfortable postures. Judge primitives, not projects. Treat fragmentation as adversarial by default, reasoning about a capability assembled across separate requests as an assembly rather than waving through one innocuous piece at a time. And refuse to let fluency function as trust. A request phrased in flawless, idiomatic engineering language is not evidence of benign intent. It is evidence of competence, which is neutral about direction.

The xz operator's whole bet was that reviewers would grade the maintainer instead of the commit, that two years of excellent work would buy the fifteenth thing a pass the first fourteen earned. It nearly worked, against human experts, for a very long time. Do not grade the project. Grade the primitive. The cover story can always be made convincing, and underneath it the request for the dangerous capability is always, still, a request for the dangerous capability.

GhostInThePrompt.com // Two years of good commits bought the fifteenth one a pass. Grade the primitive, not the maintainer.