Docker is the most confident line on a lot of resumes and the least examined file in a lot of repositories. Containerizing a service became a way to signal that you know what you are doing, and for a certain kind of engineer the signal is the point. The docker-compose.yml works on the laptop, it works in the demo, it ships, and the confidence that carried it there quietly becomes the reason nobody asks the one question that decides whether it is safe: what does this do when it breaks.
That question is where I spend my time, because it is where the entrances are. A system is rarely equally secure in every state, and the state nobody tested is almost always the weaker one. The engineers most sure of their setup are often the ones who have never once watched it fail, and the distance between that confidence and that blind spot is a place an attacker can stand.
The blind spot survives because the failure that opens it does not look like a security bug. It looks like an operations problem, and calling it a security problem means admitting the confident version of the story was incomplete. So it stays filed under flaky, unexamined, shipped, running in production under a green checkmark. Using it against a system starts with seeing the two kinds of failure as genuinely different things, which most people do not.
Two different failures
A correctness bug is a wrong answer. Given input X, the system returns Y when it should have returned Z. It lives in the domain of what the system computes. We test for it with assertions, we catch it with fuzzing and property checks, and when a security person finds one they already know the vocabulary: injection, logic flaw, broken access control.
A reliability defect is the absence of a correct answer in time. The system hangs, crashes, restarts, runs out of memory, times out, or returns a degraded result because a dependency it needed was slow or gone. It lives in the domain of whether and when the system answers at all. It gets triaged to the operations column. It gets the word flaky attached to it, which is engineering slang for we have decided not to think about this.
The reason that triage is a mistake is simple and almost never stated out loud. A system is rarely equally secure in every state. It is one thing when it is healthy and a different thing when it is degraded, and the degraded version is usually the weaker one.
Degraded states are usually weaker
Walk through what systems actually do when they come under stress, and the pattern is hard to unsee.
When an authorization dependency becomes unreachable, a great many systems fail open. The policy engine times out, the call throws, the catch block logs a warning and returns allow, because the engineer who wrote it was optimizing for availability and did not want a slow sidecar to take down the whole product. That is a reliability decision with a security consequence, and it has a name: CWE-636, failure to fail securely. The steady state checks the policy. The degraded state does not.
When a process crashes and something restarts it, there is a window during boot where it is listening before it is ready. It binds the port, then loads the config, then fetches the TLS material, then wires up authentication. For a few hundred milliseconds it is a server that answers without the controls that make it safe to answer. Steady state is locked. The restart window is not.
When a log pipeline backs up or a disk fills, logging degrades or stops. The application keeps serving, but it stops writing the record of what it served. Detection goes quiet while the system stays up. The steady state is observed. The degraded state is blind.
When a healthcheck only proves that a port is open, a process can report healthy while it is anything but, and a load balancer will keep sending real traffic to an instance that has lost the half of itself that matters.
None of these are logic bugs. Every one of them is a reliability defect. And every one of them moves the system from a stronger security posture to a weaker one, on a trigger that is often nothing more exotic than load, a slow network, or a dependency going away.
The red-team move is to force it
Here is where it stops being an availability conversation and becomes an attack one.
A reliability defect on its own is latent. It fires when the stars align, when traffic spikes at 3 a.m. or a deploy races a cache warm. The defender experiences it as bad luck. The thing that turns it from bad luck into a vulnerability is a reliable trigger. If an attacker can force the degraded state on demand, they are no longer waiting for the system to be weak. They are choosing the moment.
Say it plainly. Forcing a reliability defect converts the weakest state of a system from an occasional accident into a condition the attacker selects. You do not need the fail-open path to be reachable by luck. You need to be able to knock out the dependency whose absence opens it, whenever you want, and then walk through while it is open.
The classic red-team instinct is to ask what the system computes incorrectly. The more productive instinct, and the rarer one, is to ask how the system fails, and whether its failure mode is more or less privileged than its normal one. The second question is where the entrances are.
Why Docker is the richest ground for this
I am picking on Docker on purpose, because it is everywhere and its defaults are generous in a way that manufactures reliability defects for free. It is also where the confidence runs highest. The same engineer who will defend their container setup in a design review has usually never run it out of memory on purpose, and the defaults reward that incuriosity by working anyway, right up until someone makes them fail. The accompanying Docker Crowbar lab runs against its own containers. The configuration discussion below explains the failure modes that lab investigates. The point is to understand the mechanism well enough to find it and close it, not to point it at anyone.
Resource limits are off until you turn them on
By default a Docker container has no memory limit and no process-count limit. It can allocate until the host has nothing left, and it can spawn processes until the host runs out of PIDs. When memory runs out, the kernel out-of-memory killer wakes up and picks a victim by its own heuristic, and that victim is not guaranteed to be the greedy container. It can be a different container on the same host, or a process the host needed.
A Compose service without mem_limit, pids_limit, or cpus leaves those constraints unset at the service level. If an application can consume the host's available memory, the host OOM killer may terminate a different process, including a policy service. That is a possible failure mode, not a way to select an arbitrary victim reliably. An outage of the policy service becomes an authorization bypass only if the application handles that outage by allowing the action.
Set limits on each relevant service, size them against the host budget, and test the application's response when its policy dependency disappears. Docker Crowbar deliberately gives the policy sidecar a memory limit and an allocation endpoint so its own container reaches that limit. That controlled failure is distinct from predicting which process a host-wide OOM event will kill. Docker's resource-constraint documentation makes the host-wide risk explicit.
Limits contain resource consumption. They do not decide whether the application fails open or closed, and a restart policy does not repair an authorization decision.
Fail-open is the one that matters most
The single highest-value version of this is the authorization sidecar that fails open. The application asks the policy service for a decision on each sensitive action. In the vulnerable path, a connection failure or timeout becomes permission to proceed. The policy logic never said yes. The application supplied the yes after losing contact with it.
That fallback can look harmless while the dependency is healthy. Once the dependency stops answering, the exception path becomes the authorization path. A reachable crash, exhausted connection pool, or restart window can expose the same mistake. The specific trigger needs a test, not a list of things that sound plausible.
Failing closed denies the protected action when the application cannot obtain an authorization decision. That trades availability for preserving the boundary. Check the normal authorized path too: a fix that denies every request has closed the service, not repaired it.
This exact chain runs in a lab I built to go with this piece, called Docker Crowbar. Two copies of one app sit behind one policy sidecar and differ by a single line, whether they fail open or closed. The sidecar carries one deliberate defect, a /balloon endpoint that allocates memory until the container's mem_limit lets Docker's OOM killer take it. One request fires the crowbar. In the measured run the vulnerable app flips an unauthorized transfer from denied to allowed the moment the sidecar goes silent, and the hardened app does not move. It is at github.com/ghostintheprompt/docker_crowbar, it runs against its own containers, and the decision it turns on is a pure function with a test that needs no Docker at all.
Forcing the restart window
restart: always is the most natural line in a compose file and it quietly creates a loop you can drive. If an attacker can crash the application on demand, with a request that trips an unhandled exception, an input that triggers a panic, or the memory pressure from the first section, then restart: always hands them a fresh boot sequence every single time. CWE-703, improper handling of an exceptional condition, meets an orchestrator that is trying to be helpful.
Every boot replays the startup window. If the service listens before it finishes wiring authentication, or regenerates a default credential when it finds no config, or opens a debug or metrics port early in boot, then each forced restart is a new attempt at that window. A race that is impossibly tight to hit once becomes trivial when you can re-roll it on demand. The defense is to make liveness and readiness distinct, to not accept traffic until the security-relevant setup is finished, and to use on-failure with a backoff instead of an unconditional loop that an attacker can pump.
Healthchecks that only prove a port is open
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/"]
interval: 10s
This proves the socket answers. It does not prove the database connection is alive, the policy cache is warm, or the auth middleware loaded. A container can degrade into a state where it serves but no longer enforces, and keep reporting healthy, and the orchestrator will keep routing production traffic to it. The reliability defect is a healthcheck that measures the wrong thing. The security consequence is traffic pinned to a degraded, under-enforcing instance. A readiness check that exercises the security-critical path, not just the port, closes it.
Logs that eat the disk and then go silent
Docker's default logging driver writes JSON files with no size cap unless you set one. Drive enough error output and the disk fills. When it does, logging stops, and on many systems so do the application's own writes, which pushes you back toward a fail-open or fail-crash path depending on how the code handles a failed write. The reliability defect is unbounded log growth. The security consequence is a window where the system operates without a record, which is the condition every intruder wants and few can arrange. Bounded, rotated logs shipped off the host to somewhere an attacker on the box cannot starve are the answer.
logging:
driver: json-file
options: { max-size: "10m", max-file: "3" }
The chain has stages
A single forced fail-open is a finding. The reason to care is that the primitive scales, and each stage hands more blast radius to whoever can pull the trigger on command.
Stage one: force the defect
Everything above is stage one. Starve a container that has no memory limit, let the OOM killer take the authorization sidecar, and the app it was protecting answers yes on its own. One request, one degraded host, one unauthorized action. It runs in a lab I built for this piece, Docker Crowbar: two copies of one app differ by a single line, whether they fail open or closed, and the forced OOM flips an unauthorized transfer from denied to allowed on the one that fails open while the other does not move.
Stage two: let CI amplify it
Point the same primitive at a build pipeline and the blast radius stops being one request. It becomes everyone who pulls the result. The vector is GitHub Actions cache poisoning. A pull_request_target workflow runs with the base repository's secrets and its default-branch cache, and when it builds untrusted fork code, usually inside Docker, that code can write a poisoned entry into the shared cache. A later trusted release job restores that cache by key and runs what it finds, holding the permissions to publish. TanStack lived through exactly this in May 2026, and Angular before it. The link back to stage one is not a metaphor: the poisoned write has to land while a build is running, and one way to force that window is to evict the existing entries by filling the cache, which is the same uncontrolled-resource-consumption primitive from the first section, CWE-400, aimed at a different target. Force the reliability condition, land the poison, let the trusted pipeline carry it to every consumer downstream. The mechanism runs, marker-only, in the Docker Crowbar stage-two lab. GitHub has spent the last year closing it, which is the surest sign it is real. The Actions runtime token can no longer write to the cache after a job ends, actions/checkout is being hardened against the fork-checkout pattern, and organizations can now enforce pinning every action to a commit SHA, the fix the industry reached for after the tj-actions/changed-files compromise leaked secrets from more than twenty thousand repositories.
Stage three: the agent that trusts the build
A poisoned artifact, cache entry, or build log can become input to an AI agent wired into CI. An integration may give that agent permission to comment, label, approve, merge, add a collaborator, or run a command. That is the next trust transition: pipeline output crossing into a system that can act on it.
Docker Crowbar's third stage models that seam. Its agent is a deterministic directive parser with a tool registry and a policy gate. Every tool call appends a record to an in-memory effects list. No PR is merged, no collaborator is added, no command is executed, and no cache is reseeded. The naive configuration records all four privileged calls from the poisoned fixture. The hardened configuration keeps routine triage and blocks the privileged calls, including when the parser is explicitly configured to read directives from the data channel.
Wire those permissions to live tools and the potential damage is plain: a merge distributes the change, a collaborator can outlive it, and a reseeded cache can carry it into later builds. That is the compounding risk the fixture represents. The lab does not measure downstream spread, exponential growth, or a live model's injection success rate. Those need their own tests.
The control worth keeping is the gate. Separating data from instructions stops this parser from proposing the injected actions; the code-level policy still blocks them when that separation is deliberately bypassed in the simulation. An actual model needs its own evaluation. Its permission to merge should never depend solely on whether it understood a warning in a prompt.
Scope clarification, October 24, 2026: stage three records simulated effects. Earlier wording described those effects as if the lab performed live GitHub and cache operations. The implementation and tests demonstrate the parser boundary and policy gate; they do not establish a live-model success rate or propagation result.
How to hunt them, red and blue
The method is the same from both chairs, which is why I like it.
Enumerate the dependencies of the thing you care about. For each one, ask what happens to the protected system when that dependency is slow, then when it is gone. Write the answer down as a security statement, not an availability one: when the policy engine is unreachable, access is granted. When said out loud like that, the fail-open defaults announce themselves.
Then test the degraded path on purpose. Do not wait for it. Kill the sidecar and replay the request. Starve the container and watch what the OOM killer takes. Hold the dependency in a slow-loris state and see whether the timeout path allows or denies. Crash the app and probe the boot window. These are one-line experiments in a lab you own, and they tell you in minutes what a year of uneventful uptime will never reveal.
For the blue chair the fixes are unglamorous and they are all in this piece already. Set resource and process limits. Fail closed on the controls that matter and accept the availability cost deliberately. Separate liveness from a readiness check that exercises the security path. Bound and ship your logs. Replace the reflexive always restart with a backoff that an attacker cannot pump. None of it is clever, and none of it ships by default. Most teams will read this, agree with all of it, and change nothing, because the system looks fine right up until the day someone forces the state they never tested.
What competence actually looks like
A correctness bug tells you a system is wrong. A reliability defect tells you it is fragile, and fragility becomes a security property the moment someone can trigger it on demand. The failure modes are part of the attack surface, and most have never been looked at by anyone paid to think like an attacker, because they were filed under operations by people who were certain they did not need to look.
Containerizing a service is not competence. Knowing what your container does when it is starved, crashed, restarted, or cut off from the thing it trusts is competence, and it is rarer than the confidence that stands in for it. The engineers worth working with are the ones who will say out loud that their sidecar fails open, and then go test it. The ones who treat the question as an insult are the ones already carrying the defect into production, sure they are not.
So when the test passes and the process will not exit, when the container restarts a little too eagerly, when the sidecar times out and the request goes through anyway, do not reach for the word flaky. Ask what state the system is in, and whether it is one you would mind an adversary being able to summon. If the honest answer is yes, you have found a way in, and it has been sitting in the backlog, under a green checkmark, the whole time.
Correction, October 24, 2026: Replaced placeholder configuration and incomplete exception snippets with prose tied to the linked lab. Distinguished host-wide OOM risk from the lab's deliberately constrained sidecar failure.