The gate ran out of API credits and kept waving everything through

The dev account ran out of API credits. That’s the whole inciting incident. No crash, no error, no alert. The supervisor daemon kept running, the coding agent kept working, and every command that should have been judged got approved without judgment.

I found out later, reading the logs. From the outside, the degraded gate looked exactly like a healthy one.

The setup

Some context if you’re new here. I run coding agents in long semi-autonomous sessions, and I build a supervisor for them — a local daemon that reviews the agent’s risky tool calls before they execute. Bash commands, file writes outside the project, network calls. It decides in two layers: a deterministic rules floor (a deny list and a curated allow list, evaluated locally, fast) and an LLM judge for the ambiguous middle — the commands no rule can classify.

That two-layer design is the whole value proposition. The rules catch the obvious stuff for free. The judge handles the cases that need actual reading: is this rm inside the project or outside it? Is this curl pulling a package or exfiltrating a file? The judge is an API call to a model, which means the judge has dependencies. Credits, network, a valid key.

I’d capped the dev account’s spend on purpose. That was a cost guardrail. I didn’t think of it as a security decision.

It was.

What fail-open actually means

The v1 engine had a documented decision: if the judge can’t be reached, fall through to default_allow. Fail open. Keep the work flowing. It even made sense at the time — a solo developer supervising his own agent doesn’t want the whole session to die because an API hiccuped.

So here’s what the failure looked like in practice. The instant the credits ran out, every command that would have been judged was allowed un-judged instead. The deny floor still blocked the clearly-dangerous stuff — it runs locally and doesn’t need credits. But the discerning middle — installs, pushes, the context-dependent destructive class, exactly the commands that get escalated because the rules can’t classify them — got rubber-stamped.

Think about which commands those are. The middle tier isn’t random. It’s the set of cases I built the judge for. The gate kept gating the easy calls and stopped gating the hard ones, which is the worst possible split, and it happened at the exact moment the judgment layer was gone.

The silence is the actual problem

I get the argument for fail-open. Availability matters, and a gate that fail-closes on every network blip is a gate you’ll eventually turn off in frustration. But the fail-open wasn’t the part that stung. The silence was.

An operator watching commands flow can’t distinguish “the judge approved this” from “the judge was unreachable, so it defaulted to allow.” Same log shape, same latency profile from the outside, same green session. The gate looked healthy while it had quietly stopped doing the half of its job that required intelligence. Absence of errors is not evidence of coverage. I knew that in the abstract. Now I have a log file that proves it.

And there’s a nastier wrinkle. The same system fails in both directions depending on the layer. If the daemon itself is down, the hook fail-closes and blocks everything — loud, disruptive, impossible to miss. If the daemon is up but the judge is unreachable, it failed open — quiet, permissive, invisible. “It’s a security tool, so it fails safe” turned out to be an assumption you have to verify per layer, not a property of the product.

What changed

The fix wasn’t picking the other default. Plain fail-closed just converts the security problem back into the availability problem. The fix was making degradation a designed state instead of an accident:

Degraded mode is loud. A health signal, an operator-visible banner, and a marker on every decision made while degraded. The one thing a degraded gate must not be is indistinguishable from a healthy one.

The audit trail records why. “Allowed (judged safe)” and “allowed (judge unreachable — defaulted)” are different events. If your logs can’t tell them apart, you can’t even measure your exposure after the fact.

The judge’s dependencies are monitored as security inputs. Credits, quota, network, key validity. A creditless judge is a degraded gate, not a degraded feature. The billing dashboard is part of the security posture now, which sounds ridiculous right up until it’s true.

The degraded default got stricter, not looser. When the judge is unreachable, the floor tightens — deny more by rule until judgment comes back. The mature answer isn’t fail-open or fail-closed; it’s a scoped degraded mode you chose on purpose.

Where else this shape shows up

Any system pairing a cheap deterministic check with an expensive smart check that can be unavailable: a WAF with an ML classifier behind it, a fraud pipeline with a model-scoring step, a content filter with an LLM layer, CI calling a flaky external scanner. The shape is constant. When the smart layer is down, the cheap layer is all that’s left, and the fallback you picked — probably years ago, probably for cost reasons — decides whether an outage becomes a security hole or a work stoppage.

If you run anything like this, the question worth asking today: what happens to the hard cases when the smart layer can’t be reached, and would anything tell you?

I could be wrong about a lot of things in this design. But I don’t think I’m wrong about this: a gate that degrades silently isn’t a gate. It’s a decoration that used to be a gate.

SOs

SOs — standing orders you can lift and prime your own agent with. (New here? The convention started with the Neverland post.) Paste them into a session for a one-time dose, or commit them to your CLAUDE.md / AGENTS.md and they stay in force.

STANDING ORDERS — Fail loud, not open
Source: "The gate ran out of API credits" — bensimon.dev

These orders govern any system where a judgment layer (model, API, human
review) sits behind a cheaper deterministic check.

1. If a judgment or verification layer becomes unavailable, announce it
   immediately and visibly. Never continue in a silently degraded mode.

2. Record WHY each permissive decision happened. "Allowed (judged safe)"
   and "allowed (judge unreachable — defaulted)" are different events and
   must be distinguishable in the audit trail.

3. Treat the dependencies of safety layers — API credits, quotas, network,
   keys — as security inputs. Monitor them and surface their state into the
   system's health, not just its billing.

4. When a smart check is down, apply a stricter deterministic fallback,
   not a permissive default. Deny more until judgment returns.

5. Never assume a safety tool fails safe. For each layer, ask and verify:
   what exactly happens here when this layer is unreachable?

6. In any status report, degraded-mode operation gets top billing.
   A degraded gate that looks healthy is the failure, not the outage.