My escalation path never fired. That wasn't good news.

The judge chain was Haiku, then Sonnet, then Opus. Cheap model decides the routine cases, escalates when it isn’t sure. Standard cascade.

The trigger was simple. Escalate when Haiku reports confidence: "low".

So I ran an adversarial evaluation against it. Deliberately fed cases the judge should have been unsure about.

Haiku returned zero low-confidence verdicts. Not few. Zero. It reported high even on verdicts that were wrong, and I confirmed the wrongness in neutral re-tests.

The escalation gate had been structurally dead since the day it shipped.

Why that’s worse than it sounds

The bigger models did carry the signal. Opus returned the correct ask on cases Haiku had confidently blown. The competence was sitting right there in the chain.

Nothing could reach it.

And the corruption lands exactly where it hurts most. A judge that’s wrong-but-confident produces a high-confidence signal on precisely the cases the escalation tier existed to catch. The headline catch rate looked fine at roughly 92 to 96 percent. The residual misses were confident, destructive false-allows. Worst possible failure shape, hiding behind a decent-looking number.

The fix that didn’t work

The obvious move is to re-point the trigger at a different signal. I tried that and it died on inspection.

Both candidate signals were downstream of the same judge, or too narrow to help. The judge’s own self-categorization mis-tagged its confidently-wrong cases as “routine dev workflow” — corrupted in the same direction as the confidence score. And the deterministic risk-tagger only knew rm-class commands, so its vocabulary couldn’t reach the cases that mattered.

Confidence and category were both corrupted, on the same cases, for the same reason.

The honest conclusion is that there is no trigger you can build out of an over-confident judge’s own output. Every field it emits fails with the same bias as the verdict it accompanies.

What that means

Self-reported confidence is part of the model’s output. It is not a measurement of it. Treating it as an independent check is circular, and I’d been doing exactly that.

The second thing is nastier. Calibration failure is silent and looks like success. A dead escalation gate produces no errors, no latency, and no extra cost. The chain “works,” cheaply. You only find it by deliberately feeding it cases it should struggle with and checking whether the gate ever fires.

So if your escalation tier has never fired, that’s a finding. Not a comfort.

I think this generalizes past model cascades, though I’ve only got the one system to go on. Triage nurses self-assessing case difficulty. Junior reviewers deciding what to escalate. Support bots deciding when to hand off. The shape is the same — the competence to recognize a hard case and the competence to handle it are the same competence. So the tier most likely to need help is the tier least able to know it.

The load-bearing fixes are upstream. Recalibrate what elicits low confidence at the prompt level, or use a genuinely out-of-band signal: a deterministic classifier with adequate vocabulary, a second independent model, target-state metadata. Something the judge didn’t write.

SOs

SOs — standing orders you can lift and prime your own agent with. Paste them into a session for a one-time dose, or commit them to your CLAUDE.md / AGENTS.md and they stay in force.

STANDING ORDERS — Escalation triggers
  Source: "My escalation path never fired" — bensimon.dev

  1. Never key a safety trigger on the judged party's self-assessment.
     Confidence scores, self-assigned categories and self-rated severity
     all fail with the same bias as the verdict they accompany.

  2. When asked to audit a tiered system, ask what the escalation keys on.
     If the answer is the cheap tier's own output, say the tier may be
     structurally dead and propose an out-of-band signal.

  3. Report "the escalation path has never fired" as a finding, not as
     evidence the system is working. Zero firings means either no hard
     cases — verify that — or a dead gate.

  4. Independence is a property of the source, not the field name. Before
     trusting a second signal, check whether the same model emits it.

  5. Build the adversarial corpus before relying on the chain. A gate that
     has never been deliberately stressed has never been tested.