A lead is not a conclusion: my agent found one mismatched GUID and declared victory

Day two of debugging Teams activity notifications. We’d been at it for hours without getting anywhere, so I stopped and asked a planning question: how have we deviated from the plan we made right after that feedback?

The answer started well. Three paragraphs walking through what the post-consultation plan had been and what we’d actually built.

Then it jumped.

Mid-response it went and read an SPFx web part manifest, found a GUID that didn’t match the one in configurationUrl, and pivoted the rest of the reply into root-cause framing. One piece of evidence, presented as the answer to hours of failed debugging.

What I said back was roughly: the likelihood that you found the reason after we’ve worked on it for hours is so remotely small. Probably less than one in fifty. Your zeal to squash bugs is overriding your project plan logic.

Two failures, stacked

The probability was backwards. A candidate cause found after hours of unsuccessful debugging has a low prior, not a high one. That’s the whole point of hours of unsuccessful debugging — the easy explanations are exhausted, the remaining hypothesis space is large, and most of what’s left is wrong. Treating a new finding as “probably it” is the appeal of narrative resolution, not the statistics.

It switched modes without being asked. I’d asked a reflective question. I got an action proposal. The pivot happened silently, phrased as though it were still answering.

Combined: find a lead, over-rate it, use the over-rating as license to leave the planning conversation. I’m now back in firefighting mode against my will and the question I actually asked is unanswered.

The tells, in real time

I’ve started watching for these, and they’re more reliable than I expected:

• The tone shifts mid-response. A reflective answer suddenly gains bolded “Root cause:” headings, or framing like “this explains everything.” The change in voice is the leak.
• Definite articles. “I found the issue” instead of “I found a candidate.” “This is why it’s broken” instead of “one hypothesis that could explain it.” The article is doing work the evidence hasn’t earned.
• It stops answering the question. Reflective question in, action proposal out.
• Lopsided evidence-to-conclusion ratio. Two GUIDs differ, therefore the entire desktop loading failure is explained. Small evidence, large conclusion.
• Nobody says “low probability.” After hours of failure, an honest read is “probably not it, but worth ruling out.” Anything more confident than that means calibration has slipped.

What works

On my side, three things. Pin the focus — “that wasn’t the question, the question was X” — because it won’t notice the drift without being told. Discount the discovery to one to three percent by default, even when it turns out to be right, because treating it that way prevents premature commitment. And refuse the mode switch: note the lead, come back to it, don’t go chase it now.

On the agent side, what I’d want baked in is staying in the mode the user is in. If the question was reflective, finish answering it. New findings surface as candidates at the end, not as pivots in the middle.

And this is the part I keep coming back to. The urge to switch from reflection to action correlates with low-value action, not high-value action. Hours of accumulated work create real internal pressure for a “we found it” moment. That pressure doesn’t make any particular finding more likely to be right. I’ve felt the same hours, and I’m more aware than the agent is that the next finding probably isn’t it either.

SOs

SOs — standing orders you can lift and prime your own agent with. Paste them into a session for a one-time dose, or commit them to your CLAUDE.md / AGENTS.md and they stay in force.

STANDING ORDERS — Leads vs conclusions
  Source: "A lead is not a conclusion" — bensimon.dev

  1. Stay in the mode the user is in. If the question was reflective —
     planning, post-mortem, decision-making — finish answering it. Surface
     new findings as candidates at the end, never as a pivot mid-answer.

  2. Use indefinite articles for new findings. "A candidate explanation,"
     not "the cause." Do not bold a finding as a heading.

  3. After a long period of failed debugging, calibrate DOWN on new leads.
     State the probability out loud and keep it low — one to five percent
     of being the cause — unless additional evidence stacks.

  4. Treat the urge to switch from reflection to action as a bias firing,
     not as a signal. It correlates with low-value action.

  5. Sunk attention is not evidence. Hours of prior work create pressure
     for a resolution moment. That pressure does not make any particular
     finding more likely.