Blog — Stanislav Chernykh

An agent that can run commands across a fleet is a genuinely good debugging partner, but only if you impose the discipline that a good engineer already has. Left to itself it will do the thing everyone does under pressure: form a theory early, find something that looks like confirmation, and declare victory.

The premature root cause

The pattern is reliable enough to plan around. Something breaks shortly after a change. The agent finds a plausible link between the change and the symptom, announces the root cause, and proposes a fix. It is fast, it is articulate, and it is often wrong — not because the reasoning was bad but because the evidence was never tested against an alternative.

I hit this recently on a tunnel that died around the time a TLS certificate was swapped. The correlation was strong and the mechanism was plausible. It was also incomplete: a control test with the certificate path removed entirely failed in exactly the same way. There were two independent faults, and the tidy story about the certificate accounted for only one of them.

Controls are the whole discipline

The habit that separates debugging from guessing is asking, before believing any theory: what observation would this theory forbid?

If the CDN is stripping request parameters, then a request that bypasses the CDN must not exhibit the fault. That is a five-minute test, and it is worth more than an hour of log reading, because it can falsify rather than merely decorate the theory. Running it turned one confident wrong answer into two correct ones.

Agents are bad at this by default. They are trained to be helpful, and a control test that might demolish your own conclusion does not feel helpful. You have to ask for it explicitly, and you have to be willing to hear that the answer you already reported was wrong.

Eliminations are the artifact

By the end of a long investigation, the valuable output is rarely the fix. It is the list of things that are now known not to be the cause, each bought with real time. Certificate: no. CDN cache key: contributory but not sufficient. Stale sessions: no, both ends restarted. Config drift: no, byte-identical to a known-good era.

Write these down as they happen, in durable form, because they are exactly what evaporates when context is compacted or the session ends. An investigation that loses its eliminations restarts from zero, and the second pass is not faster than the first.

Restarts are not diagnosis

The most tempting move available to an agent with shell access is to restart something. It is cheap, it occasionally works, and it destroys evidence. Worse, when it does work you learn nothing — you cannot distinguish “stale state” from “the thing that would have recovered anyway”.

Restart when you have a specific hypothesis that predicts a restart will fix it. Otherwise, look first. State that will be gone after a restart — open connection counts, process uptimes, in-memory config that no longer matches disk — is often precisely the evidence that identifies the fault.

Reversibility over cleverness

Any change to a production system during an investigation should come with its undo already written. Back up before editing. Change one variable at a time. Prefer a temporary log level over a config rewrite. Prefer a throwaway instance on a spare port over modifying the running one.

This is ordinary engineering discipline, but it matters more with an agent in the loop, because the agent can make ten changes in the time you would make one — and ten simultaneous changes to a broken system produce a state nobody can reason about, including the agent that made them.

What agents are genuinely good at

Breadth. Checking twelve hosts for the same symptom, correlating timestamps across services that log in different formats, reading a 1.4 GB log for the four lines that matter, holding a dozen threads without dropping any. That is real, and it compresses investigations dramatically.

The judgment about what a piece of evidence means still needs a skeptic. Be the skeptic, or explicitly ask the agent to be one.