Share This:
An incident investigation moves through a series of dependent decisions. You first establish when the problem started, identify the systems involved, check the initial hypothesis against metrics, logs, or traces, and then bring the evidence together into a likely root cause.
We built an agent workflow around that same sequence. Each stage handles one part of the investigation and passes its output to the next.
That structure works well until the final answer is wrong. At that point, the output tells you very little about where the failure started. The first stage may have selected the wrong time window. The next may have ranked the wrong suspect. A later stage may have skipped a required query and still produced a plausible summary.
We ran into this while testing a multi-stage investigation workflow. Running the complete pipeline showed us that something had failed, but not where. It was also slow, consumed tokens across every stage, and gave us a different result from one run to the next.
We changed the way we tested it by borrowing a pattern from hardware bring-up: enable one stage, check it against an explicit gate, prove that it passes repeatedly, and only then add the next stage.
This blog explains how that process helped us narrow failures to a specific stage, control the cost of each test run, and catch a problem in an unexpected place: the evaluator itself.
Why the Final Answer Does Not Show You Where the Workflow Failed
The investigation pipeline had a fixed sequence:
- Establish the incident time window.
- Identify and rank the likely suspect.
- Corroborate the hypothesis against another data plane.
- Synthesize the evidence into an answer.
Each stage consumes the output of the one before it. A bad decision early in the workflow can therefore affect every downstream stage.
This is a common compound AI architecture: the workflow is deterministic, but a non-deterministic model operates inside each node.
If the final answer is incorrect, an end-to-end run leaves several possibilities open. An early stage may have passed bad context downstream. A later stage may have misinterpreted good context. The final synthesis may be fluent while hiding a broken intermediate step.
Full runs also make the development loop expensive:
- Every stage consumes tokens and may fan out into several tool calls.
- A complete run takes minutes rather than seconds.
- The same input can produce a different result on the next run.
The usual loop of running the workflow, reading the transcript, changing a prompt, and running it again does not provide enough evidence. One successful result may mean the change worked, or it may be a favorable sample from a non-deterministic system.
We needed a way to isolate failures without replacing the live behavior of the earlier stages.
Start With One Stage, Then Add the Next
The approach is cumulative. At level N, we run stages 1 through N for real and stop before stage N+1.
Bring-up is cumulative. Earlier stages remain live at every level, while downstream stages stay disabled until the current level passes consistently.
The loop looks like this:
- Trim the pipeline to the stage currently under test.
- Run it against a predefined success gate.
- Fix the current stage if the gate fails.
- Repeat the run until the stage passes consistently.
- Add the next stage and repeat.
This is different from testing every stage against frozen fixtures. If stage 3 depends on stages 1 and 2, it needs to handle the actual variance produced by those stages. A cached, ideal output from stage 2 would make stage 3 easier to test, but it would not represent the workflow it will see in production.
The earlier stages therefore remain live. We save time and cost by cutting off the stages above the one under test, including their downstream tool-call fan-out.
What a Gate Checks
Before running a stage, we define what it must guarantee. We call that condition a gate.
For example:
The table gives us a bring-up ladder. Each row adds one stage and the checks that the cumulative workflow must pass at that level.
A gate operates on the concrete effects produced by the agent:
This example is illustrative rather than the production gate set. The important part is the interface: the gate inspects what the agent did, represented as typed tool calls and arguments.
Writing reliable gates takes real engineering work. In some cases, defining the gate is harder than writing the prompt. Gates also need maintenance when the workflow requirements change. We treat the bring-up ladder as a test suite and accept the same maintenance cost.
That cost is useful because the gate also acts as a small specification. Writing “this stage must issue a metrics query with a concrete time window” often reveals ambiguity in the stage definition before the first run.
How a Failing Gate Narrows the Search
At level N, stages 1 through N-1 have already passed their gates repeatedly. If the new cumulative run fails, the latest stage is the first place to investigate.
This does not prove that earlier stages can never regress. Their gates still run at every level. It means a failure can be associated with the first gate that changed from green to red instead of being inferred from the final narrative.
Define Success Before the Run
Without a gate, it is easy to judge an agent run by whether its response sounds reasonable. That is too subjective for a multi-stage operational workflow.
The gate forces us to define the output contract before seeing the result. A stage either produced a concrete window, issued the required scoped query, or collected evidence from a second data plane. The evaluation is tied to observable behavior rather than the quality of the prose.
Why One Passing Run Is Not Enough
A single passing run is not enough for a non-deterministic system. We promote a stage only after it passes repeatedly.
There is no universal number of runs that establishes reliability. Running a paid frontier model thousands of times is rarely practical, and a small sample does not justify a claimed 99th percentile. We use a risk-based threshold instead.
A cheap early stage with limited impact can earn confidence with fewer cycles. A stage that names the likely cause of an incident needs more repeated passes because an on-call engineer may act on its output.
The practical rule is to run the stage enough times that a failure would be unexpected, then increase that standard with the blast radius of the stage.
Keep Each Test Run Bounded
During development, we run only the portion of the workflow required to test the current stage. Downstream stages remain disabled until they are needed.
This avoids paying for the entire pipeline after every small change. It also reduces the number of tool calls and the amount of output an engineer must inspect while working on a specific stage.
The Bugs This Approach Helped Us Find
The cumulative tests found several problems that a final answer could easily hide:
- An incorrect time window: One stage selected a window that was too narrow, so downstream correlation missed the actual start of the incident.
- The wrong signal at the top of the ranking: A ranking stage selected the largest spike rather than the signal most likely to have driven the incident. It treated correlation as causation.
- An overly broad discovery call: A later stage skipped the precise query required by the runbook and used a broader discovery call. The call returned data and allowed the workflow to continue, but it was not the constrained path we wanted the agent to follow during an incident.
Because each problem first appeared when a specific stage was added, we could address it as a stage-level regression rather than search through the entire workflow.
When the Evaluator Became the Bug
The most useful failure we found was not in the agent workflow. It was in the evaluation code.
One stage kept failing even though the transcript showed that the agent had followed the runbook. We changed the instructions and reran the workflow, but the result stayed red.
The scorer was searching the complete event stream for a forbidden pattern. The same pattern appeared in the runbook as part of an instruction telling the agent not to perform that action.
The string matcher could not distinguish between these two cases:
- The prompt says, “Do not call X.”
- The agent calls X.
Both placed the same text in the event stream. As a result, a correct run still failed the gate.
We saw the same issue in a check that counted spawned subtasks. The relevant phrase appeared in instructions and streaming scaffolding as well as in actual invocations, so the scorer reported more subtasks than the agent had created.
The evaluator was measuring the contents of the transcript, not the behavior of the system.
A transcript contains both instructions and actions. A string matcher can mistake a prohibited action named in the prompt for an action the agent performed.
Evaluate Tool Calls, Not the Full Transcript
We fixed the problem by pointing every behavioral gate at committed tool calls and their arguments.
The raw transcript is contaminated by design. It can contain:
- System and runbook instructions
- Model output
- Warnings about forbidden actions
- Streaming copies of the same event
- Tool-call results and framework scaffolding
That makes it useful for debugging, but unreliable as the source of truth for behavioral assertions.
Committed effects are a better boundary. If the question is whether the agent called a tool, used the correct time window, or supplied a required parameter, the scorer should inspect the typed tool call.
This is the agent equivalent of testing state rather than asserting on log output. Most engineers would not validate an application state transition by searching println output. Agent runtimes make this mistake easy because the full execution appears as one large stream of JSON.
The Go type boundary helped enforce the correction. A gate that accepts []ToolCall cannot accidentally match text in the prompt because the prompt is not part of its input.
Once we moved the gates from raw text to effects, the false failures disappeared. The remaining failure was genuine: the stage that had used the overly broad discovery call.
What We Would Keep for the Next Workflow
- Test multi-stage workflows cumulatively: Run stages 1 through N with live outputs, then stop before N+1.
- Define the gate before the run: Each stage needs an explicit behavioral contract.
- Require repeated passes: One successful run does not establish stability in a non-deterministic system.
- Scale validation to blast radius: Stages whose output can influence operator action need a higher confidence threshold.
- Keep iteration bounded: Do not run downstream stages while testing an earlier part of the workflow.
- Score effects rather than transcripts: Use committed tool calls and their arguments as the source of truth for agent behavior.
The main benefit of this approach is not only faster debugging. It gives us a defensible answer when someone asks why we believe the workflow works.
Each stage has a defined contract. Each contract is tested against live upstream variance. Each stage must pass repeatedly before the workflow expands. The final end-to-end run is then built on evidence collected at every level, rather than confidence in a single successful execution.
That is the standard we want for SRE agents. The engineer holding the pager should be able to inspect what the agent did, verify the evidence behind its output, and understand which parts of the workflow have been proven stable.
If this is the standard you expect from an SRE agent, see how we’re building it into Aiden for SRE.
