Give the Agent a Fair Environment: Director of Software Engineering on LLM-Powered RCA

This piece comes from the AWS Roundtable “Killing the 90-Minute War Room: Automated RCA” at AISRENext Bengaluru. 

The discussion started with a question about measurement: if the 90-minute war room disappeared, which metric would best capture the business impact?

Kiran Suthrave, Senior Director of Software Engineering, Cloud CoE, took the old-school view.

"My favorite, or the most popular one, I'd say, is the 5xx metric. I'm going old school. In a microservices world, everything is split across dependencies. 504s give you latency. 500s tell you where the mistakes are. That helps you pinpoint exactly where to start looking." 

The discussion then moved to LLM-powered RCA: how do you know whether the RCA it produces is any good?

A fast RCA still needs to survive technical review. Does the evidence support the root cause? Can the failure be traced across the affected dependencies? Does the diagnosis account for the symptoms engineers saw in production?

An Agent Needs the Right Incident Context 

For RCA, more context is not automatically better. An agent needs enough information to reconstruct the failure path without pulling in every signal the environment can produce.

Context selection becomes part of the RCA pipeline. Service topology, recent changes, relevant logs, traces, metrics, and dependency health can narrow the investigation. Unrelated telemetry increases the search space and gives the agent more signals to correlate without necessarily improving causality.

Context efficiency is another useful benchmark: how much incident data does the agent need before it can produce an RCA that holds up under technical review? 

RCA Quality Starts With the Data Pipeline 

Here, “stick to your basics” means getting the RCA inputs and processing in order:

• Signal collection: Metrics, logs, and traces collected consistently across services.
• Data formatting: Common timestamp and content formats across teams so the agent is not reconciling different schemas during an incident.
• Noise filtering: Critical operational logs separated from routine application noise.
• Correlation: Existing correlation engines continue to connect related signals before that context reaches the agent.

RCA accuracy is partly a data-quality problem. If timestamps are inconsistent, critical logs are buried in noise, or signals cannot be correlated across services, the agent starts its investigation with degraded evidence. Model evaluation should account for the quality of that evidence before attributing every RCA failure to the model itself. 

RCA Quality Depends on the Environment Around the Model 

Kiran pushed the model itself further down the list of priorities.

Model capability cannot compensate for missing telemetry, inconsistent timestamps, noisy logs, or weak correlation. Once those inputs are in place, teams can evaluate how much additional RCA quality a larger model actually delivers before paying the cost and latency that come with it.

"I'm not saying it's going to replace a human. But at least treat it that way, so it has a fair shot."

The moderator summarized Kiran's position at the end:

"I think this is something new: Kiran is batting for giving agents a fair chance."

A Fair Chance Starts With the Data 

Benchmark the RCA pipeline with the model. Signal coverage, timestamp consistency, noise filtering, and correlation determine what evidence the model receives during an investigation.

For model comparisons, keep those inputs fixed. Run the same incident data and context pipeline across models, then measure RCA accuracy, evidence quality, latency, and cost. Otherwise, gaps in telemetry can look like gaps in model performance.

Aiden makes that comparison possible inside a team’s own observability environment. Connect Grafana or Datadog, run an alert through Aiden, and inspect the signals it used, the correlations it found, and the confidence behind the RCA. Teams can see how the RCA was built against the telemetry they already run in production. Try Aiden for SRE Community Edition for free today.