The Two-Hour Audit: AIVar’s AI Architect on AI SRE Readiness

The AI SRE Files is our series where we sit in on conversations with practitioners about AI in the SRE world and pull out one idea worth keeping.

This piece draws from “From Tribal Knowledge to Agent Context: How SREs Become Knowledge Engineers,” a panel discussion at AISRENext Bengaluru held on June 12, 2026.

Kunal Dabir, VP Engineering at StackGen, asked the panel a practical question: if you had only two hours to audit a team’s knowledge layer, what would you look at first?

Vivek Raja P S, Senior Solutions AI Architect at AIVar, shared an example from his own work.

Five P1 Incidents Became the Test Set

Vivek’s team already had an initial version of its AI SRE agent in place. They selected around five resolved P1 incidents, along with the resolution history for each one, and passed them to the agent individually.

“We gave each incident one by one to the agent to see how well it was able to reconstruct the solution, whether it was able to apply the fix.”

Because each incident had already been resolved, the team knew what the resolution path looked like. They could compare the agent’s response with what engineers had done during the actual incident.

But the proposed fix alone could not explain where the agent succeeded or where it lost context. For that, the team had to inspect its execution path.

The Execution Path Exposed the Gaps

For every incident, the team reviewed:

• The tool calls made by the agent
• The knowledge base it accessed
• The context it retrieved
• The resolution it constructed from that context

The trace showed where the agent’s response began to diverge from the actual incident resolution. Some of the required operational context still existed only with engineers and had never been transferred to a source the agent could query.

This gave the team a clear view of what was missing. They could identify the exact point where the agent ran out of context and trace that gap back to the knowledge source.

The same review exposed a second problem: the agent could reach the runbooks, but some of those runbooks no longer matched the current resolution process.

Runbook Quality Affected the Resolution

A runbook’s presence in the knowledge base does not make it reliable. The procedure still has to reflect the current system, its dependencies, and the remediation path.

Experienced engineers often supplement a runbook with context they have built over time. They know which dependency to check first, which step no longer applies, and when a documented remediation could introduce more risk.

The agent works with the context available through its tools and knowledge sources. If a runbook omits a decision point or reflects an older system state, that gap appears in the execution path and affects the proposed resolution.

Vivek’s team could see this in the incident replay. The agent retrieved the available material, but gaps in that material prevented it from reconstructing the complete solution.

The team now had two specific areas to address:

• Operational knowledge that still had to move from engineers into agent-accessible sources
• Runbooks that had to be brought in line with the current resolution process

Once those gaps were visible, the experiment had value beyond the initial two-hour audit.

The Experiment Became a Benchmark

The team worked to close the gaps and continued to evaluate how the agent handled real incident scenarios.

“It became a sort of benchmark eval for us to keep checking.”

The benchmark followed the same process: take a resolved incident, provide the incident data to the agent, inspect the agent's tool and retrieval path, and compare its response with the known resolution.

This gave the team a repeatable way to evaluate changes to the knowledge layer. When a runbook was updated, or more operational context became available, they could check whether the agent’s resolution path moved closer to the actual incident response.

The Knowledge Layer Needed Two Fixes

Five P1 incidents gave Vivek’s team enough evidence to find two specific issues: operational knowledge that the agent could not access and runbooks that needed attention.

The audit also showed why document counts and last-updated dates provide only a partial view of knowledge readiness. The stronger test is whether the agent can use that knowledge to reconstruct an incident that engineers have already resolved.

For an SRE team, the same approach can begin with a small set of high-severity incidents. Preserve the original incident data, tool outputs, investigation path, and final remediation. Then compare those records with the agent’s tool calls, retrieved context, and proposed resolution.

The result is a concrete list of gaps tied to real incident response: missing context, inaccessible knowledge, outdated runbooks, or an incomplete resolution path.

At StackGen, Aiden is built to bring operational context into the investigation, including signals from the tools and knowledge sources your team already uses. The quality of that investigation depends on the context available to the agent and how current that context is.

Try Aiden for SRE Community Edition for free today.