Blog
/
Share this article

StackGen’s DevOps Lead on What Changes When AI SRE Meets Enterprise Reality

StackGen’s DevOps Lead on What Changes When AI SRE Meets Enterprise Reality

The AI SRE Files is a series where we take one practitioner conversation about AI in SRE and work through the idea at the center of it.

This article draws from the session “The Enterprise AI SRE Playbook,” presented by Vishwajeet Kumar at AI SRE Meetup Pune on August 8, 2026.

Building an AI SRE agent in your own environment gives you a fair amount of control. You know how the infrastructure is set up, which observability tools are connected, and where the agent can find the telemetry it needs to investigate an incident. But what happens when you take that same agent into an enterprise environment?

Some applications run on Kubernetes, while others depend on legacy VMs. Different teams use different monitoring tools, and service naming conventions may vary across environments. The agent also has to work within network restrictions, limited API access, and formal change approval processes.

Vishwajeet drew on StackGen's experience with internal AI SRE development and enterprise deployments to discuss the challenges teams face as they adopt AI SRE, from telemetry correlation and incident workflows to autonomy, security, and governance.

Internal AI SRE Tests Have a Home-Field Advantage

Vishwajeet compared StackGen's internal AI SRE environment with the conditions teams encounter during enterprise deployments. The differences go beyond infrastructure complexity. They also affect how an agent accesses telemetry, connects to production systems, and operates within existing change management processes.

table:
Operational area | Internal environment | Enterprise environment
Infrastructure | Cloud-native Kubernetes and OpenTelemetry | Kubernetes, legacy VMs, and hybrid cloud
Telemetry | Standardised metric names and consistent labels | Different naming conventions, missing tags, and fragmented logs
Network access | Direct SaaS endpoints and API connectivity | Proxies, private VPCs, and air-gapped environments
Change management | Automated deployment pipelines | Change advisory boards, ITIL processes, and strict audit windows

Consider an application running on Kubernetes that depends on a service hosted on a legacy VM. If the agent has access only to Kubernetes telemetry, it may identify where the errors appear without being able to investigate the dependency causing them.

Inconsistent tagging can make the investigation harder. Two services may be connected, but if their names and labels differ across monitoring tools, the agent may struggle to establish that relationship.

Before an agent can investigate incidents across these environments, the team needs to establish which systems it can access, what telemetry is available, and whether it has enough context to follow dependencies across the infrastructure.

An AI agent can have access to thousands of logs and metrics and still struggle to identify the root cause. Kiran Suthrave, Senior Director of Software Engineering, Cloud CoE, discussed this challenge in his session on LLM-powered RCA, where he explained why signal quality and telemetry correlation are important.

The Telemetry Problem Gets Harder Across Different Environments

During an incident, several services can generate alerts and logs for the same underlying failure. An AI SRE agent needs to establish how those signals are connected before it can investigate the likely cause.

This is the Telemetry Paradox described in the session. An agent may receive thousands of log entries, repeated stack traces, and alerts from several dependent services. If all that information is passed directly into an LLM, the model can mistake a symptom for the root cause or spend much of its context window on duplicate information.

Consider a database slowdown that causes errors across several dependent services. Each service may generate its own alerts and logs, even though they share an underlying problem. If the agent examines those signals independently, it may investigate several apparent failures before identifying the database dependency.

To address this, telemetry needs to be correlated before it reaches the LLM, using three elements:

  1. Service topology and dependency mapping: Identify how services are connected so the agent can trace failures across dependent systems.
  2. Correlation across telemetry sources: Bring together metrics, traces, and deployment information to examine related signals.
  3. Deterministic analysis: Isolate relevant changes and differences before the LLM begins its analysis of the incident.

Enterprise environments make this harder because service relationships may be missing or inconsistent across observability tools. An agent might have access to application logs and metrics but lack the deployment history or dependency information needed to investigate a possible cause.

For example, an agent may identify a recent deployment as a possible cause of an incident. However, if it lacks the telemetry needed to verify that connection, the investigation could overlook other contributing factors.

You need to know how much of the environment the agent can actually examine. If telemetry is missing or a dependency cannot be established, the investigation should make that limitation clear so your engineers can assess the RCA against the available evidence.

What Enterprise AI SRE Adoption Requires Beyond Incident Investigation

Integration With Existing Incident Workflows

An on-call engineer might receive an alert through PagerDuty, discuss the incident with the team in Slack, and open Grafana to check the relevant metrics. Once the incident is resolved, the team may use Jira to track follow-up actions.

Each tool has a role in the incident response process, and introducing an AI SRE agent adds another source of information that engineers need to work with.

The agent's investigation should be accessible through the tools engineers already use. For example, an alert in PagerDuty or Opsgenie could include an RCA summary, relevant metric changes, and additional context before the engineer starts investigating.

Slack or Microsoft Teams can provide access to the investigation within the incident channel. After the incident, integrations with Jira or ServiceNow can help prepare postmortem drafts and add relevant details to tickets.

The challenge for enterprise teams is deciding how these capabilities fit into their existing processes. An RCA may need to reach the incident owner, while a remediation recommendation may need approval from someone with production access.

These workflows are already familiar to engineers, so the agent needs to support how the team responds to incidents.

At GreytHR, the approach to AI autonomy depends on the type of incident and the risk involved. Abhishek Gaurav, Senior Director of Engineering, explained how his team decides which failures AI can handle automatically and which actions still need human approval.

Autonomy and Change Control

An agent that can investigate an incident may also be capable of recommending a fix. Before you allow it to make changes to production, you need to decide which actions it can take and what approvals are required.

The session outlined three phases of autonomy:

  1. Instant Investigator: The agent correlates telemetry, summarizes incidents, and provides diagnostic timelines for engineers to review.
  2. Action Plan Advisor: The agent prepares remediation pull requests, Terraform diffs, or runbook scripts that require human approval.
  3. Closed-Loop Auto-Heal: The agent executes remediation for well-understood, low-risk failures, with strict rollback conditions.

You can use this progression to assess the agent's investigation capabilities before you grant it greater authority.

In an enterprise environment, your change process may already involve change advisory boards (CABs), ITIL procedures, or restricted deployment windows. Even when an agent identifies the likely cause of an incident and proposes a reasonable fix, that change still needs to pass through the required approvals.

Consider an agent that identifies a configuration change as the likely cause of an incident. It could prepare a Terraform diff for your team to review, but the proposed change would still need to follow your existing approval process before execution.

The playbook recommends several controls for this process:

  • Policy-as-code: Use policies such as Open Policy Agent (OPA) to define which actions are permitted.
  • Role-based access control (RBAC): Restrict the permissions available to the agent and execution systems.
  • Audited execution: Route approved changes through CI/CD or infrastructure-as-code pipelines.
  • Audit logs: Record the proposed action, approval, and execution details.

You can evaluate these controls against your existing change process before you authorize remediation. Your engineers should be able to inspect the proposed change, understand its potential impact, and verify that the required approvals are enforced.

This also gives you a practical way to expand autonomy. You can begin with incident investigation, introduce approval-based remediation for familiar failure modes, and consider automated execution once you have enough evidence that the workflow is reliable and its risks are controlled.

Enterprise Security and Compliance

An AI SRE agent may need access to infrastructure metadata, application logs, traces, and deployment history to investigate an incident. In an enterprise environment, that information may sit across different networks, tools, and systems with their own access restrictions.

For example, your organization may require all API requests to pass through a private network or an approved proxy. Before you deploy an agent, you need to establish how it will access operational data within those restrictions and whether sensitive information can leave the environment.

Enterprise deployments also introduce several security and compliance requirements:

  • Data sanitization: Remove sensitive information, including personally identifiable information (PII) and protected health information (PHI), before telemetry leaves the environment.
  • Data retention: Establish controls over whether information sent to an LLM is stored or used for model training.
  • Private deployments: Account for environments where connectivity is restricted through private VPCs, proxies, or isolated networks.
  • Auditability: Maintain records of agent decisions and the evidence used to reach them.

These requirements affect which systems an agent can access, how it retrieves operational data, and where that data can be processed.

Your security team can assess whether the agent's access and data-handling approach meets organizational requirements, while your SRE team can evaluate whether the available evidence is sufficient to support its RCA.

AI SRE Performance Metrics and On-Call Impact

Once you introduce an AI SRE agent into your incident response process, you need to understand how much manual work it saves your engineers.

Mean time to resolve (MTTR) is one measure, but improvements in investigation time may take longer to affect overall incident resolution. Even when an agent identifies the likely cause quickly, a production fix may still require approvals or a lengthy recovery process.

You can assess its impact across three operational areas:

  1. Detection and investigation time: How quickly can your team identify an incident and investigate its likely cause?
  2. Alert noise: How many duplicate alerts, transient signals, and false-positive pages can the agent reduce?
  3. Operational toil: How much manual work remains for your engineers during incident investigation and follow-up?

Consider an incident where several dependent services generate alerts for the same underlying failure. If the agent correlates those alerts and provides an RCA with relevant metrics and logs, your on-call engineer can review the evidence without manually comparing every alert and dashboard.

You can compare the time engineers spend on these investigations before and after introducing the agent. It also helps to track how often they need to correct its conclusions or repeat the analysis themselves.

An RCA should reference specific metric graphs, log lines, and relevant changes so your engineers can verify its findings. If the agent reaches a conclusion quickly but the evidence is incomplete, your team may still need to conduct a separate investigation.

These measures can help you assess whether the agent reduces investigation effort and supports your incident response process before you expand its responsibilities.

How do you know whether an AI SRE agent can handle incidents your team has already resolved? Vivek Raja P S, Senior Solutions AI Architect at AIVar, tested an agent against five previously resolved P1 incidents. His AI SRE readiness assessment revealed gaps in operational context and outdated runbooks.

Start With One Service Before You Expand

You can start with a service your team knows well, assess how an AI SRE agent handles its incidents, and expand its responsibilities based on what you learn. This gives you a way to evaluate its performance within your existing infrastructure and approval processes.

With Aiden for SRE, you can connect your existing observability tools and select a service that generates frequent alerts. You can then compare investigation time, alert noise, and manual effort before and after introducing the agent.

Try the free Aiden for SRE Community Edition. Connect Grafana or Datadog and review how Aiden handles alerts from your environment before you expand to other services

Related posts
Nov 18, 2026
·

Building the Context Layer for Platform Engineering with StackGen

building-the-context-layer-for-platform-engineering-agents
Nov 17, 2026
·

What Is an AI SRE? A Guide for Enterprise Teams

what-is-ai-sre
Nov 13, 2026
·

Shipyard and Aiden for SRE are now on the Microsoft Marketplace

shipyard-aiden-sre-microsoft-marketplace
View more posts
×

Download the brochure

Complete the form to get your copy.