AI SRE in Production: Seven Lessons from KubeCon 2026

On Monday night of KubeCon week, we hosted AI SRE Next at Gracie's in Salt Lake City, in partnership with Splunk. Registration was oversubscribed. The room filled with SREs, platform engineers and engineering leaders from companies including Adobe, Apple, Coinbase, Ford, TriNet and Securonix.

AI SRE Next is the community we started at StackGen for engineers putting AI to work in production operations. The idea is simple: AI SRE, from the people running it. Over three talks and a panel, one theme kept coming back. An AI SRE does good work when the environment around it is built for it. The evidence it can reach, the guardrails it runs inside and the way it learns matter as much as the model.

Here are the seven lessons I'm taking back from the evening.

Sanjeev Sharma opens AI SRE Next with the evening's agenda.

1. Govern agents the way you govern people

Dave Ahr from Coder opened with "The AI Operating Layer," based on Coder's recent white paper. His starting point was that everyone is a builder now. At a Coder all-hands earlier this year, people from recruiting, HR and finance showed apps they had built themselves. Governing AI has become a whole-company question.

Risk scales with privileges, not intelligence. Once agents hold credentials and reach systems of record, their mistakes rarely look like incidents. They look like work getting done. So Dave doesn't try to make the agent incapable of a mistake. He builds the environment so a mistake can't become an incident: one layer between every builder and everything they touch, which scopes access, audits every action and controls spend. Platform teams should own that layer, and the first step is observability. You can't govern what you can't see.

Dave Ahr (Coder): build the environment so a mistake can't become an incident.

2. Could you investigate with the data your AI SRE has?

Moss Normand from Splunk asked the question I'd put to every team evaluating an AI SRE. Without evidence and context, an assistant can mistake a symptom for the cause, suggest a fix that doesn't fit your environment, or repeat an approach you already ruled out. Hand it everything instead and it burns its context window, and your tokens, on irrelevant material.

His answer starts with a telemetry baseline, which OpenTelemetry makes easy to get. Then keep knowledge with the code: a short context.md, working notes, and the decisions and failed approaches behind them, written for the next assistant and readable by people. Let the agent reproduce the issue in a disposable environment, and write its findings back where the next agent will find them.

Moss Normand (Splunk): give your AI SRE the context to investigate.

3. Move from thresholds to evidence, and keep a human on promotion

Daniel Oh from IBM tackled gray failures. The light is green, but the car is making a strange noise. A static rule like "roll back if the error rate is over 1%" can't tell a real regression from a known upstream ISP outage.

Daniel demoed a multi-agent canary system live. One agent analyzes metrics, another investigates dependencies, and a decision agent weighs the evidence and recommends GO, HOLD or NO-GO. In one case every request returned 200 OK and the tests passed, but the agents caught extra retries behind each call. Left alone, that kind of problem eats your resources two hours later. Production promotion still needs a human, and deterministic policy has the last word. That matches how we think about autonomy at StackGen: bounded by policy, with people approving the decisions that matter.

Daniel Oh (IBM) on moving canary delivery from static rules to reasoning agents.

4. Enterprise context means procedures and state

I moderated the panel with Seshadri Reddy Kommepalli of Equifax and Matthew Lewis of Michelin. Both kept coming back to context, at a scale most demos never touch.

Matthew's team is moving past knowledge graphs toward procedures and state. When a ServiceNow ticket arrives, did it land right after the nightly mainframe batch? Is a queue backing up downstream? Seshadri's team built a base agent that knows their roughly 15 business teams, their microservices and their architecture. Every investigation starts there.

The panel: Sanjeev Sharma (StackGen), Seshadri Reddy Kommepalli (Equifax) and Matthew Lewis (Michelin).

5. Agent drift is an engineering problem

Agents change behavior when the model changes, when the harness changes, when they learn something, or when someone tweaks a prompt. Matthew turns good execution traces into evals and keeps checks deterministic wherever he can. Seshadri defines roles tightly. He is the architect, and the agent is the engineer.

6. Autonomy is earned one use case at a time

Both panelists stressed picking the right problem for the proof of concept, and both expect every agent to go through security review. Then comes the question I care about most: how much do you trust the agent to act on its own?

We think about this as an autonomy index. Which tasks can be delegated, and which are one-way doors? What evidence does the agent show, and how do you sample its work so you can raise autonomy with confidence? Seshadri is taking it use case by use case, and working out how agents go through the same change approval process people do.

7. Agents should learn from every incident

The room's last questions were about learning and memory. At StackGen, our agents write what we call reflections. After an incident, the agent writes up what it found and how it resolved it, the way an engineer writes a knowledge-base article, and a human reviews it before it is used again.

Matthew's team builds a daily summary of what changed across the business, so anyone new can see exactly what was done last time. He wants to go further. After every incident, even small ones, he wants to trace what changed and which architecture decision led there, and feed that back to the architects.

Questions from the room went to how agents learn from each incident.

What I'm taking back

Every session pointed the same way. The teams that succeed with AI SRE treat governance, context, evals and autonomy as engineering work in their own right. The model matters, but most of the effort goes into the environment around it.

That is what we are building at StackGen with Aiden for SRE, and it is why we started AI SRE Next. If you want to see how an AI SRE investigates with your own telemetry, try Aiden for SRE Community Edition, free for up to two users.

Thank you to Splunk for partnering with us, to Dave, Moss, Daniel, Seshadri and Matthew, and to everyone who spent a Monday night of KubeCon week with us. I hope to see you at the next AI SRE Next.