The demo version of AI incident triage is seductive: alert fires, the agent reads it, the agent fixes it, you go back to sleep. The on-call version is messier: ambiguous alerts, partial telemetry, three services pointing fingers, and a human who still owns the pager.
We built AI triage into Aiden after watching on-call SRE teams drown in the first thirty minutes of an incident, mainly to shorten the gap between "something's wrong" and "we know where to look."
What follows is a field report from shipping this into Aiden: what held up under real on-call pressure, what didn't, and the runtime choice that makes it fast enough to be worth doing at all.
Incident triage isn't fixing. It's narrowing the search: What's broken? What changed? Who's impacted? What do we already know?
The work is repetitive and easy to parallelize. Every investigation starts with the same set of steps:
It's also where fatigue sets in. At 3 AM, it's harder to keep every thread in your head. The information you need is spread across dashboards, logs, deployment history, and past incidents. Pulling it together takes time, even when you know exactly what you're looking for.
We wanted an agent that handles the repetitive work upfront so engineers can focus on making decisions instead of gathering context.
Aiden doesn't walk your stack sequentially. When an alert hits the webhook path, the runtime triggers a parallel context-gathering workflow with bounded concurrency. Metrics spikes, recent rollouts, and similar past incidents are analyzed in tandem, then merged into a highly structured payload designed for the model's reasoning engine.
Here's a production-inspired example of how the agent structures this incoming data before it even begins to process a response:
Where humans spend fifteen minutes frantically switching between browser tabs, the orchestrator compresses that entire multi-system data-gather into a split-second parallel operation.
It hands the LLM a clean, structured set of facts rather than an empty, ambiguous prompt. The token budget stays tightly under control because the agent isolates the "gather" phase entirely from the "reasoning" phase.
The highest-value triage step was connecting dots humans reach for manually: deploy timestamps next to error spikes, saturation metrics next to queue depth, customer reports next to regional failure patterns.
The agent didn't need to decide the root cause. It needed to present a hypothesis board — "here are five things that changed in the last hour, ranked by plausibility, with links to evidence."
Operators told us this alone saved meaningful time. The agent never gets tired and never forgets to check the CDN when the API looks sick, something a fatigued human two hours into an incident easily misses.
Writing "12:04 — alert fired, 12:07 — deploy completed, 12:09 — error rate elevated" in Slack while debugging is friction nobody needs. An agent that drafts the timeline from audit logs and alert history, for human editing, keeps channels coherent without pulling engineers out of investigation mode.
Institutional memory is uneven. The engineer who fixed this exact failure two years ago might be on vacation. Searchable runbooks help, but only if someone remembers the right keywords.
Semantic search over past incident summaries and postmortems gave on-call a "have we seen this before?" answer in seconds.
Quality depended entirely on how well past incidents were documented—garbage in, garbage out, no AI magic. Embeddings don't invent missing postmortems; they only retrieve what you bothered to write.
On the ground, we spent more engineering time on chunk hygiene and metadata (service, severity, timeframe) than on the embedding model itself. The process of cleaning and structuring data before it is indexed mattered more than swapping vector backends.
After mitigation, writing the RCA is another chore. An agent that drafts structure, timeline, impact, contributing factors, and open questions from execution traces gave teams a head start. Humans still hold conclusions; the draft simply prevents staring at a blank doc.
Restarting pods because error rates spiked feels heroic in a demo. In production, it masks underlying issues, violates change policy, and trains teams to distrust the agent. We learned this lesson overlaps heavily with defense in depth for tool calls: triage agents investigate first.
|
The Anti-Pattern (What Doesn't Help) |
The SRE Framework (What Helps) |
|
Aiden: “The root cause of this incident is a memory leak in the payment service caused by the latest deployment of v2.4.1.” |
Aiden Triage Summary: Hypothesis 2 (Medium Likelihood): Bad deploy. Evidence: Image v2.4.1-rc3 rolled out ~4 minutes before the alert ([CD]). |
|
Why it fails: It is overly confident, lacks telemetry links, and forces the engineer to rebuild the entire investigation from scratch anyway. |
Why it works: Instead of forcing the user to blindly trust a black-box conclusion, it acts as a transparent verification framework by exposing explicit telemetry links. |
Instead of forcing the user to blindly trust a black-box conclusion, it acts as a transparent verification framework by exposing the exact telemetry links and structured context.
Building that assistant well means separating gather (concurrency, structured payloads) from narrate (the model).
This is what worked for us in production. See how it works for your stack, talk to us about Aiden for SRE.