Blog

Trust But Verify: Talkdesk's Platform Engineering Lead on Human-in-the-Loop RCA

Written by Anjali | Aug 4, 2026, 11:13:31 AM

The AI SRE Files is our series where we sit in on conversations with practitioners about AI in the SRE world and pull out one idea worth keeping.

This piece draws from "Killing the 90-Minute War Room," a panel discussion at AISRENext Bengaluru held on June 12, 2026, featuring leaders across enterprise reliability, product, research, and platform engineering.

Partway through the session, a question came up that every SRE team eventually has to answer: where exactly does human-in-the-loop end and autonomous RCA begin? Are we there yet, and what would it actually take to get there?

Mudit Mathur, Senior Director and Platform Engineering Lead at Talkdesk, shares:

"A single AI agent should never be given end-to-end responsibility for generating or approving an autonomous RCA."

Agents Can Correlate. Trusting Them Is the Challenge.

Talkdesk runs highly event-driven systems, handling live customer interactions across voice, self-service flows, and contact center agents in real time. In that kind of environment, an infrastructure event in one place can ripple into an application several layers away, through dependencies nobody mapped ahead of time.

Mudit's approach to that complexity came down to one philosophy:

"It's always trust but verify philosophy."

An agent can build a knowledge graph. It can build a context graph at the moment an issue starts. It can correlate an alert against history and tell you it's happened fifteen times in the last month. All of that is useful. None of it, on its own, is a decision an agent should be allowed to make alone.

 

Two Places Where Judgment Still Beats Automation

Mudit split the problem into two distinct moments where human judgment does work an agent can't.

Before the incident is even declared:

Not every alert deserves a call. Some are false positives. Some are operational quirks an engineer recognizes on sight and lets resolve on its own. An AI agent, by contrast, tends to push toward action:

"What you observe is that you give that to an AI agent, and it will always go outcome-based to an action that you need to take."

That bias toward action is exactly why a human still needs to sit in front of the decision to escalate.

Understanding business context:

This is the harder one. Agents and monitoring systems will tell you what happened. AI can reason over technical signals, but severity ultimately depends on customer commitments, business criticality, ongoing deployments, regulatory obligations, and operational context: areas where experienced incident leaders still provide essential judgment.

"It's very hard to train an agent to understand the business context of the problem. That's where I believe humans are very, very critical."

Two components can misbehave at the same moment and carry completely different weight depending on what they actually serve. That weighting still lives with people.

A Tool Migration Is Its Own Incident, All the Time

Mudit described the reality of standing up DevOps agents on top of an existing observability stack, moving between Dynatrace, Datadog, New Relic, and Grafana:

"Every time I've gone through it, 20 to 30% of SRE capacity is going into transitioning from one tool to another."

That's not a one-time migration tax. That's not a one-time migration tax. Tool migrations consume enormous engineering effort because operational knowledge, dashboards, alerts, runbooks, and workflows must all be rebuilt or validated. Any autonomous RCA strategy has to account for the fact that the underlying observability layer itself is rarely stable.

 

Systems Provide the Most Objective Evidence Available During an Incident

Mudit's take echoes what Abhishek Kundalia said in "Systems Don't Lie" — that systems, not human memory, hold the ground truth of an incident. Mudit picks up that same idea in the second half of his answer, when he moves to what happens after an incident is mitigated: writing the RCA.

He called it the task every engineer dreads, and the process he described explains why. Developers get pulled into a room, sometimes for hours, and each person naturally writes up the part of the system they know best.

Where he sees real value is in how agent-generated RCAs reconstruct the sequence of events directly from telemetry and timelines, reducing dependence on memory and minimizing hindsight bias:

"Systems don't lie. Systems give you what systems have done, what they've worked and operated upon."

An agent has no stake in how the story gets told, so it reports what ran, what didn't, and when, without needing to be reconciled across five different perspectives.

What it can't yet do is capture the texture of the incident bridge itself: which theory the team chased first, which fix actually worked, and what got ruled out along the way. Mudit discussed the importance of incorporating incident discussions into future RCA workflows, partly because some Talkdesk customers, particularly in healthcare, expect a level of transparency about what happened and why.

The Common Thread Across This Panel

Mudit's "trust but verify" framing sits close to what Abhishek Kundalia of Pocket FM said in the same conversation about systems replacing human memory as the starting point for an investigation.

Both agree systems hold the ground truth.

What Mudit adds is the reminder that ground truth and full autonomy aren't the same thing, and that the business judgment layered on top of that data is still squarely a human job.

On whether the industry has arrived there, Mudit summed up:

"Today, we're not at a state where we can say autonomous RCA is something we can completely rely on. For us, it's still human in the loop at all of these checkpoints, to verify.

Rather than relying on a single autonomous agent, Mudit described a future where specialized agents collaborate: one assembling telemetry, another building dependency context, another correlating historical incidents, and another drafting the RCA. Human engineers remain responsible for validating conclusions and making business-impact decisions."

At StackGen, Aiden is built around that same split: agents assemble context and surface correlated signals fast, while the decision to escalate, act, or defer stays with the engineer. Try Aiden for SRE Community Edition for free today.