How Online Services Actually Break: A Data-Backed SRE Failure Mode Taxonomy

When a service goes down, the SRE instinct is to ask what broke — but the more useful question is how did it break? The pattern of failure tells you more about detection speed, remediation path, and prevention strategy than any single root cause.

We analyzed 178,000+ status page incidents across 360+ online services and 1,037 detailed RCA reports to build a structured vocabulary for how systems fail: the StackGen Failure Mode Taxonomy. 30 named patterns, 8 families, grounded in data.

For the full dataset and interactive benchmarks, see the StackGen State of Reliability 2026 report.

Why Naming SRE Failure Modes Matters

Two incidents with identical root causes can look completely different depending on how the failure propagated. The failure mode is the pattern. The root cause is the trigger. Confusing them leads to the wrong remediation.

The 8 Families

Family 1: Propagation Failures (28%)

Failures that originate outside your system and spread into it. Your code is clean — the incident comes from a dependency you don't control, or from hidden coupling between internal services.

• FM-01 Cross-Org Cascade (4,516 incidents): An upstream provider fails and triggers cascading outages across downstream services. AWS us-east-1 Oct 2025 hit 137 downstream companies within 24 hours. CrowdStrike July 2024 affected banking, airlines, and healthcare globally. Azure Oct 2024 cascaded across European operators.
• FM-23 Hidden Internal Coupling (579 incidents): Internal services with undocumented or underestimated dependencies fail together when one component degrades — even when each appears healthy in isolation.

Read more: The SRE Cascade Tax — Why 1 in 5 Incidents Is Caused by a Provider You Don't Control →

Family 2: Tail / Outlier Failures (3%)

Low-frequency failures that are disproportionately hard to detect. Standard monitoring misses them because aggregate metrics stay healthy while a subset of requests experiences severe degradation.

• FM-06 Aggregate-Masked Tail Degradation: Average latency and error rates appear normal while P99 latency is spiking. Affected users experience timeouts that don't register in dashboards.
• FM-35 In-Flight Compatibility Break (481 incidents): Partial rollouts create version-boundary failures where requests processed by different service versions behave inconsistently, causing hard-to-reproduce errors.

Family 3: Change-Induced Failures (31% — the largest)

The most common failure category in the dataset. Something was deployed or modified — code, configuration, or infrastructure — and the system broke within minutes to hours. The change management trail is the first place to look.

• FM-09 Deploy-Induced Regression (3,764 incidents): A code deployment introduces a regression that degrades or breaks service. Anthropic's data shows 73% of their incidents map to specific model version regressions. MTTR is typically short once the deployment is identified — rollback resolves most cases.
• FM-10 Config-Induced Failure (1,763 incidents): A non-code change — configuration value, feature flag, IAM policy, DNS record, or infrastructure setting — triggers an incident. Cloudflare's Nov 2025 global outage traced to a ClickHouse permissions change that doubled a feature file size. Harder to detect than deploy regressions because config changes often live outside the standard deployment audit trail.

Read more: Deploy-Induced Regression — The Most Common SRE Incident Your Team Is Causing Itself →

Read more: SRE Config-Induced Failures — The Incident That Starts With "Nothing Changed" →

Family 4: Capacity and Resource Failures (13%)

Failures caused by a finite resource — CPU, memory, disk, connection pools, GPU compute, API quotas — being consumed faster than it can be replenished or scaled. The surface symptom varies, but the underlying pattern is consistent.

• FM-13 Resource Exhaustion (2,185 incidents): A bounded resource depletes completely: connection pool hits zero, memory leaks to ceiling, disk fills with logs, GPU capacity runs out under inference load.
• FM-25 Autoscaling Pathology (231 incidents): Autoscaling fails to respond correctly — scaling too slowly, overcorrecting into a cost spike, or entering a flap cycle. Common in services with spiky or unpredictable load profiles. Both FM-13 and FM-25 are Tier C: highly automatable.

Read more: SRE Resource Exhaustion — The Incident Pattern That Looks Different Every Time →

Family 5: External / Adversarial (1%)

Failures caused by deliberate external action — attacks, exploits, or compromised third-party components. Low in frequency but among the highest in MTTR and business impact when they occur.

• FM-15 External Attack: Service degradation or outage caused by a directed attack — DDoS, credential stuffing, or API abuse. Detection is often delayed because attack traffic mimics legitimate load spikes before the signature becomes clear.
• FM-16 Supply-Chain Breach: A compromised third-party library, SDK, or service credential creates a backdoor into your systems. The Salesloft/Drift OAuth breach Aug 2025 ran for 70+ days before detection — one of the longest active incidents in the 2025 dataset.

Family 6: AI-Specific Failure Modes (2%)

Failure modes that emerge specifically from AI/ML systems — model serving, training pipelines, and inference infrastructure. The 2% share understates real-world impact: training pipeline failures and model quality regressions rarely surface on public status pages.

• FM-17 AI Service Output Quality Degradation (250 incidents): Model outputs degrade in quality without a clear service error — latency looks normal, error rates stay flat, but responses become less accurate or misaligned with expected behavior. Affects LLMOps, MLOps, DataOps, and DevOps toolchains relying on AI-generated outputs.
• FM-33 GPU / Accelerator Fleet Heterogeneity (Provisional): Inference behavior diverges across GPU generations or driver versions in a mixed fleet, producing inconsistent outputs that are difficult to reproduce and trace.

Family 7: Recovery / Process Failures (11%)

Incidents where the technical fix is complete but the incident remains open because data consistency, queue draining, or operational dependencies haven't fully resolved. These are the incidents that won't close.

• FM-21 Phased Data Recovery (2,315 incidents): A multi-stage recovery process — backfills, queue draining, consistency checks — extends the incident well beyond the initial fix. Datadog June 2025: 2.1-day recovery window. OpenAI Compliance API delays July 2025: 5.4 days.

Read more: The SRE Incident That Won't Close — Understanding Phased Data Recovery →

Family 8: Foundational Integrity (12%)

Failures in the underlying systems that reliability work depends on — monitoring, data integrity, control planes, and state consistency. Often the hardest to detect because the failure may not manifest as a visible service outage.

• FM-26 Silent Data Corruption (859 incidents): The service appears up and healthy, but data is wrong — write failures, calculation errors, or storage corruption that bypass error detection. Often discovered by customers before engineering teams.
• FM-30 Control Plane Failure (836 incidents): The management layer that orchestrates infrastructure breaks down, affecting the ability to scale, deploy, or recover services. AWS Oct 2025 and GCP Jun 2025 both produced control plane incidents that amplified downstream impact significantly.
• FM-27 Monitoring Blind Spot (375 incidents): Alerting and observability gaps mean the incident is reported by customers before internal detection fires. The failure mode is real but invisible to the on-call team.
• FM-31 State Divergence (196 incidents): Distributed system components fall out of sync — caches, replicas, or stateful services hold inconsistent views of the same data, producing unpredictable behavior.

The 55/28/17 Autonomy Split

55% of incidents fall into AI-Closed modes. 28% need AI-Augmented human judgment. 17% remain Human-Led.

Key Takeaways

• 31% of incidents trace to change-induced patterns — highest automation value
• 28% are propagation failures — cross-org cascades. Detection speed is the lever.
• Foundational integrity (12%) is hardest — requires human judgment on data truth
• AI-specific modes under-counted at 2% due to disclosure gaps

Explore the data at stackgen.com/state-of-reliability-2026. Sign up for the LinkedIn webinar.