SLA Breach Postmortem — Root Cause with Incomplete Evidence

NexGenIT's Q3 uptime was 98.7% — below the 99.9% SLA commitment to their largest client. The NOC team logged 4 incidents, but the timeline has gaps, deployment records are partial, and two team members gave conflicting accounts. The IT manager must reconstruct the incident chain, separate confirmed facts from inference, and determine the true root cause before the client penalty meeting.

# Aplly.xyz Case Study Submission

## Title
SLA Breach Postmortem — Root Cause with Incomplete Evidence

## Type
Technology/IT

## Difficulty
Advanced

## Estimated Time
75 minutes

## Overview
NexGenIT's Q3 uptime was 98.7% — below the 99.9% SLA commitment to their largest client. The NOC team logged 4 incidents, but the timeline has gaps, deployment records are partial, and two team members gave conflicting accounts. The IT manager must reconstruct the incident chain, separate confirmed facts from inference, and determine the true root cause before the client penalty meeting.

## Case Details

Function Focus: Root-cause analysis with incomplete evidence, timeline reconstruction, fact vs inference separation, incident chain-of-causation reasoning

Scenario:
NexGenIT's flagship client, FinBank, has a contractual SLA of 99.9% uptime (max 43 minutes of downtime per 30-day month). Q3 actual uptime was 98.7% — approximately 5 hours and 30 minutes of cumulative downtime across the quarter. The penalty is ₹12 lakh per 0.1% below threshold. The NOC manager must produce a root-cause analysis report for the client meeting in 48 hours. He has the incident log, deployment records, monitoring screenshots, and two engineer statements — but the records are incomplete (the monitoring tool retains only 30 days of detail, and one engineer was on leave during part of the period). He must separate what is definitely known from what must be inferred, and give the client a defensible root cause.

Dataset Structure:
- Incident log: 4 entries with date, duration, system affected, reported cause, and evidence level (Confirmed / Partial / Speculative)
- Deployment timeline for Q3
- Monitoring excerpts (partial — only 30 days retained)
- Engineer statements (2 engineers, partially overlapping coverage)
- SLA calculation reference

Tasks:
1. Reconstruct the full incident timeline in chronological order, labeling each incident's evidence strength as Confirmed, Partial, or Speculative based on the available data
2. For each incident, trace the likely chain of causation — identify which facts are directly supported by evidence and which are assumptions you must make to connect the dots
3. Identify any incident that could share a common root cause with another — build a causal tree showing relationships
4. State the single most likely root cause of the SLA breach, and explicitly list what additional evidence would be needed to confirm it beyond doubt
5. Only after completing your manual analysis, compare your causal tree with what an AI or automated RCA tool would produce from the same inputs — note where the tool would overstate confidence in unconfirmed links

Expected Output:
A root-cause analysis packet containing: (a) a reconstructed timeline table with evidence-strength labels, (b) a fact-vs-inference log listing each claim and its evidence basis, (c) a causal tree diagram (text-based or described), (d) a stated root cause with a confidence level and a specific list of missing evidence needed to reach full confidence, and (e) a note on where an AI/tool-based RCA might introduce false certainty.

Evaluation Criteria:
Honest labeling of evidence strength (not treating partial evidence as confirmed), logical causal chain construction, clear separation of fact from inference in every claim, identification of shared root cause patterns across seemingly separate incidents, and specificity in listing missing evidence.

## Data Sources

Incident Log (Q3 — Jul–Sep):
| ID | Date | Duration | System | Reported Cause | Evidence | Evidence Level |
|---|---|---|---|---|---|---|
| INC-01 | Jul 12 | 1h 10m | Payment Gateway | Database connection pool exhausted | Full logs from DB team show 5,000 concurrent connections against a 2,000 pool limit. No code change preceded this. Traffic spike from a flash sale. | Confirmed |
| INC-02 | Aug 5 | 2h 05m | Core Banking API | "Unknown — API returned 502 for 125 min" | NOC ticket: "API server unresponsive. Restarted service. No root cause identified." Monitoring graphs for Aug 5 are missing (retention window expired). Deployment records show a new auth middleware was pushed Aug 3 at 22:00. | Partial |
| INC-03 | Aug 28 | 45m | Customer Portal | "Database replication lag" | DB team: "Replica fell behind by 37 min. Primary CPU at 98% during the window." No deploy on Aug 27–28. A bulk data migration ran Aug 28 02:00–04:00 (scheduled). Portal incident began at 09:15. | Partial |
| INC-04 | Sep 15 | 1h 30m | Payment Gateway | "SSL certificate expired" | Cert expiry log confirms gateway SSL cert expired at 00:00 Sep 15. Monitoring shows gateway errors from 00:03. Cert was renewed by 01:33. On-call engineer's statement: "I got paged at 00:05, took 90 min because I had to find the cert file — previous engineer didn't document the location." | Confirmed |

Deployment Timeline (Q3):
| Date | Change | System | Rollback? |
|---|---|---|---|
| Jul 5 | v3.1.2 — Bug fix for payment retry logic | Payment Gateway | No |
| Jul 28 | v3.2.0 — New rate limiting middleware | Core Banking API | No |
| Aug 3 | v3.2.1 — Auth middleware update | Core Banking API | Not rolled back |
| Aug 10 | v2.0.5 — UI theme update | Customer Portal | No |
| Aug 20 | v4.1.0 — Database connection pooling increase | Payment Gateway | No |
| Sep 1 | v3.3.0 — Logging overhaul | Core Banking API | No |
| Sep 10 | v2.1.0 — Portal search feature | Customer Portal | No |

Engineer Statements:

Statement A — Rohan (NOC, on-call Aug 3–10):
"I remember the Aug 5 incident. The API went down around 10:30 AM. I checked the deployment logs — the auth middleware was pushed Aug 3 night. I suspected a memory leak because the API was consuming 6 GB by the time I restarted it. But I couldn't prove it because the monitoring tool only keeps 30 days of metrics. I filed the ticket as 'unknown cause' because I didn't have hard proof. I think the auth middleware caused it, but I can't be sure."

Statement B — Priya (DB team, on-call Aug 25–31):
"The Aug 28 replication lag — we ran a bulk migration at 2 AM that inserts 12 million rows into the primary. The migration finished at 4 AM. The portal slowdown at 9 AM was probably unrelated — the replica should have caught up by then. But someone might have left a long-running query open from the migration. I didn't check the active query list at 9 AM. The portal team said they didn't change anything. I'm not sure what caused it."

SLA Reference:
- Contractual uptime: 99.9% per month (max 43 min downtime per 30-day month)
- Q3 actual: 98.7% (total downtime ~5h 30m across 92 days)
- Penalty: ₹12 lakh per 0.1% below 99.9%
- Client meeting: 48 hours from now

## Solution Frameworks
Root-cause analysis, evidence-strength classification, fault tree analysis, fact-vs-inference separation, postmortem best practices

## Solver Guidance & Tutorials
Link to: "Postmortems with Gaps — Root-Cause Analysis Under Uncertainty" tutorial

## What You'll Learn
- Reconstructing incident timelines from incomplete records
- Separating confirmed facts from assumptions in postmortem analysis
- Building causal trees that honestly reflect evidence strength
- Communicating uncertainty to stakeholders without losing credibility

## Tags
SLA breach, root cause analysis, incident postmortem, IT operations, evidence-based reasoning

## Registration Links
- Register as Solver
- Register as Evaluator

Incident Log (Q3 — Jul–Sep):
| ID | Date | Duration | System | Reported Cause | Evidence | Evidence Level |
|---|---|---|---|---|---|---|
| INC-01 | Jul 12 | 1h 10m | Payment Gateway | Database connection pool exhausted | Full logs from DB team show 5,000 concurrent connections against a 2,000 pool limit. No code change preceded this. Traffic spike from a flash sale. | Confirmed |
| INC-02 | Aug 5 | 2h 05m | Core Banking API | "Unknown — API returned 502 for 125 min" | NOC ticket: "API server unresponsive. Restarted service. No root cause identified." Monitoring graphs for Aug 5 are missing (retention window expired). Deployment records show a new auth middleware was pushed Aug 3 at 22:00. | Partial |
| INC-03 | Aug 28 | 45m | Customer Portal | "Database replication lag" | DB team: "Replica fell behind by 37 min. Primary CPU at 98% during the window." No deploy on Aug 27–28. A bulk data migration ran Aug 28 02:00–04:00 (scheduled). Portal incident began at 09:15. | Partial |
| INC-04 | Sep 15 | 1h 30m | Payment Gateway | "SSL certificate expired" | Cert expiry log confirms gateway SSL cert expired at 00:00 Sep 15. Monitoring shows gateway errors from 00:03. Cert was renewed by 01:33. On-call engineer's statement: "I got paged at 00:05, took 90 min because I had to find the cert file — previous engineer didn't document the location." | Confirmed |

Deployment Timeline (Q3):
| Date | Change | System | Rollback? |
|---|---|---|---|
| Jul 5 | v3.1.2 — Bug fix for payment retry logic | Payment Gateway | No |
| Jul 28 | v3.2.0 — New rate limiting middleware | Core Banking API | No |
| Aug 3 | v3.2.1 — Auth middleware update | Core Banking API | Not rolled back |
| Aug 10 | v2.0.5 — UI theme update | Customer Portal | No |
| Aug 20 | v4.1.0 — Database connection pooling increase | Payment Gateway | No |
| Sep 1 | v3.3.0 — Logging overhaul | Core Banking API | No |
| Sep 10 | v2.1.0 — Portal search feature | Customer Portal | No |

Engineer Statements:

Statement A — Rohan (NOC, on-call Aug 3–10):
"I remember the Aug 5 incident. The API went down around 10:30 AM. I checked the deployment logs — the auth middleware was pushed Aug 3 night. I suspected a memory leak because the API was consuming 6 GB by the time I restarted it. But I couldn't prove it because the monitoring tool only keeps 30 days of metrics. I filed the ticket as 'unknown cause' because I didn't have hard proof. I think the auth middleware caused it, but I can't be sure."

Statement B — Priya (DB team, on-call Aug 25–31):
"The Aug 28 replication lag — we ran a bulk migration at 2 AM that inserts 12 million rows into the primary. The migration finished at 4 AM. The portal slowdown at 9 AM was probably unrelated — the replica should have caught up by then. But someone might have left a long-running query open from the migration. I didn't check the active query list at 9 AM. The portal team said they didn't change anything. I'm not sure what caused it."

SLA Reference:
- Contractual uptime: 99.9% per month (max 43 min downtime per 30-day month)
- Q3 actual: 98.7% (total downtime ~5h 30m across 92 days)
- Penalty: ₹12 lakh per 0.1% below 99.9%
- Client meeting: 48 hours from now

Root-cause analysis, evidence-strength classification, fault tree analysis, fact-vs-inference separation, postmortem best practices