SOC Analyst Alert Fatigue — Metric Deep-Dive

SecureNet's SOC dashboard shows "mean time to detect (MTTD) improved 40% this quarter." But the SIEM alert volume dropped 60% and the SOC manager suspects the metric is being gamed — manually, before presenting to the CISO.

# Aplly.xyz Case Study Submission

## Title
SOC Analyst Alert Fatigue — Metric Deep-Dive

## Type
Data Analytics

## Difficulty
Advanced

## Estimated Time
75 minutes

## Overview
SecureNet's SOC dashboard shows "mean time to detect (MTTD) improved 40% this quarter." But the SIEM alert volume dropped 60% and the SOC manager suspects the metric is being gamed — manually, before presenting to the CISO.

## Case Details

Function Focus: Metric validity reasoning, Goodhart's Law, leading vs lagging indicators

Scenario:
The CISO wants to publicly recognize the SOC team for "improved detection speed." The SOC manager suspects the MTTD improvement is an artifact of alert suppression — the SIEM tuning team raised thresholds to reduce noise, which means only the loudest, most obvious alerts remain. True detection capability may have degraded, not improved. The manager has weekly data across two quarters before the tuning change.

Dataset Structure:
- Week, Total Alerts Generated, Alerts Tuned/Suppressed, Alerts Investigated, True Positives, Mean Time to Detect (hours), Escalations to Tier 2, Missed Incidents (post-hoc)

Tasks:
1. By hand, compute the true positive rate and the investigation coverage rate (investigated / generated) week by week — do not use spreadsheet formulas for the first pass
2. Identify the inflection point where the tuning change hides a degradation in detection
3. Cross-check MTTD against the other 6 metrics to find the specific inconsistency: which metric moves opposite to what the "MTTD improved" headline claims
4. Write the explanation you would give the CISO, correcting the naive "MTTD improved = SOC getting better" narrative
5. Propose one alternative metric or metric pair that would have surfaced this problem earlier

Expected Output:
A hand-computed cross-metric table, identified inflection point with reasoning, written CISO narrative, and one alternative metric proposal.

Evaluation Criteria:
Correct computation of true positive rate and investigation coverage trends, correct identification of the metric tradeoff (MTTD improved because only easy alerts remain), quality and actionability of the alternative metric proposal.

## Data Sources

| Week | Alerts Generated | Alerts Suppressed | Alerts Investigated | True Positives | MTTD (hrs) | Escalations | Missed Incidents |
|---|---|---|---|---|---|---|---|
| Q1-W1 | 4,200 | 0 | 4,010 | 380 | 8.2 | 95 | 2 |
| Q1-W2 | 4,350 | 0 | 4,120 | 395 | 8.5 | 102 | 1 |
| Q1-W3 | 4,180 | 0 | 4,050 | 375 | 8.0 | 98 | 3 |
| Q1-W4 | 4,300 | 0 | 4,080 | 390 | 8.3 | 100 | 2 |
| Q1-W5 | 4,250 | 0 | 4,100 | 385 | 8.1 | 97 | 2 |
| Q1-W6 | 4,400 | 0 | 4,150 | 400 | 8.4 | 105 | 1 |
| Q2-W1 | 4,100 | 500 | 3,550 | 360 | 7.5 | 90 | 4 |
| Q2-W2 | 3,800 | 800 | 2,980 | 320 | 6.8 | 78 | 6 |
| Q2-W3 | 3,200 | 1,200 | 2,010 | 260 | 5.9 | 62 | 8 |
| Q2-W4 | 2,800 | 1,500 | 1,320 | 210 | 5.2 | 48 | 11 |
| Q2-W5 | 2,400 | 1,800 | 620 | 165 | 4.5 | 35 | 14 |
| Q2-W6 | 2,100 | 2,000 | 350 | 130 | 4.1 | 28 | 17 |

(Diagnostic pattern: MTTD drops from ~8.2 to ~4.1 hours, but missed incidents rise from 2 to 17 per week. The ratio of true positives to alerts investigated stays ~9.5%, but the investigation coverage rate collapsed from ~95% to ~17% as alert suppression concealed the real threat volume. MTTD improved because only trivial alerts survived the threshold tuning.)

## Solution Frameworks
Metric cross-validation, Goodhart's Law, coverage rate analysis, leading vs lagging indicators

## Solver Guidance & Tutorials
Link to: "SOC Metrics That Actually Measure Security" tutorial

## What You'll Learn
- Critical evaluation of security operations metrics
- Detecting metric gaming through cross-metric inconsistency
- Communicating a nuanced finding upward without triggering defensiveness

## Tags
SOC, metrics, alert fatigue, security operations, analytical reasoning

## Registration Links
- Register as Solver
- Register as Evaluator

| Week | Alerts Generated | Alerts Suppressed | Alerts Investigated | True Positives | MTTD (hrs) | Escalations | Missed Incidents |
|---|---|---|---|---|---|---|---|
| Q1-W1 | 4,200 | 0 | 4,010 | 380 | 8.2 | 95 | 2 |
| Q1-W2 | 4,350 | 0 | 4,120 | 395 | 8.5 | 102 | 1 |
| Q1-W3 | 4,180 | 0 | 4,050 | 375 | 8.0 | 98 | 3 |
| Q1-W4 | 4,300 | 0 | 4,080 | 390 | 8.3 | 100 | 2 |
| Q1-W5 | 4,250 | 0 | 4,100 | 385 | 8.1 | 97 | 2 |
| Q1-W6 | 4,400 | 0 | 4,150 | 400 | 8.4 | 105 | 1 |
| Q2-W1 | 4,100 | 500 | 3,550 | 360 | 7.5 | 90 | 4 |
| Q2-W2 | 3,800 | 800 | 2,980 | 320 | 6.8 | 78 | 6 |
| Q2-W3 | 3,200 | 1,200 | 2,010 | 260 | 5.9 | 62 | 8 |
| Q2-W4 | 2,800 | 1,500 | 1,320 | 210 | 5.2 | 48 | 11 |
| Q2-W5 | 2,400 | 1,800 | 620 | 165 | 4.5 | 35 | 14 |
| Q2-W6 | 2,100 | 2,000 | 350 | 130 | 4.1 | 28 | 17 |

(Diagnostic pattern: MTTD drops from ~8.2 to ~4.1 hours, but missed incidents rise from 2 to 17 per week. The ratio of true positives to alerts investigated stays ~9.5%, but the investigation coverage rate collapsed from ~95% to ~17% as alert suppression concealed the real threat volume. MTTD improved because only trivial alerts survived the threshold tuning.)

Metric cross-validation, Goodhart's Law, coverage rate analysis, leading vs lagging indicators