Less alert noise, lower cloud cost
Fleet alerting runs on 23646 monitors.
I led the monitor cleanup and the cluster resize.
Datadog watches our Kubernetes clusters. Its monitors, the rules that raise an alert, page the on-call engineer.
Why it was needed
Each cluster had its own copy of every monitor, the alerts were noisy, and some monitors could never fire. One cluster also reserved far more CPU than it used.
How it was done
Rebuilt 236 monitors as 46 fleet monitors, one rule per check grouped by cluster, with a heartbeat per cluster so a silent cluster still alerts.
Moved paging to JSM Ops, the on-call tool.
Lowered the CPU requests (the CPU each pod reserves) to measured use, then removed a node.
Deleted 164 unused EBS disks.
Problems on the way
The out-of-memory monitor missed 17 kills in 14 days.
It read events Datadog never receives, then a sampled value that misses short kills. Fix: Turned on Datadog's kernel-level (eBPF) kill check on every cluster and proved it with a test pod.
Removing a node as-is would have pushed reserved CPU to about 90%.
Pods reserved far more CPU than they used. Fix: Lowered the requests first, so every pod fit before the node went.
A monitor that cannot fire is worse than none.
Results
- 236 → 46monitors
- −53%reserved CPU on one production cluster
- 164unused EBS disks deleted