Alert evaluation engine
Your task
Design the alert evaluation engine that turns metric/log conditions into alert states for an org with 8,000 alert rules. Problems to solve: a metric oscillating around its threshold causes a rule to flap (fire/resolve every 30s), producing pager spam; planned maintenance and deploys should suppress alerts; and some alerts should only fire if a condition holds for several minutes, not on a single bad sample. Design the rule model, the state machine, flap damping, and suppression.
It covers
- Flapping
- Hysteresis
- Maintenance windows