Security event correlation pipeline

DesignHard50 minPalantir and Microsoft style

Your task

Design a SIEM / security-log anomaly pipeline that ingests 4M security events/s from endpoints, network sensors, cloud audit logs, and applications, correlates them to detect intrusions (lateral movement, privilege escalation, data exfil), and alerts within a target of under 60s for high-severity patterns. It must retain raw logs for 1 year for forensics and compliance, support analyst ad-hoc hunting queries over that history, and resist an attacker who tries to disable or tamper with logging to cover their tracks. Cover ingestion/normalization, the streaming detection layer, storage tiers, and tamper-evidence.

It covers

  • Siem
  • Anomaly detection
  • Log pipeline
  • Threat detection