Problem
Streaming log analytics that establishes a rolling baseline, computes z-scores on event rates, and surfaces statistical anomalies in real time before they become incidents.
Anomaly DetectionStreamingObservabilityStatisticsPythonCase study / interactive demo
Case study
From problem to working system
My role
Full-stack software engineer
Solution
Streaming log analytics that establishes a rolling baseline, computes z-scores on event rates, and surfaces statistical anomalies in real time before they become incidents.
Architecture
The implementation combines the following technologies and system concerns.
Anomaly DetectionStreamingObservabilityStatisticsPython
How it was built
- Established a rolling baseline over streaming log events so normal is defined by recent behaviour rather than a static threshold.
- Computed z-scores on event rates per source and per event type, surfacing statistical outliers instead of keyword matches.
- Handled the cold-start problem by withholding verdicts until the baseline window is populated, rather than alerting on the first spike.
- Separated volume anomalies from novelty anomalies, since a brand-new event type at low volume matters as much as a flood of a known one.
Security decisions
No project-specific security control is documented in the current project record. The case study avoids claiming controls that were not verified.
Major challenges
- Established a rolling baseline over streaming log events so normal is defined by recent behaviour rather than a static threshold.
- Computed z-scores on event rates per source and per event type, surfacing statistical outliers instead of keyword matches.
- Handled the cold-start problem by withholding verdicts until the baseline window is populated, rather than alerting on the first spike.
Verified evidence
Results and measurable impact
- Handled the cold-start problem by withholding verdicts until the baseline window is populated, rather than alerting on the first spike.
- Separated volume anomalies from novelty anomalies, since a brand-new event type at low volume matters as much as a flood of a known one.
- Kept the alert rate controlled with hysteresis so a metric hovering at the threshold does not oscillate between states.
- Presented each anomaly with the baseline, the observed value, and the deviation, so an analyst can judge it without re-querying the data.
No separate numeric outcome is documented, so this section shows shipped technical evidence without inventing metrics.
Screenshots and access
Product view
Interactive Demo
A scoped, fully functional recreation of this project's core feature runs below, live in your browser. Reset it, resize it, or expand it to full screen.
loglens.zainkhalilkhan.com
LogLens
AI product
Local inference mock
LogLensWorkspace6 updates
LogLens · Anomaly Detection
Each service streams events per second. LogLens keeps an independent rolling baseline per stream and flags any reading whose z-score exceeds your threshold in either direction as a statistical anomaly.
events/s
0
baseline μ
320
z-score
+0.0
anomalies
0
Inject into API Gateway
Incident feed0
No anomalies detected. Baselines are nominal across all streams.
Zain Khalil Khan
Client-side sandbox. State is in memory and nothing is sent to a server.