Anomaly Detector for Business Metrics

An anomaly detector is a service that scans business metrics daily (or more frequently) and raises a flag when a metric behaves unusually. The logic is simple: the model learns from historical data, establishes a normal range accounting for seasonality and trend, and flags points outside that range. The team learns about a revenue dip, a spike in churn rate, or an unusual conversion at the moment the signal appears — not two weeks later at a retrospective.

What a typical setup includes:

1. Connection to a data source — a data warehouse or a direct query to a BI tool.
2. Definition of the metric set: revenue by channel, MRR, active users, funnel stage conversion, churn, average order value, order size, inventory levels, runway.
3. Training baseline models on historical data for each metric — daily and weekly seasonality, holidays, and trend are taken into account.
4. Regular execution of checks (cron or event-driven) with calculation of the deviation from the expected value.
5. Publishing an alert in Slack or Teams with context: metric, current value, expected range, deviation magnitude, link to the dashboard.
6. Logging confirmed anomalies and false positives — for retraining thresholds.

What the detector will NOT do:

• It does not explain the cause of the anomaly. The signal says 'something is off here,' but root cause analysis remains with the human.
• It does not work without clean historical data. If the revenue data mart breaks down once a week or the metric recently changed its formula — the model will produce noise.
• It is not responsible for business decisions. An alert in Slack is an input trigger for investigation, not an instruction to stop a campaign or raise prices.

Architecturally, the service consists of four layers: data source, calculation engine, alert engine, delivery channel. The custom-code approach is chosen when off-the-shelf SaaS platforms for anomaly detection are excessively priced or do not fit well with the specifics of the client's metrics.

Technical flow

1. The scheduler (Airflow, Prefect, Dagster, or cron in kubernetes) runs a batch job on a schedule — once per hour or once per day, depending on the metric.
2. The job runs an SQL query against the data warehouse and retrieves the time series for the required metric with a history of 90-365 days.
3. The detection module applies one of the models: STL decomposition and z-score for most metrics, Prophet or ARIMA for series with pronounced seasonality, isolation forest for multivariate cases.
4. Calculation of the expected range for the current point. If the actual value falls outside the confidence interval boundaries, an anomaly is recorded with the direction and magnitude indicated.
5. Post-processing: duplicate filtering (one anomaly does not alert twice), aggregation of related signals, classification by severity.
6. Composing a message in Slack or Teams via webhook — metric, value, expectation, delta, time window, link to the BI dashboard for drill-down.

Implementation steps

1. Metric audit and prioritization: a list of 10-20 critical KPIs worth monitoring (more than that — you will drown in alerts).
2. Data preparation: quality checks, a unified metrics table in the DWH or materialized view, documenting the SLA for data freshness.
3. Stack selection: Python with libraries for time-series analysis, an orchestrator, secrets for connecting to the DWH and messenger.
4. Prototype on 2-3 metrics, manual threshold calibration, a run on historical data to verify accuracy.
5. Coverage expansion, adding severity levels, channel separation (P1 alerts go to the on-call channel, P2 — to the digest).
6. Two-week shadow mode: alerts are written to the log but not sent to Slack — false positive frequency is verified.
7. Launch to production, monthly review of thresholds and effectiveness.

System components

Infrastructure costs are low: calculation takes minutes, the load on the DWH is small. The main resource is the time of a data engineer or ML engineer for model calibration and working with metric owners.

What should be in place on the client side before the project starts:

Data and access:

• Data warehouse or a centralized analytics database with at least 6 months of metric history (a year is better).
• Documented SQL queries or dbt models for key metrics. If each metric is calculated ad hoc in different ways — we establish order first.
• A service account for reading data and a webhook URL in Slack or Teams for sending alerts.
• An understanding of seasonality: the team knows that revenue drops on Saturday and the average check grows in December — the model is trained with this in mind.

Team and owners:

• A metrics on-call person — someone who responds to an alert. Without an owner, the service turns into a noise channel.
• An analyst or data engineer who owns the metric logic and assists with calibration.
• A DevOps or platform engineer for deployment (Docker, secrets, access to DWH from the infrastructure).

Technology stack:

• Python 3.10+, Docker, an orchestrator (if not yet in place — we set up a simple Prefect or cron in the existing kubernetes cluster).
• Access to Slack or Microsoft Teams via incoming webhook.

Timeline:

• Prototype with 2-3 metrics: 2-3 weeks.
• Full set of 10-20 metrics with calibration and shadow mode: 4-6 weeks.
• If the data warehouse is not set up or the data is dirty — add 2-4 weeks for preparation.