Calibration, not detection power

Most drift detectors are judged on one number: how often they catch a real anomaly. That number matters, but it hides the more important question — when you ask a detector for a 5% false-alarm rate, does it actually give you 5%, or does it drift as soon as the data stops looking perfectly Gaussian? A detector that silently over-alarms trains your team to ignore it. One that silently under-alarms is worse.

Our own first implementation failed this test. It used a completely standard, textbook construction for a drift score — the kind most tools still ship. Asked for a 5% false-alarm rate on clean, unseen data, here is what it actually delivered:

On real Gaussian data, the naive path looks fine. On heavy-tailed or skewed data — which most real production features actually are — it flags up to 6× too often. Every number in that "shipped" column is Kirelta’s calibration doing its actual job: holding the promised rate whatever shape the data has. That’s the whole reason the shipped design exists, not a nice-to-have layered on top.

How that compares to standard alternatives

Same protocol, same four data shapes, against real scikit-learn detectors — no per-method tuning, everyone asked for the same 5% rate:

Read that IsolationForest row carefully: it’s reasonably well-calibrated (0.0019 error) and still misses almost 40% of large anomalies — calibration and detection power are genuinely separate properties, and a method can have one without the other. OneClassSVM has the opposite problem: strong detection, but it over-alarms on clean data by up to 8 points. Kirelta and EllipticEnvelope are essentially tied for the best combination of both, here.

Calibration also has to survive the data changing shape

A fixed calibration is only correct for the regime it was set on. Real production data doesn’t stay in one regime — traffic goes quiet, then a feature’s variance spikes, then it settles into something seasonal. Kirelta adapts its calibration as the data moves. Measured across seven synthetic regimes, with that adaptation turned off and then on:

Target is 0.050 in every row. Without adaptation, a sudden spike in variance alone pushes the realised rate to 0.1177 — more than double the target, exactly the kind of drift that should trigger a review, silently retraining the alarm to ignore itself instead. With adaptation on, every regime lands within a few thousandths of target. It is on by default; turning it off is what it took to produce the left column.

What this benchmark doesn’t show: one feature count (p=4), one sample size, one anomaly type (a uniform 6σ shift), default hyperparameters for every competitor. It’s real evidence about calibration under a fixed protocol — not a general ranking of detectors, and we’re not claiming it is. You can check calibration on your own data: fit a baseline on part of a clean period, assess another clean part, and about 5% of rows or fewer should be flagged.

The alarm’s false-alarm rate, measured

On a baseline of 2,400 rows (3 features, standard normal), we assessed 300 independent clean batches of 80 rows: the drift alarm fired on 2 of them — 0.7%, against a bound of 5%. On 60 batches with every feature moved by one standard deviation, it fired on 60. You can run the clean half of this check on your own data with Try it.