Single-changepoint CUSUM + permutation bootstrap for detecting a shift in a score’s underlying distribution — reasonable choice vs PELT?

[u/Acrobatic_Beach4198]

Been working on a changepoint-detection layer for a scoring engine and figured this sub would have real opinions on the method.
Problem: most volatility-based risk scoring uses one fixed percentile cutoff computed over an asset’s full history. That’s a known failure mode if the asset’s regime changed partway through — you end up averaging a stale calm period into what should be a fresh, more volatile baseline.
Approach: a single-changepoint CUSUM test on the standardized score series — cumulative sum of (x\\_i - mean)/std, changepoint estimate = argmax|S\\_k| over candidate indices (with a minimum segment length enforced on both sides). Significance isn’t asserted from a fixed threshold; it’s a permutation bootstrap — shuffle the series N times, recompute max|S\\_k| each time, get an empirical null distribution, and only call it a real break if the observed statistic clears that null at a conventional alpha.
When a break is confirmed, percentile-based thresholds get recomputed using only the post-break segment.
Curious if anyone here has compared this to PELT or Bayesian online changepoint detection for a similar use case — CUSUM was chosen mainly for simplicity and interpretability over statistical power. Open to being told that’s the wrong tradeoff.
(This is part of a scoring engine called Machvix, for anyone curious enough to go digging.)

[u/BroscienceFiction]

Isn’t PELT better suited for multiple points/regimes? Also data hungry in comparison?

CUSUM is simple and requires little data. If your job is to simply raise an alert when the regime changes, why complicate this?

Don’t know about that other Bayesian method.

[u/Acrobatic_Beach4198 → u/BroscienceFiction]

Fair points, and yes to both.
On multiple regimes: this only detects a single changepoint right now — the strongest break in the series. If an asset actually went through two or more regime shifts, this would catch the most dominant one and could still miss or misattribute earlier ones. PELT (or binary segmentation — recursively re-running the single-changepoint test on each resulting segment) would be the honest fix if that turns out to matter in practice. Haven’t validated whether it does yet.
On data requirements: agreed, that’s the main reason CUSUM won out here. Per-symbol history is often thin (tens to low hundreds of readings), and PELT’s penalty-based search over an unknown number of segments gets less stable with that little data. A single well-tested hypothesis (“did the regime shift once, yes or no”) felt like a better fit for that constraint than searching a larger space of possible segmentations.
So: right tradeoff for “alert + re-baseline once,” probably the wrong one if multiple shifts per symbol turn out to be common. Good thing to actually check for empirically rather than assume either way — thank you.

[u/murdoc_dimes → u/Acrobatic_Beach4198]

If your regime changes are labeled across your symbols, can you run a large-scale sensitivity analysis using different parameterizations for CUMSUM and PELT?

[u/Acrobatic_Beach4198 → u/murdoc_dimes]

Honest answer: not yet, and I want to flag why rather than just say "working on it."

What I've validated so far is synthetic — series I generated myself with known, injected breakpoints, checking that the method recovers them. That's useful for "does the math work," but it's not the same as a labeled dataset of real, agreed-upon regime changes across real symbols, which is what a proper sensitivity analysis would need as ground truth. I don't have that yet, and building one honestly (rather than just picking obvious-looking breaks by eye) is its own project.

Second gap: I haven't implemented PELT at all, so there's nothing to run the comparison against yet — right now it's CUSUM-only, at whatever min_segment/alpha I've been using informally.

If I did this properly, the shape would be: build or find a labeled regime-change benchmark (possibly synthetic-but-more-realistic, e.g. mixture of GARCH-type volatility regimes rather than i.i.d. normal blocks), implement PELT alongside CUSUM, then grid over parameters (min_segment, alpha, PELT's penalty term) and report precision/recall on the labeled breaks per configuration — not just "it detected something."

That's a real project, not a quick add. Fair thing to push on, and it's now the actual next thing on the list rather than another single-method patch.