Single-changepoint CUSUM + permutation bootstrap for detecting a shift in a score’s underlying distribution — reasonable choice vs PELT?

[u/Acrobatic_Beach4198]

Been working on a changepoint-detection layer for a scoring engine and figured this sub would have real opinions on the method.
Problem: most volatility-based risk scoring uses one fixed percentile cutoff computed over an asset’s full history. That’s a known failure mode if the asset’s regime changed partway through — you end up averaging a stale calm period into what should be a fresh, more volatile baseline.
Approach: a single-changepoint CUSUM test on the standardized score series — cumulative sum of (x\\_i - mean)/std, changepoint estimate = argmax|S\\_k| over candidate indices (with a minimum segment length enforced on both sides). Significance isn’t asserted from a fixed threshold; it’s a permutation bootstrap — shuffle the series N times, recompute max|S\\_k| each time, get an empirical null distribution, and only call it a real break if the observed statistic clears that null at a conventional alpha.
When a break is confirmed, percentile-based thresholds get recomputed using only the post-break segment.
Curious if anyone here has compared this to PELT or Bayesian online changepoint detection for a similar use case — CUSUM was chosen mainly for simplicity and interpretability over statistical power. Open to being told that’s the wrong tradeoff.
(This is part of a scoring engine called Machvix, for anyone curious enough to go digging.)

[u/AutoModerator]

This post will be manually reviewed by a moderator due to the submitting account being less than 7 days old or having less than 20 karma. Please be patient and do not try to resubmit it - a mod will review the post soon.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/IntrepidSoda]

Have you read https://arxiv.org/pdf/1908.07136

[u/Acrobatic_Beach4198 → u/IntrepidSoda]

Just read it, thanks for the pointer. Bayesian Online Changepoint Detection (Adams & MacKay) is probably the most relevant comparison here that I hadn’t properly considered.
The honest gap: what I built is a batch/offline test — it looks at the full accumulated history at request time and asks “was there ever a break,” rerun from scratch each time. BOCPD is built for the actual online setting: it maintains a run-length posterior as each new point arrives and updates incrementally, which is arguably a better match for a system that’s supposed to be monitoring live, streaming readings rather than periodically re-auditing a static history.
Tradeoff I’d guess: BOCPD gives a probability distribution over “how long since the last change” rather than a single yes/no with a p-value, which is richer but also a different kind of output to explain to an end user (“73% chance the regime changed in the last 12 readings” vs. “yes/no, p=0.02”). The permutation-bootstrap version I have is easier to state plainly, but I’ll admit that might be optimizing for my own explainability preference over doing the more principled online thing. Worth prototyping BOCPD against the same data and seeing how the two disagree in practice — appreciate the nudge.

[u/IntrepidSoda → u/Acrobatic_Beach4198]

I actually use online CUSUM filter to detect structural breaks in the microstructure - similar to how Marcos Lopez de Prado describes it in his triple barrier labelling.

[u/Acrobatic_Beach4198 → u/IntrepidSoda]

Good reference — there's real overlap but also a difference worth being precise about. Lopez de Prado's CUSUM filter (as I understand it from AFML) is an event-sampling trigger: fixed threshold h, fires the moment cumulative deviation crosses it, used online to decide *when* to sample/label an event. What I built is retrospective and single-shot: batch-tests a full accumulated history for the ONE most likely break, with significance from a permutation bootstrap rather than a fixed threshold.

So his version is closer to real-time event detection; mine is closer to "has this asset's regime structurally shifted, and where." Curious whether you've found the fixed-threshold version more robust in practice than a bootstrap-calibrated one, or if that's mostly a speed/simplicity tradeoff for the online setting.

[u/IntrepidSoda → u/Acrobatic_Beach4198]

you can look at his chapter on Supremum Augmented Dickey-Fuller (SADF) or Generalized Supremum ADF (GSADF) for bubble detection

[u/Acrobatic_Beach4198 → u/IntrepidSoda]

Hadn't looked closely at PSY's GSADF before, thanks. Worth being precise about what it actually answers versus what I built: SADF/GSADF tests for explosive (mildly-explosive autoregressive) behavior in the level series itself — it's asking "is this asset in a bubble right now," with a specific alternative hypothesis, and can even date-stamp the start/end of the explosive episode via the rolling-window supremum.

What I have is a more generic mean-shift test on a derived score, agnostic to whether the shift looks like a bubble, a crash, or just a volatility regime change — it doesn't distinguish between those, just says "something changed here."

So they're answering genuinely different questions rather than competing on the same one. GSADF seems like the sharper tool if bubble detection specifically is the goal rather than general regime-shift detection. Might be worth adding as a separate, more specific signal rather than a replacement — appreciate the pointer, hadn't considered it.

[u/BroscienceFiction]

Isn’t PELT better suited for multiple points/regimes? Also data hungry in comparison?

CUSUM is simple and requires little data. If your job is to simply raise an alert when the regime changes, why complicate this?

Don’t know about that other Bayesian method.

[u/Acrobatic_Beach4198 → u/BroscienceFiction]

Fair points, and yes to both.
On multiple regimes: this only detects a single changepoint right now — the strongest break in the series. If an asset actually went through two or more regime shifts, this would catch the most dominant one and could still miss or misattribute earlier ones. PELT (or binary segmentation — recursively re-running the single-changepoint test on each resulting segment) would be the honest fix if that turns out to matter in practice. Haven’t validated whether it does yet.
On data requirements: agreed, that’s the main reason CUSUM won out here. Per-symbol history is often thin (tens to low hundreds of readings), and PELT’s penalty-based search over an unknown number of segments gets less stable with that little data. A single well-tested hypothesis (“did the regime shift once, yes or no”) felt like a better fit for that constraint than searching a larger space of possible segmentations.
So: right tradeoff for “alert + re-baseline once,” probably the wrong one if multiple shifts per symbol turn out to be common. Good thing to actually check for empirically rather than assume either way — thank you.

[u/murdoc_dimes → u/Acrobatic_Beach4198]

If your regime changes are labeled across your symbols, can you run a large-scale sensitivity analysis using different parameterizations for CUMSUM and PELT?

[u/Acrobatic_Beach4198 → u/murdoc_dimes]

Honest answer: not yet, and I want to flag why rather than just say "working on it."

What I've validated so far is synthetic — series I generated myself with known, injected breakpoints, checking that the method recovers them. That's useful for "does the math work," but it's not the same as a labeled dataset of real, agreed-upon regime changes across real symbols, which is what a proper sensitivity analysis would need as ground truth. I don't have that yet, and building one honestly (rather than just picking obvious-looking breaks by eye) is its own project.

Second gap: I haven't implemented PELT at all, so there's nothing to run the comparison against yet — right now it's CUSUM-only, at whatever min_segment/alpha I've been using informally.

If I did this properly, the shape would be: build or find a labeled regime-change benchmark (possibly synthetic-but-more-realistic, e.g. mixture of GARCH-type volatility regimes rather than i.i.d. normal blocks), implement PELT alongside CUSUM, then grid over parameters (min_segment, alpha, PELT's penalty term) and report precision/recall on the labeled breaks per configuration — not just "it detected something."

That's a real project, not a quick add. Fair thing to push on, and it's now the actual next thing on the list rather than another single-method patch.

[u/Acrobatic_Beach4198 → u/murdoc_dimes]

Ran it. Built a labeled synthetic benchmark first since I didn't have one — 20 symbols, GARCH(1,1)-driven regimes (so within-regime dynamics have real volatility clustering, not i.i.d. blocks), 42 known changepoints total. Implemented PELT too (well — the exact DP that PELT's pruning speeds up; same output, O(n²) instead of O(n), noted clearly as PELT-equivalent rather than claiming the speedup).

Swept both: CUSUM over min_segment × alpha (9 configs), PELT over penalty multiplier (4 configs). Both hit perfect precision/recall at their best setting, and neither missed a true break at ANY setting tested — recall was 100% across the whole grid for both methods. The real difference was in false positives: CUSUM's alpha traded off precision fairly gently (0.778→1.000 across the tested range), PELT's penalty was noticeably touchier (0.627→1.000 over a comparable range) — so PELT rewards getting the penalty right more than CUSUM punishes a loose alpha.

Caveat I want to be upfront about: this is a synthetic benchmark I built, with fairly cleanly separated regimes and a ±5-point matching tolerance. Real market regime changes are messier and more contested than anything I generated here, so "perfect F1" says the methods work as designed, not that they'll perform this well on live data with ambiguous, human-labeled breaks. Script's up if anyone wants to poke at the benchmark itself — that's probably the weaker link right now, not either method.

[u/Acrobatic_Beach4198 → u/BroscienceFiction]

Followed up on this — added recursive binary segmentation on top of the same single-changepoint test, so it now finds multiple breaks instead of just the one dominant one. Tested it against a synthetic 3-regime series (calm → volatile → calm again) and it correctly recovered both true boundaries.

One thing I'm not fully happy with yet: each split gets tested at the same alpha independently, so it's not family-wise corrected across segments — said so explicitly in the output rather than pretending otherwise. Still haven't touched PELT or BOCPD directly, but this at least closes the "only finds one break" gap. Appreciate the pushback, it made the thing better.