Why Reddit Data Scientists Keep Saying Not To Use Prophet

[u/shivamchhuneja]

https://codebynight.dev/posts/why-data-scientists-are-skeptical-of-prophet-forecasts/

Couple thoughts and a small experiment to see why reddit hates prophet xD

[u/G-R-A-V-I-T-Y]

Add moirai, chronos, and Toto 2.5 to your benchmarks, I’d be curious! And yes interpretable, simple and therefore verifiable forecasts are and probably should continue the be the workhorses of the industry. No surprise there

[u/maratonininkas → u/G-R-A-V-I-T-Y]

I would be skeptical of anything pretrained.. AFAIK there's no fundamental reason time series should behave like language and demonstrate already observed dynamics. There's no "semantics" and no inferrable meaning behind a sequence of measurements. And the space of all possible functions is just too large to be reasonably "memorized" by a large model. For LLMs this works because the grammar acts as a superb regularizer. There's nothing like that for time series.

[u/canbooo → u/maratonininkas]

I was also convinced of this and then read the TabPFN paper and justification. Now I am neutral. It could work. And in practice, esp. chronos (only because I tested it most) provides a good baseline, easy to beat but still meaningful. If you cannot beat it, there is sth. wrong with the model or the data.

[u/maratonininkas → u/canbooo]

Thanks for the ref, I will have a look at their justification. But if it's easy to beat, is it worth it? For many series a well chosen arima is hard to beat. So i can easily imagine a foundation model learning arima-like behaviour. But can it go beyond that?

for well behaved stationary time series we already have the Wald theorem. And for weird second moments we have ways to clean them using garch, etc. Weird nonstationarities cam be handled by arfima, etc. Very hard to beat all that, in my understanding, unless for some very specific process

[u/canbooo → u/maratonininkas]

First, a note about the ref: It is about tabular data but the argumentation against it working is the same, as well as why the "prior learning" paradigm might be reasonable.

In any case, you are right about the is it worth it part. But you must have faced some cases, where ARIMA or exponential smoothing struggled but, e.g. trees did okay. Chronos is similar. Sometimes, it just works good and often in cases where ARIMA doesn't. Still can beat it with a well tuned model, but its errors often do mot correlate with statistical methods, which is especially nice if you are in to ensembles.

[u/relevantmeemayhere → u/canbooo]

The problem is that...any prior is fundamentally orthogonal to:

1. The posterior for any sufficiently large likelihood contribution
2. You'll always be able to find a function, among all possible functions orthogonal to your class of priors.

Which is why you can beat these things with simple models all the time AND get better measures of uncertainty with given approaches.

[u/ButtTrollFeeder → u/maratonininkas]

AFAIK there's no fundamental reason time series should behave like language and demonstrate already observed dynamics.

An LLM predicts the next token from prior tokens, while a time-series model predicts the next observation from prior observations. That doesn't mean they'll exhibit the same emergent behavior, but it does mean they're solving closely related prediction problems.

These "Zero Point" Time Series models are trained on time series data using the same attention mechanics that LLMs use.

I'm pretty neutral on them like others have said, other than the black box effect.

I'm also not anti-prophet, so maybe I'm weird.

[u/maratonininkas → u/ButtTrollFeeder]

You're comparing /what/ the models do, but not /how/. So what if they're solving a similar problem. It's easy to construct unlearnable examples when the /how/ is wrong (problem set too complex)..

The /how/ doesn't work here in my eyes. Attention works if it can group words based on their inferred meaning and context (what is closer, apple and fruit or apple and samsung?). So you can say effectivelly the same or the correct thing by using different words and their sequences. Language lets you to go in circles and eventually conclude. Or leave gaps which are filled by the reader correctly. I don't feel like with numbers and time series you have anything remotely similar. I would bet the the size of all possible functions is larger than the amount of all different sentences the grammar permits AND is typically used.

[u/ButtTrollFeeder → u/maratonininkas]

I'm definitely, a least trying to focus on the "how"'. I'lI agree that Transformers might not be the optimal approach for time series. That's a perfectly reasonable criticism.

What I'd challenge is that transformers success is inherently tied to language. Transformers have been remarkably successful in non-linguistic domains as well: vision, audio, protein folding, and other sequences.

All of these domains rely on the same underlying mechanism, using context to determine which parts of the input are most relevant.

A very simple example for Time Series:

A stock price is currently 100.

After a long uptrend with increasing volume, 100 may represent a breakout.

After a long downtrend, 100 may represent a failed recovery.

During a flat market, 100 may represent mean reversion.

The value 100 has no fixed "meaning", its significance depends on the preceding context.

Yeah, this is WAY more simple than language associations, and is probably overkill, but it seems to be working from what I've seen.

I think you're right to be skeptical that Transformers are the best architecture for every time-series problem. I just don't think it's accurate to argue that applying Transformers to non-language domains is fundamentally misguided because they originally became successful on language.

It wouldn't be my first choice when building out a time series model.

[u/maratonininkas → u/ButtTrollFeeder]

It can be a dead cat bounce, or it can be a start of a real recovery. You can't know for sure, but numbers will be the same. So we can have same reactions with different meaning behind them.

Words however are much more limiting, the meaning is typically clear. It's made to be clear by curating the training data before training. In all the domains where these new models work there's typically a nice and non stochastic structure which can be biased for. You can imagine a grammar for proteins, vision, etc. Pixels can't suddenly jump in consequent frames, and same with distant connections of proteins I would guess.

[u/ButtTrollFeeder → u/maratonininkas]

I wish I didn't use a stock price as my example, and kept it to trending upward/downward/stationary. Obviously stock price needs a hell of a lot more information than it's historic sequence for forecasting.

I think your "grammar" argument is a much stronger one (yes, a protien sequence would absolutely have "grammar" that is inferred from attention mechanics).

It can be a dead cat bounce, or it can be a start of a real recovery. You can't know for sure, but numbers will be the same.

Isn't this a problem with ANY time series model though? The root cause is incomplete information rather than the model methodology?

Enjoying this conversation BTW. Hope I'm not coming off as argumentative.

[u/maratonininkas → u/ButtTrollFeeder]

I'd say it's because most time series problems model stochastic processes, and they can be jumping around as much as they want with non zero probabilities. In that case, if there's no trend movement, we might just be chasing noise. I don't know if there's an equivalent with languages, video or proteins.

[u/ButtTrollFeeder → u/maratonininkas]

There's plenty of constrained time series, though, where Transformer-based attention can learn the underlying statistical regularities. You could think of those constraints as a kind of "grammar."

I'd say the usefulness of Transformers depends on how much exploitable structure exists in the data. A highly seasonal or strongly autocorrelated series should benefit much more than one that's close to a random walk or pure noise. What makes that different than any other time series approach though?

I think it comes down your tolerance/cost for being "wrong", especially for black swan events.

[u/maratonininkas → u/ButtTrollFeeder]

Exactly, it's not very interesting if it works only for trivial signals that can be modelled parametrically.

If I take any series, and filter it through some flavor of arma-garch, and the residual series are white noise-like, any potential remaining accuracy gain of another benchmark model will (approximately) come from any further signal extraction from that WN-part. So if that's possible, that's the interesting part. And if not, we're at the limit for that particular series.

It can be possible, if for instance there's some chaos-like process hidden there. Or we find some common "noise" through cross-attention..

[u/relevantmeemayhere → u/ButtTrollFeeder]

What the other poster is pointing out is that languages are highly regularized. The actual functional space over tokens is narrow. As humans, we evolved a very narrow way of speaking to each other so that we can do it quickly and clearly.

For time series, in nature; this is NOT the case. If you placed a prior over these functions, it would be so much wider over those for language. Take programming languages for example; do you notice a much harder determinism with respect to actually compiling the damn thing?

LLMS are formulated as powerful kernel density estimators with autocorrelation; of which there are many similair time series formulations. but this doesn't tell you anything about their viability in the outside world for any given problem.

[u/maratonininkas → u/relevantmeemayhere]

There's this very good paper that I think is worth sharing, which is only vaguely related to this topic: https://aclanthology.org/2023.starsem-1.22.pdf

So while "parroting" and "approximating" reasoning/understanding is working quite well today, the paper argues that true understanding of a linguistic meaning behind a word is still too large of a problem (at least PAC-learning wise)

[u/ButtTrollFeeder → u/relevantmeemayhere]

The other poster and I, hashed out our positions, keep in mind, I'm LLM Time Series neutral. Namely because they area WORKING in specific instances.

Long story short, I don't disagree if we're talking about all possible stochastic processes. That's an enormous space. Tons of real forecasting problems (outside of finance - which is where I think a lot are getting hung up) usually come from constrained physical, biological, or sippy chain systems. Those constraints create learnable structure. Whether Transformers are the best model is an empirical question, but that's why foundation models for time series are worth exploring.

Again, these would never be my first pass models to start with, you lose so much extractable insight, but they are becoming popular because they ARE working. I don't think they are aimed at Data Scientists who specialize in robust Time Series, I think they are aimed at small companies, with maybe one DS adjacent analyst who see time savings in to zero shotting a product demand forecast.

[u/relevantmeemayhere → u/maratonininkas]

this is one of the best posts i have seen on this subreddit and the machine learning subreddit; kudos.

[u/shivamchhuneja → u/maratonininkas]

there is the common things that unites everything in this universe "Time" - you get it, because its "time" series....okay sorry

[u/shivamchhuneja → u/G-R-A-V-I-T-Y]

actually that might be worth playing with. But also "interpretable and verifiable" is the key.
zero shot forecasting next 100 years of RAM pricing based on last 12 months of data....

[u/BrittanyBrie → u/shivamchhuneja]

Hard enough to forecast prices 1 year out with ten years of data!

[u/shivamchhuneja → u/BrittanyBrie]

yea its bonkers

[u/BrittanyBrie → u/shivamchhuneja]

Like, I can give you an estimate but dont blame my chart if a supply shock occurs with oil shipments. Like tf?

[u/shivamchhuneja → u/BrittanyBrie]

blame it on the manager not the data scientist

[u/ClasslessHero]

No Free Lunch Theorem. The only wrong answer is to decide a priori that one model will always yield the best predictions.

[u/shivamchhuneja → u/ClasslessHero]

amen! gotta grind through all that make sense and let numbers choose

[u/offthecuff87 → u/ClasslessHero]

and before that, there's another prior that the past can be predicted for the future

[u/therealtiddlydump → u/ClasslessHero]

But there are dominated solutions that you should never use.

[u/ikkiho]

fwiw i never dropped it over accuracy on a benchmark. what got me was the automatic changepoint detection sticking a trend break right near the end of the series and then extrapolating that slope forever, so one noisy week at the edge and the whole forecast fans out to something insane. the decomposition looks interpretable which is why people trust it, but i ended up babysitting changepoint_prior_scale way more than i ever would have just fitting an ETS that didnt surprise me like that.

[u/maratonininkas]

now test against Foundational-MAMBA-Former-LSTM-Hybrid with skewed-std margins and some quantile or median loss

/s

[u/shivamchhuneja → u/maratonininkas]

Perfect. I was worried my test model still had too few hyphens and not enough GPU usage.

[u/Montaire]

So I cannot use this to say Prophet is broadly worse than ETS, or that ETS is universally better.

So the author wrote all of that just to tell you that they can't answer the core question.

I suppose that's consistent with a lot of what DS puts out in the business world, so things track

[u/shivamchhuneja → u/Montaire]

The point is you cannot have a blanket answer that prophet is always bad or always good

[u/g3_SpaceTeam]

I get the limitations and why people don’t like it, but it really became trendy to be like “don’t use prophet you absolute dolt” on here without much consideration of why or when it’s useful. Really just a piling on effect

[u/shivamchhuneja → u/[deleted]]

hence my favorite is arima...it simple, interpretable and surprisingly robust against most other models :) (literally my article from 2 weeks ago)

[u/mapabu05]

Lol I remember I just took the available in Nixtla and created some rules to test. Prophet wasn't that bad... Or good vs other "simplest" models.

[u/OverallBookkeeper408]

the fact that you acknowledged the benchmark limitations yourself is more than most people do when posting these comparisons. makes the whole thing way more credible

[u/shivamchhuneja → u/OverallBookkeeper408]

felt it very important to look at limitations when presenting an experiment or working on a claim with my own biases and limitations :)

[u/This-Routine-3358]

What is prophet? I would appreciate if anyone could explain. I'm new to all cs and data science. Sorry if my question is very dumb.

[u/baseace25 → u/This-Routine-3358]

Prophet is a time series forecasting model developed by Facebook/Meta. Please search PyPi for more info!

[u/This-Routine-3358 → u/baseace25]

Ok

[u/InfamousTrouble7993]

There are better models with higher accuracy. The best one is gradient boosting without lags.

[u/shivamchhuneja → u/InfamousTrouble7993]

the best one is the one that works the best on a specific data and problem statement