- Report the seasonal naive baseline next to every forecast you produce. A model that does not beat last season is not a model, it is a slower way of guessing.
- A random train test split leaks the future into training and produces beautiful numbers that collapse in production. Use rolling origin evaluation instead.
- Pick the model by how much data you have and how many series you run, not by what is newest. Gradient boosted trees on lag and calendar features win a great deal of real business forecasting.
- Percentage error metrics break when demand hits zero, which is most days for most products in a long tail catalogue.
- Zero shot foundation models are genuinely useful for series with no history, and their published results are hard to read because of contamination risk.
- The determinants nobody plans for are known future events such as promotions and holidays, intermittent demand, and reconciling forecasts across levels.
Somebody has asked you to forecast demand. Maybe it is stock for the next quarter, maybe staffing, maybe traffic. The advice arriving from every direction points at a neural network, and you have a nagging suspicion that the boring thing, last year's number adjusted for growth, might do about as well.
That suspicion is correct often enough that it should be the starting point of the whole project rather than an afterthought. This piece is a selection ladder ordered by cost, an evaluation protocol that stops you fooling yourself, and the handful of practical details that decide whether a forecast survives contact with a real business.
What is the baseline, and why does it decide everything?
The seasonal naive forecast says that next Tuesday looks like last Tuesday, or that next December looks like last December. It costs nothing, it needs one season of history, and it is startlingly hard to beat on many real series.
Report it alongside every model you build, always, on the same evaluation and the same data. This single habit does more for forecasting quality than any model choice, because it converts an unanswerable question, is this forecast good, into an answerable one: is it better than doing nothing clever. A model that beats seasonal naive by 3% on a series that drives a six figure stock decision may still be worth building. A model that loses to it has told you something important and you should stop.
The reason this needs stating is that the loss is common and it goes unnoticed. Nobody computes the baseline, so nobody discovers that the project produced a negative result, and the model ships.
The ladder, ordered by what it costs you
Work down this list and stop at the first rung that beats the one above it by enough to matter. Most businesses stop earlier than they expect.
| Rung | What it is | When it is the right answer | What it needs |
|---|---|---|---|
| Seasonal naive | Repeat the last equivalent period | Always, as the baseline. Sometimes as the answer. | One season of history |
| Exponential smoothing | Weighted average with trend and seasonality | Few series, clear seasonality, no external drivers | A couple of seasons, minutes to fit |
| Classical statistical models | ARIMA and relatives, fitted per series | Tens of series, stable behaviour, a need to explain the model | Several seasons, and someone who can read a diagnostic |
| Gradient boosted trees | One model over all series, using lag and calendar features | Hundreds or thousands of series with shared structure, external drivers | A feature pipeline, and history across the catalogue |
| Deep sequence models | Neural networks trained across many series | Very many series, long histories, and a team to maintain it | Substantial data, compute, and ongoing ownership |
| Zero shot foundation models | Pretrained on many series, forecasts one it has never seen | Cold start, new products, long tail series with no history | Almost nothing, which is the point |
The fourth rung deserves the attention it rarely gets. Training one model across your whole catalogue rather than one model per product is what unlocks the volume advantage, because a new product with six weeks of history borrows the shape of the thousand products that came before it. That approach is what the largest public forecasting competition was built to test, and the section below returns to it.
Whether the deep rungs are worth it is genuinely contested. Zeng, Chen, Zhang and Xu asked whether transformers are effective for long term forecasting and found that a one layer linear model outperformed sophisticated transformer based forecasters on nine real world datasets, often by a large margin, arguing that self attention loses the temporal ordering that this problem depends on. Their result did not end the argument, and it did establish that a simple model is a serious competitor rather than a courtesy comparison.
What the biggest forecasting test actually studied
Worth grounding the ladder in the largest public exercise of its kind. The M5 accuracy competition asked entrants to forecast 42,840 time series of hierarchical unit sales from Walmart, which is a retail problem rather than a synthetic one: real products, real stores, real promotions, real zeros. Machine learning methods, and gradient boosting in particular, outperformed the statistical benchmarks the organisers supplied.
Two features of that result are more useful to a small business than the leaderboard. The winning approaches learned across series rather than fitting each product separately, which is the mechanism that lets a thin catalogue borrow strength from a thick one. And the benchmarks the field measured against were the simple ones: seasonal naive, simple exponential smoothing, and Croston's method for intermittent demand. Even at that scale, with prize money involved, the reference point was the boring forecast.
The trap in reading a competition result is assuming the ranking transfers. It was measured on one retailer's data with a specific hierarchy and a specific horizon. What transfers is the method of judging, not the winner.
How much history do you actually need?
More than people hope for seasonal patterns and less than people fear for everything else. To learn an annual seasonal shape a model needs to have seen that season happen more than once, which means two full years before an annual pattern is anything other than an assumption. Weekly seasonality is far cheaper: a few months of daily data will show you that Saturdays differ from Tuesdays.
This has a direct consequence for a business under three years old. Your annual seasonality is not measured, it is borrowed, either from category knowledge or from a model trained on other people's series. Say that out loud when you present a forecast for next December, because the confidence interval a model reports does not include the uncertainty about whether the pattern exists at all.
The other quantity that matters is series count. One product with three years of history supports a per series statistical model. Two thousand products with six months each support a cross learned model and nothing else. Knowing which of those two shapes you are in decides the rung before any accuracy number is computed.
Why is a random train test split wrong here?
Because it trains your model on the future and then congratulates it for predicting the past. Shuffle rows from a time series into training and test sets and the model sees next March while learning to predict February. The resulting error is not merely optimistic, it is measuring something that cannot happen in production.
The correct procedure has a name and a clear description. In Forecasting: Principles and Practice, Hyndman and Athanasopoulos set out time series cross validation, in which there is a series of test sets each consisting of a single observation, and the training set for each contains only observations that occurred prior to it, so that no future observations can be used in constructing the forecast. Accuracy is averaged over the test sets, and they note the procedure is sometimes called evaluation on a rolling forecasting origin because the origin rolls forward in time.
Two consequences follow that are worth building in from the start. You can extend the same procedure to multi step forecasts, which matters because a one step ahead number is rarely the decision you are making. And because you get an error per horizon, you can see the accuracy decay as you forecast further out, which is the honest way to answer how far ahead you can plan.
If a vendor or a colleague shows you a forecast accuracy figure, ask one question before any other: how was the data split. If the answer is a percentage of rows held out at random, the number means nothing and no further discussion is useful. This is the single most common way forecasting projects produce results that evaporate.
Which error metric matches the decision?
The metric is not a formality. Each one encodes a different opinion about what kind of mistake hurts, and picking the wrong one steers the model away from the outcome you wanted.
| Metric | What it rewards | Where it breaks | Use it when |
|---|---|---|---|
| MAPE, mean absolute percentage error | Small errors relative to the actual value | Division by zero when demand is zero, and it penalises over forecasting more heavily than under forecasting | Series that never approach zero and where relative error is the business unit |
| MASE, mean absolute scaled error | Beating a naive benchmark, on any scale | Little. It needs a benchmark defined, which is a feature | Comparing across products with wildly different volumes |
| RMSE | Avoiding large errors, since they are squared | Dominated by a few extreme periods, which may be the ones you care least about | When a single big miss is the expensive event |
| Quantile or pinball loss | Getting the whole distribution right, not the middle | Harder to explain to a stakeholder | Any decision with an interval, which includes almost every stock decision |
MAPE deserves the warning because it is the default in most business reporting. Divide the absolute error by the actual value and a zero sales day gives you a division by zero. Near zero days give you an enormous percentage from a trivial miss. In a long tail catalogue, where most products sell nothing on most days, MAPE is not a slightly imperfect choice, it is undefined for most of your data.
The last row is the one that changes decisions. You are almost never asking how many units will sell. You are asking how many to hold so that you run out less than one time in twenty, which is a question about a quantile of the distribution rather than about its centre. A model tuned to the average will systematically under stock, and no amount of accuracy improvement fixes a mismatch between the metric and the decision.
Are foundation models worth trying?
Yes for one specific situation, and read their benchmarks carefully. Chronos, from Ansari and colleagues at Amazon Science, tokenises time series by scaling and quantisation and trains language model architectures on them, reporting comparable and occasionally superior zero shot performance on new datasets against methods trained specifically on those datasets, across 42 datasets. Zero shot means the model forecasts a series it has never seen, with no fitting step.
That capability maps onto a real and expensive problem: the new product, the new store, the seasonal line with no history. Traditional methods have nothing to fit there, and the usual workaround is to copy a similar product by hand.
The caution is about evaluation rather than about the models. Meyer, Kaltenpoth, Zalipski and Müller argue that benchmarking these models is compromised by information leakage in two forms, train and test sample overlaps where datasets are reused, and temporal correlation overlap where training and test series share correlated periods, and warn that ignoring this risks overly optimistic performance estimates that fail to generalise. As pretraining corpora get larger, the chance that a public benchmark series was seen during training goes up rather than down.
The practical response is the same one that works for every model claim: evaluate on your own data, on a rolling origin, against your own baseline. A zero shot model is unusually easy to test this way because there is nothing to train. An afternoon settles it.
The three details that decide whether it works
None of these are model choices, and all three sink more projects than model choice does.
Known future events. You already know when your promotions run, when the holidays fall, when the sale starts, when the product goes out of stock deliberately. A forecast that does not receive that information is being asked to guess something you could simply tell it. This is the highest return feature engineering available and it is usually the last thing anybody does. The same point applies to the data underneath: a forecast is only as good as the record of what actually happened, which is why we put data readiness before modelling in the guide to what an inventory forecast needs first.
Intermittent demand. Many products sell in ones and zeros with long gaps. Standard methods handle this badly and standard metrics describe it worse. If most of your catalogue looks like this, treat it as a different problem with its own methods rather than as an inconvenient case of the main one, and consider whether you need a forecast at all or simply a reorder rule.
Reconciliation across levels. You will forecast at product level and also need a total, or forecast by store and need a region. Forecast them independently and they will not add up, which is not a rounding issue but a credibility issue the first time somebody puts two of your numbers in the same slide. Decide early whether you forecast the bottom and aggregate, forecast the top and split, or reconcile properly.
What should a small business actually do?
Six steps, and the first four cost nothing but time.
One. Assemble the history: sales by product by day or week, going back at least two seasons, with the stockouts marked. A day with zero sales because you had none in stock is not a day with zero demand, and treating it as one teaches the model to under forecast exactly the products that sell best.
Two. Compute the seasonal naive forecast and its error on a rolling origin. Write that number down. It is the target for everything that follows.
Three. Add the calendar and the known events. Holidays, promotions, paydays, whatever moves your demand.
Four. Try one model from the middle of the ladder, evaluated the same way. For most catalogues with more than a few hundred products, gradient boosted trees on lag and calendar features is the rung with the best return per unit of effort.
Five. Convert the forecast into the decision. Pick your service level, forecast the quantile that supports it, and check what the recommendation actually does to your stock position before you act on it.
Six. Re run the evaluation monthly. Forecast quality degrades quietly as the business changes, and the failure mode is not a wrong number, it is a number nobody checked for a year. Where that monitoring habit sits inside a wider operational practice is set out in our piece on what to build first as MLOps maturity grows.
How do you keep this section current?
The foundation model landscape is the part of this article that will age, and it is deliberately separated so that updating it does not require rewriting anything else. Everything above it, the baseline habit, the rolling origin protocol, the metric to decision mapping, and the three practical determinants, has been true for decades and will outlive whatever ships next year.
When a new model appears, the test does not change. Run it against your seasonal naive, on your data, on a rolling origin, with a metric that matches your decision. If it wins, adopt it. If the only evidence is a benchmark table, remember the leakage argument above and treat the table as marketing until you have replicated it on something the model has not seen.
That discipline is not specific to forecasting. It is the same argument we make about evaluating any model for your own task, and it has the same conclusion: the only benchmark that matters is the one built from your own data, and the businesses that own that data outright are the ones who can actually run it. If your sales history is trapped inside a platform you cannot export, that is a forecasting problem before it is anything else, which is one of the reasons we built our store builder around code and data you own.