Why forecasts that test well still fail in production

The accuracy number that misleads

Most forecasting projects report an accuracy figure early, and it is usually a good one. Then the model goes live, the first month comes in, and the forecast is nowhere near what happened. The team retunes the model, tries a different algorithm, and gets the same result.

The algorithm is rarely the problem. The test was.

The most common approach is a random split: shuffle the history, hold back a fifth of it, train on the rest and measure the error on what was held back. For many machine learning problems that is fine. For forecasting it quietly breaks the test. A random split scatters the test points across the whole history, so the model is trained on data from after many of the points it is asked to predict. It has, in effect, already seen the future.

The score that comes out describes a situation that can never occur in production, because in production the future is never available.

Validate the way you will deploy

The fix is to test the model exactly the way it will be used. Pick a date. Train only on what was known before it. Forecast the months that follow. Record the error. Move the date forward and repeat, many times, across the whole history.

This is walk-forward validation, and it produces a lower number than a random split. That lower number is the first one you can plan against. Run it across at least one full seasonal cycle, so the model is judged in the monsoon as well as the dry months, during festival demand as well as quiet periods.

The second leak: inputs that know too much

Even with honest validation, a model can still cheat through its inputs. Suppose a price forecast uses the month's actual rainfall as a feature. In testing, that rainfall is sitting in the data. In production, when the forecast is made, the month has not happened yet and the rainfall is unknown.

Every input has to pass one test: would this value have been available at the moment the forecast was made? Usually the answer is to use lagged values, or a forecast of the input rather than its actual figure. Watch for data that gets revised after it is first published, too. The value in today's database may not be the value anyone could see at the time.

Accuracy is not the same as usefulness

A forecast nobody trusts never reaches a decision. The person accountable for buying a commodity or setting a production plan needs to understand why the number moved this month, and to judge whether they believe it.

So a useful forecast ships with more than a figure. It carries a range, not a single point, so the plan can account for uncertainty. It carries the drivers that moved it, so a planner can see that the change came from a weather signal or a currency shift. And it carries a record of how accurate the previous forecasts turned out to be.

A short checklist

Before trusting any forecasting model, ask for these:

• Validation by walk-forward testing, across at least one full seasonal cycle.
• A check that every input would have been available when the forecast was made.
• A comparison against a simple baseline, such as last year's figure or a moving average. If the model does not beat it clearly, it is not ready.
• Error reported in the units the business uses, such as rupees or tonnes, not only percentages.
• A commitment to track and publish real accuracy after go-live.

None of this requires an exotic algorithm. In our own commodity price work, the gain from around 70% to 97% accuracy came from honest validation and the right inputs, with an ordinary model doing the arithmetic.