How This Uncertain Pandemic Influenced The Time Series Model In Production ?

The COVID-19 pandemic broke many production time series models because they were trained on data that contained nothing like it. Models built before 2020 could not anticipate the sudden collapse in demand for some products and the surge for others, so they had to be retrained and monitored more closely.

The episode also highlighted several good practices for building forecasting models that hold up when conditions change.

Key takeaways

• Pre-pandemic forecasting models had no examples of a pandemic in their training data, so they failed to predict the drop in demand.
• Government decisions and public response were external factors that most models did not include.
• Time series need cross-validation that respects time order, such as walk-forward validation.
• Choosing the right forecasting granularity shapes which features are useful.
• Models need ongoing maintenance, and forecasts at different levels should agree with each other.

Why was forecasting so difficult during the pandemic?

Many businesses build projections from time series data, using methods that range from conventional statistical forecasting to deep learning. But real-world data is messy, and a few unusual events can undermine otherwise useful algorithms. The pandemic shut many companies for months and left gaps in historical records. Volatile supply chains, changing workforce capacity and unplanned production halts added further difficulty.

Retail and food businesses illustrate the problem. They want to cut waste while keeping fresh goods in stock, so they rely on demand forecasts and automated ordering. These forecasts are often made at a fine level, such as store, item and day.

Swiggy, the Indian on-demand delivery platform, is a useful example. Companies like it must adapt quickly to changes in key business conditions, and their IT and data teams need to track swings in demand to forecast and keep adequate stock. During the pandemic some industries shut down while others, such as online food delivery, saw sharp increases in use.

How did the pandemic affect time series models in production?

The pandemic caused a major disruption across a wide range of products and services. The effect differed by sector:

At the peak of the pandemic, most production time series models failed to capture the abrupt drop in demand. Pre-pandemic models had no data resembling the pandemic, so they were unprepared for its effects.

In addition, many government decisions were not among the external variables that models could learn from during training. Most forecasting models built before the pandemic were of limited use and needed retraining with a new perspective that reflected policy changes and public response.

What are the best practices for building a production time series model?

Acquiring and assembling the right training data is often a major part of any forecasting project. Beyond that, the following practices matter.

Use the right cross-validation scheme

A suitable cross-validation (CV) strategy is vital for an accurate model. The standard k-fold scheme does not work well for forecasting, because it can end up using future values to predict past ones.

A common alternative is walk-forward CV, which always trains on earlier data and validates on later data. Another important point is to make sure validation folds cover the full range of variation in the target values. Otherwise the model risks underfitting or overfitting.

The CV scheme also matters when building a final ensemble of models for production. In some cases, a custom CV scheme that accounts for both concerns is the best option.

Decide on granularity

Granularity defines how much detail the model should capture. In a retail sales forecast, for example, one must decide whether to forecast at the category level, the item level or the promoted product group (PPG) level.

This choice also determines which features to include. A feature that is useful at one level of granularity may add nothing at another.

Plan for model maintenance

Building a good model is not the end of the work. The central challenge is to maintain it and adapt when needed. Teams should invest time in defining the accuracy thresholds below which a deployed model is considered acceptable, and above which changes are needed, whether to the feature set, the hyperparameters or the CV scheme.

Reconcile forecasts across levels

Leading organizations keep forecasts at several levels for different uses. An aggregate forecast supports high-level decisions such as budget planning, while detailed forecasts support planning and inventory for specific areas.

The worst outcome is a set of detailed forecasts that add up to a very different story from the high-level forecast. Finding the right balance is usually a mix of business knowledge and data science.

Frequently asked questions

Why did models fail during the pandemic?

Their training data did not include anything like a pandemic or the policy responses to it, so they could not anticipate the sudden changes in demand.

Why is standard k-fold cross-validation a poor fit for forecasting?

It mixes time periods, which can let a model learn from future data to predict the past. Walk-forward validation avoids this.

What is forecast reconciliation?

It is the practice of making sure forecasts at different levels, such as store-level and company-wide, are consistent with each other.

Do production forecasting models need retraining?

Yes. Conditions change, so teams should monitor accuracy and retrain or adjust the model when performance drops below an agreed threshold.

Conclusion

The pandemic showed that forecasting models trained only on normal conditions can fail badly when the world changes. Time-aware validation, sensible granularity, regular maintenance and consistent forecasts across levels make production models more resilient.

No model can predict every shock, but teams that monitor and adapt their models recover faster.