A Brief History of Forecasting in the Food IndustryWritten by Sam Mourad on Nov 7, 2019 6:14:00 PM
Forecasting in the food industry still faces many of the same challenges it faced half a century ago, including the following:
- Forecasts are often made with insufficient consideration of external factors, such as campaigns and promotions, moving holidays, weather, and other events. These factors are often either stored in separate, siloed systems or not collected at all.
- Algorithms currently in use present high uncertainty for long-term forecasts, as well as slow reaction to change.
- The existence of an abundance of forecasting techniques that are suitable for specific demand patterns, thus requiring high domain expertise.
- The realities of traditional forecasting algorithms, such as inherent deficiencies where long daily or weekly time series are concerned.
- Scattered data sources and ad hoc solutions, with an overall lack of data centralization
- A general lack of capabilities to consider and account for retailer data when generating forecasts.
We believe the solution is to bridge the data gap across the supply chain by 1.) centralizing all data sources in one platform, and 2.) applying cutting-edge machine learning algorithms that can account for a wide range of external factors. This will then allow for accurate forecasts and more informed decision-making, thus leading to reduced waste throughout the supply chain.
To dissect the problem—and thus understand the solution—let’s delve into the history of forecasting in the food supply chain.
Forecasting in the Food Supply Chain
FACT: Forecasting is a century-old field of study, which makes it one of the oldest areas of predictive analytics.
Where We Began: The concept of “stochasticity in time series” was introduced by the Scottish statistician Udny Yule in the late nineteenth century: put succinctly, he postulated that every time series can be regarded as the realization of a stochastic—or random—process. This one simple idea led to the development of autoregressive and moving average models, which are still being used in many forecasting models today.
Forecasting Techniques Through the Decades
Fast-forward about half a century: academic journals began publishing regularly about statistical approaches to forecasting since the late 1950s, and the first companies to integrate forecasting in their processes reported increased economic gains. Quantitative forecasting methods have been used in multiple industries since the 1960s, leading the Harvard Business Review to publish a guide for selecting among forecasting techniques in 1971.
Another of the most popular forecasting techniques, exponential smoothing (a more advanced variant of this being the Holt-Winters method), originated in the 1960s and was developed further until the 1980s. In the midst of this, the Box-Jenkins methodology for time-series estimation was developed in mid-1970, which then led to the development of the associated computerized version—ARIMA or Auto-Regressive Integrated Moving Average—after the advent of adequate computer technology.
Food Supply Chain Forecasting Challenges: 2000 and Beyond
The early 2000s wrought significant advances that include the development and publishing of state-space approaches to the same forecasting models. However, while these methodologies are coherent and fairly versatile, they still face challenges related to forecasting accuracy, given high-frequency time series with multiple seasonalities with high correlation to external factors. Furthermore, over the past decade, the artificial intelligence (AI) community has compiled a wealth of knowledge and undergone significant development, neither of which the industry has thus far put to good use.
As early as 2001, multiple research papers reported performance gains by using Long-Short Term Memory Networks, and in 2013, Recurrent Neural Networks were proven useful for generating sequences (such as time-series). Since then, multiple deep-learning approaches have been tested and demonstrated to be helpful for time-series, including probabilistic forecasting with autoregressive recurrent networks, the basis of the recently announced Amazon forecast.
We developed the Crisp forecasting technology because we believed there was still a large gap to fill—a gap that is in large part represented by existing tools' difficulty in dealing with the following complexities:
- Drifting holidays
- Multiple coinciding seasonalities
- Non-periodic change-points such as irregularly high demand before an expected winter storm
- Effects of a wide range of external variables such as promotion variables and number and size of customers
- A lack of centralized data, which makes it impossible to reap the benefits of cutting-edge AI and machine learning algorithms
- Existing tools’ inability to account for user feedback and improve forecasting models' performance over time
Here is an example forecast for one of our clients' products that highlights the above points:
Working with daily time series requires complex seasonalities; in this case, yearly and weekly are shown, as are the sales patterns for each day of the year.
Points to note:
- The upward trend in December, modeled in the annual seasonality component
- The demand spikes before Christmas 2018 and Easter 2019 were both accurately predicted, even though:
- Christmas 2018 spike is 20% lower than Christmas 2017.
- Easter is a drifting holiday.
- A slight demand increase in February 2019 related to another local holiday was accurately predicted because of the aforementioned seasonalities, in addition to campaigns. In this case, the conclusion of one of the campaigns provided a signal to the model that demand would be slightly lower, which is also demonstrated on the graph.
For reference: Here is a blog post by Professor Rob Hyndman, a pioneer in the area of forecasting, that explains the difficulties traditional models have in forecasting such complexities.
Why is the history so important? Put simply: as with most other pursuits when generating solutions to complex problems, context is key! A thorough investigation of the industry—past and present—led to our belief that there had to be a better solution for forecasting. Modeling numerous complex, shifting components to generate accurate forecasts is challenging (to say the least) when using traditional forecasting techniques, but with Crisp's forecasting solution, they are done—and done reliably—in the blink of an eye.
Subscribe to our blog to stay current on the latest and greatest in the field of demand forecasting.