What is Demand Forecasting?
Though we've answered it many times on our blog, this question never gets old. Demand forecasting is the process of estimating future demand through data analysis, which helps companies plan and allocate resources effectively. In very simplified terms, it means predicting how much interest consumers will have in buying a product or service. These predictions are based primarily on historical sales data and current market trends, but also on other external factors. Demand forecasting is also called demand or demand forecasting. We also explain the difference between demand forecasting vs. demand planning in this article, which are often used synonymously.
Classic demand forecasting techniques.
Let's start with the classic methods used in demand forecasting.
1. Time series
- Moving averages: including simple and weighted moving averages – they use historical demand data to create short-term forecasts.
- Exponential smoothing: methods such as simple exponential smoothing, the Holt method and the Holt-Winters method (see Fig. 1) take into account the level, trend and seasonality of the data.
- Seasonality adjustment: techniques for removing or isolating seasonal influences to increase forecasting accuracy.
(Fig. 1: Concepts on which Holt-Winters' exponential smoothing is based, timeseriesreasoning.com)
2. Regression and causal analyses
- Linear regression: Examines the relationship between demand and independent variables such as price or advertising spending to understand influencing factors. This method is often used to understand how various factors influence demand.
- Econometric models: Use statistical methods to quantify cause-and-effect relationships between demand and external influencing factors.
3. Qualitative methods
- Delphi method: Expert opinions are collected in several rounds and evaluated anonymously to create a consensus forecast.
- Sales forecasts: Based on the assessments and experiences of the sales team, which interacts closely with the market and customers.
- Market research: Uses surveys, customer feedback and focus groups, particularly useful when there is limited historical data or when launching new products.
Reasons for their use
Traditional methods are frequently used because they are quite simple and easy to understand. They are just as easy to understand as they are to implement and have low data requirements, so they can often be applied with limited historical data. This is because organisations often lack this data. Furthermore, these methods have proven themselves reliable over decades in various industries and have demonstrated their effectiveness. And last but not least, they are integrated into many common spreadsheet programs and statistical software, which makes them even easier to use. These classic methods thus lay the foundation for many prediction processes and often serve as a benchmark for evaluating newer, more advanced techniques. And these are what we will now look at.
Advanced demand forecasting methods
Now it's getting technical – but it's still worth taking a closer look at these methods, because some of them will have a significant impact on the future of demand forecasting.
1. Machine learning
Artificial neural networks (ANN): Model complex non-linear relationships in the data and learn patterns from training data. They help to identify complex relationships between various influencing factors and demand that are difficult to capture using simple methods.
Decision trees and random forests: Use tree structures to make decisions based on data characteristics. Random forests combine multiple trees to improve accuracy. They are used to identify important factors influencing demand and to understand how different variables affect demand.
Support Vector Machines (SVM): Find the optimal dividing line between data classes and are effective in cases of high dimensionality. They are suitable for predicting demand in complex data sets with many variables, especially when the relationships are nonlinear.
Gradient Boosting Machines (e.g. XGBoost, LightGBM): ensemble methods that combine weak learning models to create strong forecasts. They improve the accuracy of demand forecasting by correcting the errors of previous models and thus providing more precise results.
2. Deep Learning
Recurrent Neural Networks (RNN): Designed for sequential data and take into account temporal dependency.
Long Short-Term Memory (LSTM) Networks: An extension of RNNs that can capture long-term dependencies in time series, especially when past events influence future demand.
Gated Recurrent Units (GRU): Simplified version of LSTM with similar performance. They provide a more efficient way to model temporal dependencies in demand, which reduces computing time for large data sets.
3. Advanced time series models
ARIMA (AutoRegressive Integrated Moving Average): Combines autoregression and moving averages to model stationary time series. ARIMA models are used to forecast demand patterns by extrapolating past patterns.
SARIMA (Seasonal ARIMA): Extension of ARIMA to account for seasonality in data. They are used to model and predict seasonal fluctuations in demand.
VAR (Vector Autoregression): Multivariate model that analyses the interactions between multiple time series variables. VAR models help to understand the mutual influences of several variables on demand.
4. Bayesian methods
Bayesian networks: Model probabilistic relationships between variables and update forecasts with new information. They allow you to take account of uncertainties in demand forecasting and to update forecasts when new data becomes available.
Hierarchical Bayesian models: Capture multi-level data structures and enable complex data analyses. They are used to create forecasts at different levels, such as products, regions or time periods, and to take into account the hierarchy in the demand data.
5. Big Data Analytics and Data Mining
Use of large and diverse data sets to identify patterns and trends that may be overlooked in traditional analyses. This helps to discover hidden patterns and relationships in large data sets, which helps to improve forecasting accuracy.
6. AI-based forecasting tools
Facebook Prophet: an open-source time series forecasting tool that uses additive models and is particularly suitable for business time series. It enables quick and easy creation of demand forecasts, even with irregular or missing data.
TensorFlow Probability: a library for combining probabilistic modelling and deep learning. It is used to create complex demand forecasting models that can both take account of uncertainties and learn deep patterns.
Overall, these advanced methods benefit from the availability of large amounts of data and powerful computing resources to create more accurate and robust demand forecasts.
Improving forecasting accuracy through data analytics in demand forecasting
Due to increasing digitisation and the availability of large amounts of data, data analytics has established itself as a key technology for significantly improving the accuracy of forecasts. This chapter shows how data analytics is used and the numerous advantages it offers companies.
1. The role of data analytics in demand forecasting
Data analytics refers to the systematic computer-based analysis of data to identify patterns and gain insights. In the context of demand forecasting, data analytics makes it possible to:
- Data-driven decisions: By analysing historical data and current trends, you can create well-founded forecasts.
- Identify patterns and trends: Recognise seasonal fluctuations, trends and cyclical patterns that influence demand.
- Adaptability: Respond quickly to market changes and adjust forecasts in real time.
2. Enhanced data quality and preparation.
High-quality data is the key to accurate forecasting. Data analytics improves data quality by:
- Data cleansing: removing duplicates, correcting incorrect entries and dealing with missing values.
- Data integration: combining data from different sources (e.g. sales figures, stock levels, market research).
- Feature engineering: creating relevant variables that improve the forecasting models.
3. Using advanced analytics and models
Data analytics can be used to implement advanced models that recognise complex relationships in the data:
- Machine learning: algorithms such as decision trees, random forests and gradient boosting capture nonlinear patterns.
- Deep learning: neural networks, in particular LSTM networks, are effective at analysing time series and sequence data.
- Predictive analytics: predictive models that forecast future events based on historical data.
4. Integrating external data sources
Data analytics makes it possible to include external factors that can influence demand:
- Weather data: Weather conditions can significantly influence the sale of certain products.
- Social media and trends: Analysis of social media activity to identify trends and sentiment.
- Economic indicators: Macroeconomic data such as the unemployment rate or consumer price index.
5. Real-time analysis and forecasting
With advanced data processing techniques, forecasts can be created and updated in real time:
- Streaming Analytics: processing of data streams in real time for immediate adjustment of forecasts.
- Adaptive algorithms: models that continuously improve based on new data.
6. Pattern Recognition and Segmentation
Cluster analysis and segmentation can be used to identify specific customer groups or product categories:
- Customer segmentation: Adjusting forecasts for different customer groups based on their behaviour.
- Product classification: Identifying products with similar demand profiles.
7. Improving forecasting accuracy using ensemble methods
Ensemble methods combine multiple models to increase forecasting accuracy:
- Model combinations: Exploiting the strengths of different models to minimise forecasting errors.
- Stacking and blending: Advanced techniques for optimising model combinations.
Continuous development of data analytics and related technologies will further expand the possibilities in demand forecasting. Integrating data analytics into demand forecasting leads to significant improvements in forecasting accuracy. Companies that use these technologies early and effectively will benefit from competitive advantages in both the short and long term.
Challenges and solutions
Although data analytics offers numerous advantages in demand forecasting, it also comes with its own specific challenges. These are not only technical in nature, but can also lie in data protection or the organisation itself. We will discuss the main obstacles to the application of data analytics in the context of demand forecasting and present appropriate solutions.
1. Data quality and data management
Challenge:
- Incomplete or inconsistent data: missing values, duplicates and errors in the data records affect forecasts.
- Data silos: data is often isolated in different departments and not centrally available.
- Heterogeneous data sources: different formats and structures make integration difficult.
Possible solutions:
- Implementation of data governance: establishment of guidelines and standards for data collection, storage and use.
- Data cleansing and validation: regular data quality checks and use of tools for automated data cleansing.
- Data integration: use of ETL (extract, transform, load) processes to consolidate various data sources into a central data warehouse or data lake.
2. Model complexity and lack of interpretability
Challenge:
- Black box models: advanced algorithms such as deep learning can be difficult to interpret.
- Barriers to understanding: employees without a technical background may have difficulty understanding the results.
Possible solutions:
- Explainable AI (XAI): Use of techniques such as SHAP values or LIME to make the decision-making processes of the models transparent.
- Training and advanced training: Investment in employee training to promote understanding of data analytics.
- Easy visualisation: Presentation of the results in comprehensible graphics and dashboards that make even complex models tangible.
3. Data protection and compliance
Challenge:
- Legal requirements: laws such as the GDPR set strict guidelines for the processing of personal data.
- Data security: risk of data leaks and unauthorised access.
Possible solutions:
- Anonymisation and pseudonymisation: processing data without direct personal reference.
- Security protocols: implementing firewalls, encryption and regular security checks.
- Compliance management: monitoring and adapting processes to current legal requirements.
4. High resource consumption
Challenge:
- Computing power: Complex models require powerful hardware.
- Costs: Investments in infrastructure, software and specialised personnel can be high.
Possible solutions:
- Cloud computing: Use of scalable cloud services to obtain computing power on demand.
- Open-source tools: Use of free software such as TensorFlow, PyTorch or R for data analytics.
- Efficient modelling: Selection of suitable models that offer a balance between accuracy and resource requirements.
5. Resistance to change and organisational hurdles
Challenge:
- Cultural resistance: Employees may be sceptical of new technologies.
- Silo mentality: A lack of cooperation between departments makes data integration more difficult.
Possible solutions:
- Change management: Active communication of the advantages and integration of employees in the change process.
- Interdisciplinary teams: Promoting collaboration between IT, specialist departments and management.
- Pilot projects: Introducing data analytics in small steps to demonstrate success and increase acceptance.
6. Integration of external data sources
Challenge:
- Data access: Difficulties in accessing external data sources or costs for their use.
- Data compatibility: Different formats and quality standards.
Possible solutions:
- APIs and data feeds: Use of interfaces to efficiently integrate external data.
- Data partnerships: Collaboration with data providers or other companies for data exchange.
- Standardisation: Application of data standards to unify different data sources.
7. Dealing with the uncertainties
Challenge:
- Unforeseen events: External shocks such as pandemics or political events can render models obsolete.
- Model over-adjustment: Risk that models are too closely aligned with historical data and fail to recognise new patterns.
Possible solutions:
- Scenario analysis: developing various forecasts based on different assumptions.
- Regular model validation: checking and updating models at frequent intervals.
- Robust models: using methods that deliver stable results even when there is a degree of uncertainty.
8. Measuring and evaluating the accuracy of forecasts
Challenge:
- Lack of key data: Without clear metrics, it is difficult to evaluate the performance of the forecasting models.
- Short-term focus: Overemphasising short-term accuracy can lead to overlooking long-term trends.
Possible solutions:
- Defined KPIs: Definition of key indicators such as MAPE (Mean Absolute Percentage Error) or RMSE (Root Mean Squared Error).
- Long-term monitoring: Combination of short-term and long-term forecasts for comprehensive evaluation.
- Feedback loops: Continuous feedback from the results into the model adjustment.
Targeted approaches can be used to apply data analytics. A holistic approach that takes into account technical, organisational and human factors is essential. By taking proactive measures, companies can overcome hurdles to achieve the desired benefits.
Future trends in demand forecasting
Finally, let's take a look at the three most important future trends in demand forecasting: the ongoing integration of artificial intelligence and machine learning, the use of big data and the Internet of Things (IoT), and the development towards real-time and adaptive forecasting models.
1. Artificial intelligence and machine learning
The first significant trend is the increased application of artificial intelligence (AI) and machine learning (ML) in demand forecasting. While traditional methods often reach their limits, AI and ML offer the opportunity to recognise complex patterns and nonlinear relationships in large data sets. In particular, deep learning algorithms such as neural networks and recurrent neural networks (RNNs) make it possible to efficiently process historical data and real-time information.
In the future, the further development of algorithms such as reinforcement learning and generative adversarial networks (GANs) will further improve forecasting capabilities. These technologies can produce self-learning models that dynamically adapt to transformed market conditions. Furthermore, Explainable AI (XAI) is gaining importance to increase the transparency and interpretability of ML models. The world is complex enough as it is, and this simplification and accessibility strengthens the trust of all users and promotes the acceptance of AI-supported forecasts in a business context.
2. Big Data and the Internet of Things (IoT)
The second trend is the increasing importance of big data and the Internet of Things in demand forecasting. With the exponential increase in networked devices and sensors, companies have even more structured and unstructured data at their disposal. This data comes from a wide range of sources, such as social media, mobile applications and industrial sensors.
The integration of IoT data enables forecasts to be more granular and up to date. For example, real-time data on customer interactions, stock levels or supply chain events can be used to respond immediately to changes. Big data technologies such as Hadoop and Spark provide the infrastructure needed to store and process these large amounts of data. By combining big data and advanced analytics, companies can gain deeper insights and continuously improve their forecasting models.
3. Real-time and adaptive forecasting models
The third key trend is the development towards real-time and adaptive forecasting models. In a globalised and highly dynamic economy, organisations must be able to react promptly to market changes. Traditional static forecasting models are not sufficient to meet real-time requirements.
Adaptive models continuously use incoming data to update forecasts in real time. This is made possible by technologies such as streaming analytics and in-memory computing, which can process and analyse data instantly. This enables companies to proactively respond to changes in demand and further optimise inventories and supply chains.
In addition, cloud technology is boosting the scalability and flexibility of these models. Cloud-based platforms allow computing resources to be scaled as needed and new models to be implemented more quickly. This enables companies to test innovative forecasting methods and quickly roll them out if successful.
Conclusion
Future innovations in demand forecasting offer enormous potential for increasing forecasting accuracy and boosting operational efficiency. Companies that adapt these developments early on will be better equipped to successfully meet the challenges of a constantly changing market. pacemaker will accompany you on your journey into that future, providing solutions for every demand forecasting challenge.