Measures of Forecast Error

Since the first principle of forecasting is that forecasts are (almost) always wrong, organizations need to track the forecast against actual demand results and find ways to measure the size and type of error. Note that the size of an error can be measured in units or percentages, but often finding a way to put a monetary value on the error can help in focusing.

Forecast Error

Forecast error is the difference between actual demand and forecast demand, stated as an absolute value or as a percentage.

Forecast Error = | A – F |

Forecast Error as Percentage = | A – F | / A

Where:

A = Actual demand

F = Forecast demand

Forecast Accuracy

Forecast accuracy is simply the inverse of the forecast error as a percentage, expressed as follows:

Forecast Accuracy = 1 – Forecast Error as Percentage

Bias and Random Variation

Bias

  • Forecast error can be the result of bias or random variation.
  • Bias is a consistent deviation from the mean in one direction (high or low). A normal property of a good forecast is that it is not biased.
  • Bias exist when the cumulative actual demand differs from the cumulative actual forecast.
Cumulative Forecast Error = Cumulative Actual Demand – Cumulative Forecast Demand
  • Any answer that does not result in zero reflects a bias.
  • The size of the number reflects the relative amount of bias that it present.
  • A negative result shows that actual demand was consistently less than the forecast, while positive result shows that actual demand was greater than forecast demand.

Random Variation

In terms of measuring errors, random variation is any amount of variation in which the cumulative actual demand equals the cumulative forecast demand.

Mean Absolute Deviation (MAD)

  • A common way of tracking the extent of forecast error is to add the absolute period errors for a series of periods and divide by the number of periods. This give you Mean Absolute Deviation (MAD).
MAD = ∑ |A – F| / n

Where:

|A – F| = Total of absolute forecast errors for the periods

n = Number of periods

 

  • The average of the absolute values of the deviations of observed values from some expected value. It can be calculated based on observations and the arithmetic mean of those observations. An alternative is to calculate absolute deviations of actual sales minus forecast data. These data can be averaged in the usual arithmetic way or with exponential smoothing.

NOTE: With absolute values, whether the forecast falls short of demand or exceeds demand does not matter; only the magnitude of the deviation counts in MAD.

  • An analyst would provide actual MADs for a given service level. If a specific service level is desired, such as 98 percent of orders with no stock outs, analyst can calculate the exact MAD to use a s a multiplier in the calculation of units of safety stock. The multiplier is called a safety factor. For example, if a 98 percent service level has a safety factor of 2.56 MAD, the calculation would be as follows:

    2.56 Safety Factor x 8.23 MAD in units = 21.07 Units of Safety Stock

Tracking Signal

  • The tracking signal is the ratio of the cumulative algebraic sum of the deviations between the forecasts and the actual values to the mean absolute deviation. Used to signal when the validity of the forecasting model might be in doubt.
Tracking Signal = Algebraic Sum of Forecast Errors / Mean Absolute Deviation (MAD)
  • Note that the algebraic sum of forecast errors is a cumulative sum that does not use absolute value for the errors. Therefore, the tracking signal could be either positive or negative to show the direction of the bias. Organizations use a tracking signal by setting a target value for each period, such as ±4. If the tracking signal exceeds this target value, it would trigger a forecast review.

Standard Deviation

In addition to MAD, another way to calculate forecast error would be to use standard deviation, which is commonly provided in most software programs. An approximation for standard deviation when you know the MAD.

Standard Deviation (approximate) = MAD x 1.25

Mean Squared Deviation (MSE)

  • Another method of calculating error rates, the mean squared error (MSE), magnifies the errors by squaring each one before adding them u and dividing by the number of forecast periods.
  • Squaring errors effectively makes them absolute since multiplying two negative numbers always results in a positive number.
MSE = ∑(Error for each period)²/ Number of forecast periods

 

MSE and MAD Comparison

  • Note that the process of squaring of each error gives you a much wider range of numbers.
  • The greater range gives you a more sensitive measure of the error rate, which is especially useful if the absolute error numbers are relatively close together and reduction of errors is important.
  • Measuring the extent of deviation helps determine the need to improve forecasting or rely on safety stock to meet customer service objectives.

Mean Absolute Percentage Error (MAPE)

  • There is a drawback to the MAD calculation, in that it is an absolute number that is not meaningful unless compared to the forecast.
  • MAPE is a useful variant of the MAD calculation because it shows the ratio, or percentage, of the absolute errors to the actual demand for a given number of periods.
MAPE = ∑( |A – F| / A ) % / n

 

  • Note that the result is expressed as a percentage.
  • Exception rules for review can be applied to any stock keeping unit or product family that has a MAPE above a certain percentage value. Percentage – based error measurements such a s MAPE allow the magnitude of error to be clearly seen without needing detailed knowledge of the product or family, whereas when an absolute error in units (or an error in $ amount) is provided, it requires knowing what is considered normal for the product or product family.

Qualitative Methods

  • When a product is new, however, or when data are lacking for one reasons, you have to rely on judgement and intuition. In such cases, you are best advised to find the most experienced, market-savy, objective person – or better yet, group of experts – and rely on them for a rough estimate of likely demand.
  • A dose of intuition from a reliable source can be helpful even when working with plentiful data.
  • Five major types of qualitative forecasting are discussed here:
    • Personal insight.
    • Sales force consensus estimate
    • Management estimate (panel discussion)
    • Market research
    • Delphi method

Personal Insight

Forecasts may sometimes be based upon insight of the most experienced, most knowledgeable, or most senior person available.

Sales Force Consensus Estimate

  • The sales and marketing area (or areas) brings special expertise to forecasting, because they maintain the closest contact with customers. While they may participate in gathering data for quantitative forecasts, their special contribution comes at the qualitative level. Even when there is a quantitative forecast, the sales force should be given a chance to review it to see if it is consistent with their knowledge of the marketplace.
  • Bringing the entire field sales force together to create a consensus forecast provides the firm – and, at least indirectly, its supply chain partners – a view of the whole market, including all sectors or geographic regions. Demand, obviously, can vary greatly in different market segments.
  • Like all functional areas, sales and marketing may bring a special bias to their demand forecasts, generally an optimistic – sometimes an overlay optimistic – one.

Management Estimate

The management estimate relies upon a consensus of panel members. Generally, the panel of management-level experts conducts a series of forecasting meetings, with the results of one meeting providing the basis for the next until the panel reaches a consensus. In the process, the panel may rely upon various techniques, including pyramid forecasting and forecasting by historical analogy.

  1. Pyramid Forecasting

    Pyramid forecasting, or rationalizing high- and low-level forecasts, enables management to review and adjust forecasts made at an aggregate level and keep lower-level forecasts balanced. In the process, item forecasts first are aggregated by product group. Management then makes a new forecast for the group. The value is then transferred to individual item forecasts so that they are consistent with the aggregate plan.

  2. Historical Analogy

    When there are no data on a new product or service, forecasters may instead study past patterns of demand for a similar product or service.

Market Research

  • Market research (also known as marketing research) is the systematic gathering, recording, and analyzing of data about problems relating to the marketing of goods and services. Such research may be undertaken by impartial agencies or by business firms or their agents.
  • Agent as one who acts on behalf of another (the principal) in dealing with a third party. Examples include a sales agent and purchasing agent.

Market research includes the following approaches:

  • Market analysis, including product potential studies, which seeks to determine the size, location, nature, and characteristics of a market.
  • Sales analysis, or sales research, which undertakes the systematic study and comparison of sales data.
  • Consumer research, such as motivational research, focus groups, questionnaires, and other methods used to discover and analyze consumer attitudes, reactions, and preferences.
  • Test marketing, which introduces a product or service in a limited pilot area.

Note that when collecting information with questionnaires or surveys the number of responses compared to the number of nonresponses or incomplete answers should also be tracked to determine if the data are statistically valid.

Delphi Method

  • The Delphi method, like sales force and management estimate forecasting, relies upon a panel of experts in the field being studied. Also like other panel based methods, it relies upon the experience, wisdom, insight, and even the intuition of disciplined observers acting in concert.
  • In Delphi method, questionnaires are generally submitted to the individual experts for their anonymous responses in successive rounds. After responding to the questions in one round, the experts comment on replies from the previous round. After hearing replies and responses, the experts have a chance to revise their own previous work. This iterative process aims to reduce differences in thinking as the answers of experts converge, round by round, upon an increasingly accurate consensus forecast.
  • A key feature of the Delphi method is the maintenance of anonymity throughout the process. Instead of meeting face to face, the experts submit their responses, comments, and revisions to a panel director, who is empowered to delete irrelevant information. This reduces the defensiveness that can cause group members to resist changing mistaken views when challenged in person. It also reduces or eliminates the opposite problem: the “groupthink” effect that can cause even a collection of independent thinkers to become emotionally committed to an unrealistic forecast. This is especially likely to happen when a charismatic, opinionated member takes over leadership of the group and steers it in a mistaken direction.
  • The Delphi method suffers the mixed results as it is based on human judgement (like other forecasting systems).
  • The Delphi method can be time-consuming and is best for long-term forecasts.

Quantitative Methods

Quantitative forecasting techniques rely on historical data. They can be divided into two types:

  • Extrinsic
    • (Definition) Extrinsic Forecasting is a forecast method based on a correlated leading indicator, such as estimating furniture sales based on housing starts. Extrinsic forecasts tend to be more useful for large aggregations, such as total company sales, than for individual product sales.
    • Extrinsic techniques are known as casual techniques, because they analyze data on conditions thought to result in changes in demand for a particular item or group of items, such as forecasts of diaper demand based on the birth rate, i.e. the seek to find correlation or a cause-and-effect relationship between the indicator and overall market demand.
    • (Definition) Market Demand is the total demand that would exist within a defined customer group in a given geographical area during particular time period given a known marketing program.
  • Intrinsic
    • (Definition) Intrinsic Forecast is a forecast based on internal factors, such as an average of past sales.
    • Intrinsic techniques are known as time series models because they incorporate data collected during set intervals of time – hours, days, weeks, months.
    • Intrinsic techniques tend to be best for short-term forecasting.

The best practice for organizations is to do some form of both intrinsic and extrinsic forecasting and then modify these results with some type of qualitative method.

“Intrinsic” and “Extrinsic’ are simply referring to the source of the data.

Data Gathering and Formatting

Look at some rules to follow for gathering good data:

  • Record data in the terms needed for the forecast
    • Forecasts contain three dimensions:
      • Amount of demand
      • Time period
      • Items demanded
    • Track demand, not shipments. You want to know when customers actually wanted the items, not when the items were sent out.
    • Keep records for the same time periods to be used in scheduling – weeks, months, quarters.
    • Forecast demand for the items manufactured, including all product options, as well as total demand for the product. Demand forecasts for the product groups are more efficient when an overall forecast for a product group is coupled with a percentage breakdown for each subset of the group rather than producing multiple independent forecasts for each item.
  • Subtract returns and cancellations from the demand data
    • Be sure to record returns and cancellations and subtract them from the demand data for the period. They do not represent true demand.
  • Record events that may influence demand
    • Some events that may influence demand include sales promotions, strikes, and a competitor’s product introduction.
  • Keep separate demand records for each customer group
    • Your customer may typically buy in large or small lots, on different schedules, and so forth. Average overall demand will not necessarily be helpful in setting your schedules.

Quantitative Forecasting Techniques Based on Extrinsic Data

  • Extrinsic data is most commonly used in quantitative forecasting.
  • Some of the leading indicators that financial experts watch to predict coming trends include housing starts, construction contract award, automobile production, farm income, steel production, and gross national income. Unlike seasonal variations, these economic trends move in cycles that extend over a period of years. They are more likely to be helpful in predicting aggregate demand than demand for specific products or services.
  • Extrinsic data are more useful if they relate to very recent event s and trends. The more time that has passed since the time period the data refer to, the less useful the data become.
  • While it is important to verify that the extrinsic data used in forecasts are relatively fresh, the key challenge is to select an indicator that has true correlation to the demand being forecasted.

Example of Extrinsic Data in Chart Form (Source: www.calculatedriskblog.com)

Quantitative Forecasting Techniques Based on Intrinsic Data

  • Quantitative forecasting techniques using intrinsic data are also called time series models because they embody the notion that data distributed over time showing past performance can be used to predict performance in the future – almost always with some degree of error.
  • To be credible, time series forecasts should include an estimate of the degree of potential error.
  • A time series shows some combination of four types of variability: trend, seasonality, random variation and cycle.

  • Following are some basic techniques for developing forecasts from time series data, listed in ascending or descending order of sophistication:
    • Naïve Approach
    • Moving Averages
    • Weighted Moving Averages
    • Exponential smoothing

Naive Forecasting

  • Naïve forecasting assumes that demand in the next time period will be the same as demand in the last time period. For example, if a retailer sells 600 pairs of booths in February, the naïve forecast would be for demand of 600 pairs of boots in March.
  • This approach rules out all the types of fluctuation – trends, seasonality, random variation and cycles.
  • An alternative type of naïve forecast takes seasonality into account by assuming that demand for the month will be the same as demand for the same month in the prior year. This type of forecast can be useful if the trend tends to be flat and there is little random variation.

Moving Average Forecasting

  • Moving averages (or simple moving averages) represent a step up in sophistication from the naïve approach.
  • Instead of using the most recent period to forecast demand for the next period, a moving average uses the average demand from a series of preceding periods to forecast the next period’s demand.
  • It’s a “moving” average because it is recalculated for each new period.
  • A three month moving average, for instance, takes the average demand per month for the three preceding months and updates the calculation each month.

    3-Month Moving Average = (M1+M2+M3) / 3

  • Moving averages can be quiet useful – and many firms do use them – when demand is fairly constant from period to period.
  • The moving average mitigates the effects of random variations, so that orders do not vary in amount quiet so much from time to time as they would with a naïve forecast. This makes production more predictable, and that can be a money saver.
  • The naïve forecast methodology prevents an overreaction to any one month’s random variation.
  • The moving average method can be of limited usefulness for a product with wide seasonal variations in demand on top of random fluctuations.
  • Moving averages generally fail to recognize trends or seasonal effects.
  • On the plus side, the moving average method dies tend to correct for random variations. The more periods included in the average (say 6 months instead of three) the more it corrects for chance variations. However, the moving average method lags behind actual demand, and the more periods you include in the average, the more it lags.

The pluses and minuses of using the moving average method of forecasting can be summed up as follows:

  • The moving average smoothies out random variations, with longer periods removing more of the randomness. But longer periods also make the average less sensitive to real changes in the data.
  • Moving averages are not sensitive to trends or seasonal effects. The average of previous results by definition cannot predict movement to a higher or lower value than the period preceding the forecast.
  • As more periods are included in the average, calculations become more complex and you have to collect more data for every product/group.

Weighted Moving Average Forecasting

  • Weighted moving averages can provide somewhat more sophisticated basis for discerning data trends. To do this weighted average usually places more emphasis on recent periods and less on earlier periods, assuming that more recent periods have better predictive relevance.
  • Greater weight might alternatively be placed on earlier periods when a trend or seasonal effect is expected to change, for example, in the later stages of a product life cycle.
  • Weighing is done simply by multiplying recent demand numbers to make them larger before inserting them into the equation for calculating the average.

    3-Month Weighted Average = (W1 x M1) + (W2 X M2) + (W3 x M3) / (W1+W2+W3)

  • The challenging part of weighted averaging is selecting the weights. There is no formula to help you decide which weights will work best. You have to rely on guesswork or trial and error, which can be facilitated by the right software.

Exponential Smoothing

  • Exponential smoothing is more sophisticated version of the weighted moving average that is relatively easy to use and required very little record keeping.
  • The exponential smoothing equation requires only three basic terms:
    • The last period’s forecast.
    • The last period’s actual demand and
    • A smoothing constant (a number greater than 0 and less than 1 represented by (α), which is basically a percentage weighting where 1=100 percent).
  • There are two ways to present the same equation:
    • The first method helps call out how this forecast is built:

    New Forecast = Last Period’s Forecast + α (Last Period’s Demand – Last Period’s Forecast)

    • The second method is easier to calculate and is more commonly used:

    New Forecast = (α x Last Period’s Demand) + [(1-α) x Last Period’s Forecast]

  • Smoothing can make your forecasts somewhat more responsive to trends than simple moving averages. Like any weighted moving average, however, exponential smoothing lags behind changes in the data.
  • Generally speaking, firms use exponential smoothing constants that fall between 0.05 and 0.5. The higher the constant, the more weight your forecast gives to the actual demand data from the preceding period. A constant of 0.05 would give minimal weight to the preceding period. A constant of 1.0 would yield the same result as a naïve forecast, because it would include the entire last period’s demand (100%) and none of the last period’s forecast amount (0%).
  • Using best possible smoothing constant is crucial. The accuracy of your forecast depends upon it. Like the constants used in simple weighted averages, the exponential smoothing constant is not a given. It has to be determined and thus reflects the best judgement of experts, the series of previous forecast and demand numbers, or a combination of the two. Like the constants used in weighted averaging, the smoothing constant can be selected by trying various constants on the historical data to see which works best.

Modifying Quantitative Methods

  • Quantitative methods are usually modified to account for trends (including cycles) and seasonal effects.
  • As part of forecasting process, forecasters decompose data to remove trend and seasonality and then proceed with the forecast projection. Once the qualitative forecast of intrinsic data is completed, the forecasters recompose the forecast by applying any trend, cycle and seasonality adjustments to the forecasted data. Adjustments include modifying forecasts for internal trends (such as historical growth rate in demand of five percent for a product) or external trends (such as a recession suppressing demand to historically low levels).

Seasonal Index

  • Seasonal variations play havoc with forecasts, even those using weighted averages.
  • Since moving average always lags behind an upward or downward trend, forecasters have to develop other quantitative methods to take account of these seasonal movements.
  • “Seasonal” in this context can refer to any forecasting period, from actual seasons, to months, to days, or even to parts of days.

Example

Computing seasonal index for home owner market segment for gas fireplaces requires the following calculations. We will use 3 years’ worth of monthly demand data:

  1. What is the seasonal average of monthly demand for each of the 12 months in the past 3 years?

    To determine this, divide the sum of the demand in the three January’s by 3, divide the sum of all February demand by 3, and so on till December.

  2. What is the deseasonalized average monthly demand during those 3 years?

    Calculate this by adding 12 monthly averages from the preceding step and divide the total by 12. (This monthly average is said to be “deseasonalized”, because it ignores the effect of seasonal variation. Note that using deseasonalized demand is an example of decomposing data prior to proceeding to the quantitative forecast computation).

  3. What is the seasonal (monthly) index?

    This is the average of particular month divided by the overall deseasonalized monthly average demand.

Seasonal Index = Average Demand for Period (e.g. Month) /
Deseasonalized Average Demand for all Periods (e.g. Year)

 

Quantitative and Qualitative Forecasting

Quantitative Forecasting Techniques

(Definition) An approach to forecasting where historical demand data are used to project future demand. Extrinsic and Intrinsic techniques are typically used.

Qualitative Forecasting Techniques

(Definition) An approach to forecasting that is based on intuitive or judgmental evaluation. It is used generally when data are scarce, not available, or no longer relevant. Common types of qualitative techniques include: personal insight, sales force estimates, panel consensus, market research, visionary forecasting, and the Delphi method. Examples include developing long-range projections and new product introduction.

**Combination Methods

  • Good forecasting is best done with combination of quantitative and qualitative considerations.
  • When using combination methods, both the quantitative forecast and the qualitative adjusted forecast can be measured separately for error to determine the degree to which qualitative methods are helping or hindering forecasting.

Demand Forecasting Concepts

  • (Definition) Forecasting is the business function that attempts to predict sales and use of products so they can be purchased or manufactured in appropriate quantities in advance.
  • (Definition) Demand Forecasting is forecasting the demand for a particular good, component, or service.
  • Forecasts are subject to uncertainty, and this uncertainty is one potential contributor to the bullwhip effect.

Principles of Forecasting

  • Forecasts are (almost) always wrong
    • A forecast is at the best an estimate of what may happen in the future – if there are no surprises.
    • Circumstances and minds can change. For this reason, forecasts require regular review.
    • Forecasting techniques should be subject to alteration if forecast errors grow too large.
  • Forecasts should include an estimate of error
    • Demand forecasts should include an estimate of how large the forecast error is likely to be.
    • Statistical analysis of the variability of demand around the average demand provides the basis for this error estimate.
    • Error estimates should also be given in terms of the monetary value of the error so that the errors with the most dollars at risk can be addressed first.
  • Forecasts are more accurate for groups than for single items
    • Accuracy generally increases with the size of a product group, assuming that forecasts for each item in the group are as likely to be too high as too low. The low forecasts tend to balance out the high forecasts, at least in sizable groups.
    • (Definition) Mix Forecast is a forecast of the proportion of products that will be sold within a given product family, or the proportion of options offered within a product line, even though the appropriate level of units is forecasted for a given product line, an inaccurate mix forecast can create material shortages and inventory problems.
    • The general principle at work in these cases is risk pooling – taking individual risks and combining them into a pool. The overall risk for the pool tends to be less than the average of all the risks that flow into the pool.
  • Forecasts of near-term demand are more accurate than long-term forecasts
    • The further you extend your forecast into the future, the more likely that chance and change will derail your estimates.
    • Hence, the need for periodic review and update of demand forecasts in comparison to actual results.
    • Long-term forecasts are generally reviewed on an annual or quarterly basis.
    • Medium-term forecasts are generally reviewed on monthly basis.
    • Short-term forecasts are generally reviewed on a weekly basis.
    • In addition to regular reviews, taking steps to shorten the required lead time for items can shorten the forecasting period and thus improve the accuracy of the forecasts.

Components of Demand

The core components of demand include the following:

  • Trend
    • Demand can stay the same, or it can rise or fall.
    • (Definition) A Trend is general upward or downward movement of a variable over time (for example: demand, process attributes).
  • Seasonality
    • Demand may fluctuate depending on time of the year, for example holidays, weather, or other seasonal events.
    • (Definition) Seasonality, also known as seasonal variation, is a repetitive pattern of demand from year to year (or other repeating time interval) with some periods considerably higher than others.
  • Random Variation
    • Many factors affect demand during specific time periods and occur on a random basis.
    • The size of this variation can usually be measured.
    • (Definition) Random Variation is a fluctuation in data that is caused by uncertain or random occurrences. These random changes are generally very short-term, mere bumps and dips on the road up or down a trend line.
  • Cycle
    • Long-term upward and downward cyclical moves generally correlate with the business cycle, but the duration of these economic trends is difficult to predict and is therefore generally left to economists.
    • Publications such as the Purchasing Managers Index (PMI) can be used to predict economic trends.

The difference between seasonality and cycles can be clarified as follows:

  • Seasonality is a demand pattern that, based on history, will repeat itself on calendar basis such as month, week, day of the week, hour of the day, etc., and therefore can be predicted.
  • Cycles are demand patterns that repeat but follow a wavelike pattern that can span multiple years and therefore cannot be predicted easily.

Independent and Dependent Demand

  • Demand can be classified into two types:
    • Dependent
    • Independent
  • Demand for finished product is independent; demand for a component used in making the product is dependent.
  • (Definition) Dependent Demand that is directly related to or derived from the bill of material structure for other items or end products. Such demands are therefore calculated and need not and should not be forecast.
  • (Definition) Independent Demand is the demand for an item that is unrelated to the demand for other items. Demand for finished goods, parts required for destructive testing, and service parts requirements are examples of independent demand.
  • Forecasting should be done only for the independent demand; dependent demand can then be calculated from the forecast using material requirements planning.
  • A given item, however, may be subject to both types of demand. For example, the demand for automobile tires is dependent in relation to new cars but independent when considered as replacement item to be stocked in a repair shop.