Learn AI Create the Future — with Pranay
Main site

Retail, Demand and Supply Chain

Predict then optimise

The most accurate forecast is not always the one that leads to the best decision, because some forecasting mistakes cost far more than others once you act on them.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A forecast is only useful once it drives a decision. The most accurate forecast is not always the one that leads to the best decision.

Think about a wedding caterer estimating how many guests will actually eat. Getting the total headcount slightly wrong barely matters for the rice and dal, which are cheap and always used up. Getting it wrong for the expensive mutton dish matters a great deal. Too little, and honoured guests go without. Too much, and money is wasted on food nobody eats.

A caterer who spreads their guessing effort evenly across every dish is not doing their job as well as one who is careful. The careful one focuses on the expensive dish, even if that means being a bit sloppier about the rice.

Why it exists

Every earlier lesson in this section ends the same way: forecast, then decide. That decision might be how much to order, how to price, or how to split stock across stores. Usually those two steps are built separately. A forecasting team builds a model to be as accurate as possible, on average, across every product. A separate planning team then feeds those forecasts into an ordering or allocation decision.

This split works fine, until resources run short. A delivery truck can only carry so much. A warehouse only has so much shelf space. When something has to be left behind, the decision is forced to pick which product's forecast error you can afford to be wrong about. A forecast trained to be equally accurate on every product has no idea some errors are far more expensive than others.

The fix is to connect the two steps: build or evaluate the forecast with the actual downstream decision in mind, not a generic accuracy score.

How it works

STEP 1: forecast demand for three products
STEP 2: load a truck that can only carry 250 units total, most valuable product first

A forecast that is "accurate on average" across all three products
    can still be badly wrong on the one product where being wrong is expensive —
    and only mildly wrong on the one product where nobody cares.

The result: two forecasts with very different overall accuracy
    can lead to very different profit, once you act on them.

This is the general shape behind turning a forecast into an order quantity. That earlier lesson showed this for a single product, with a known cost of over- and under-ordering. This lesson is the same idea, scaled up to many products competing for one shared, limited resource.

Where you have already seen it

  • An airline overbooking a flight. The forecast that matters is not "how many passengers will show up on average." It is "how many can I overbook without too often bumping a paying customer."
  • A hospital's operating theatre schedule. A forecasting error on a routine, low-cost procedure barely matters. The same-sized error on a complex, high-cost surgery can cascade through the whole day's schedule.
  • A delivery company loading vans each morning. Space is limited, so an error on a bulky, low-value item and an error on a compact, high-value item are not equally costly.

Remember this

  • A forecast is a means to a decision, not the end goal by itself.
  • When resources are limited, some forecasting mistakes cost far more than others.
  • A model built to be accurate everywhere, on average, can still make worse decisions than a less "accurate" model that protects the expensive mistakes.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy pandas

Minimal runnable code

Three products, one truck, limited capacity. We compare two demand forecasts with very different overall accuracy, and check which one actually leads to more profit once the truck gets loaded.

predict_then_optimise.py
import numpy as np
import pandas as pd

products = pd.DataFrame({
    "product": ["premium_ghee", "cooking_oil", "salt"],
    "margin": [40, 12, 2],          # profit per unit sold, in rupees
    "true_demand": [60, 150, 300],  # what actually gets bought, revealed only after the fact
})

truck_capacity = 250   # the truck cannot carry more than this many units combined


def allocate(forecast_demand, capacity, margins):
    """Greedy: fill the truck starting with the highest-margin product first."""
    order = pd.Series(0.0, index=forecast_demand.index)
    remaining = capacity
    for i in margins.sort_values(ascending=False).index:
        take = min(forecast_demand[i], remaining)
        order[i] = take
        remaining -= take
    return order


def realized_profit(order, true_demand, margins):
    sold = np.minimum(order, true_demand)   # cannot sell more than customers actually want
    return (sold * margins).sum()


true_demand = products["true_demand"]

# Model A: "accurate on average" -- a small, ordinary-looking error on every product,
# including a modest 15-unit underestimate of the high-margin ghee.
forecast_generic = pd.Series([45, 155, 290])

# Model B: near-perfect on the two products that actually decide the truck's loading
# order, and badly wrong on salt -- the lowest-margin item, ordered last with whatever
# capacity happens to be left.
forecast_decision_aware = pd.Series([60, 150, 200])

for name, fc in [("generic (small error everywhere)", forecast_generic),
                  ("decision-aware (error dumped onto the cheap item)", forecast_decision_aware)]:
    mse = ((fc - true_demand) ** 2).mean()
    print(f"{name}:")
    print(f"  forecast = {fc.tolist()}, true demand = {true_demand.tolist()}, MSE = {mse:.1f}")
print()

oracle_order = allocate(true_demand, truck_capacity, products["margin"])
generic_order = allocate(forecast_generic, truck_capacity, products["margin"])
decision_order = allocate(forecast_decision_aware, truck_capacity, products["margin"])

print("truck loading plan (ghee, oil, salt), highest margin loaded first:")
print("  oracle (perfect foresight):     ", oracle_order.tolist())
print("  generic forecast's plan:        ", generic_order.tolist())
print("  decision-aware forecast's plan: ", decision_order.tolist())
print()

profit_oracle = realized_profit(oracle_order, true_demand, products["margin"])
profit_generic = realized_profit(generic_order, true_demand, products["margin"])
profit_decision = realized_profit(decision_order, true_demand, products["margin"])

print(f"realized profit, oracle plan:            Rs.{profit_oracle:,.0f}")
print(f"realized profit, generic forecast plan:  Rs.{profit_generic:,.0f}  (lost Rs.{profit_oracle - profit_generic:,.0f} vs oracle)")
print(f"realized profit, decision-aware plan:    Rs.{profit_decision:,.0f}  (lost Rs.{profit_oracle - profit_decision:,.0f} vs oracle)")
Output
generic (small error everywhere):
  forecast = [45, 155, 290], true demand = [60, 150, 300], MSE = 116.7
decision-aware (error dumped onto the cheap item):
  forecast = [60, 150, 200], true demand = [60, 150, 300], MSE = 3333.3

truck loading plan (ghee, oil, salt), highest margin loaded first:
  oracle (perfect foresight):      [60.0, 150.0, 40.0]
  generic forecast's plan:         [45.0, 155.0, 50.0]
  decision-aware forecast's plan:  [60.0, 150.0, 40.0]

realized profit, oracle plan:            Rs.4,280
realized profit, generic forecast plan:  Rs.3,700  (lost Rs.580 vs oracle)
realized profit, decision-aware plan:    Rs.4,280  (lost Rs.0 vs oracle)

What actually happened

Look at the two MSE numbers first. The decision-aware forecast is nearly 30 times worse by plain mean squared error — 3333.3 against 116.7. By any ordinary accuracy scoreboard, the generic forecast wins comfortably.

Then look at the profit. The decision-aware forecast matches the perfect-foresight oracle exactly, Rs 4,280. The "more accurate" generic forecast loses Rs 580, because its modest 15-unit underestimate of ghee — the highest-margin product — caused the truck to load 15 fewer units of the one product where that mistake was expensive.

The decision-aware forecast dumped almost all of its error onto salt, the lowest-margin product. That error barely mattered, because salt is loaded last, from whatever capacity happens to be left over, and true salt demand (300) is so far above that leftover capacity that being somewhat wrong about it changes nothing about how much gets loaded.

allocate() is a greedy algorithm — it makes the locally best choice (load the highest-margin item first) at each step, without reconsidering earlier choices. That single design detail is exactly why forecast errors on different products are not equally costly here.

Common mistakes

Reporting only an aggregate accuracy metric to the business. MSE or MAPE averaged across every product hides exactly the pattern this example demonstrates. Always ask "which errors actually change a decision?" before trusting an aggregate score.

Assuming a lower-error model is automatically the better one to ship. It is the better one to ship only if the downstream decision does not have this kind of capacity constraint or asymmetric cost structure.

Optimising the forecast and the allocation as two completely separate projects, run by two separate teams that never compare notes. The margin numbers and the capacity constraint need to reach whoever is building or evaluating the forecast.

Ignoring this for genuinely unconstrained decisions. If capacity were unlimited here, every product's forecast would be equally consequential, and plain accuracy would be a perfectly fine target. This pattern matters specifically because of the shared, limited resource.

Try it yourself

Set truck_capacity = 600 — enough to carry the full true demand of all three products. Re-run, and check that both forecasts now reach the same, oracle-matching profit. The capacity constraint was doing all the work in the original example.

What to learn next

Researcher — Mathematics and papers.

Formalising "predict, then optimise"

The general setting: a downstream optimisation problem

z*(c) = argmin_{x in Feasible}  c^T x
  • x — the decision variable (here, the order quantities)
  • c — the true cost or profit coefficients, only known after the fact
  • Feasible — the constraint set (here, the truck capacity)

In the standard two-stage pipeline, a predictive model produces c_hat, an estimate of c, trained to minimise a generic loss like squared error, sum (c_hat_i - c_i)^2. The decision actually taken is x_hat = argmin_{x in Feasible} c_hat^T x, and its quality is measured by regret:

Regret(c_hat) = c^T x_hat  -  c^T x*(c)

The developer example instantiates exactly this: c is realised demand, c_hat is the forecast, and regret is profit_oracle - profit_forecast. The core empirical finding — a forecast with much higher squared-error loss can produce zero regret, while a forecast with much lower squared-error loss produces large regret — is the entire motivation for the "smart predict, then optimise" literature.

Smart Predict-then-Optimise (SPO)

Elmachtoub and Grigas (2022), Smart "Predict, then Optimize", Management Science, formalise a loss function that trains the predictive model directly against downstream decision quality rather than prediction accuracy:

SPO_loss(c_hat, c) = c^T x*(c_hat)  -  c^T x*(c)

This is exactly regret, used as a training loss rather than only an evaluation metric. Because x*(c_hat) involves solving an optimisation problem (often piecewise-constant in c_hat), the SPO loss is not differentiable in the ordinary sense; Elmachtoub and Grigas derive a convex surrogate, SPO+, that upper-bounds it and admits a subgradient, making gradient-based training against decision quality tractable for linear and some combinatorial optimisation problems.

Differentiable optimisation layers

A parallel line of work makes the optimisation step itself part of a differentiable computation graph, so a neural network can be trained end to end against decision quality. Amos and Kolter (2017), OptNet: Differentiable Optimization as a Layer in Neural Networks, and Wilder, Dilkina and Tambe (2019), Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization, both implicitly differentiate through the optimality (KKT) conditions of the downstream problem, back-propagating a gradient of decision quality into the upstream forecasting model's weights. This family of methods is often called decision-focused learning.

Where two-stage prediction is still the right call

None of this implies the classical two-stage pipeline is always wrong. Decision-focused training adds real engineering cost: the downstream optimisation problem must be known, stable, and differentiable (or have a usable surrogate) at training time. When the same forecast feeds many different downstream decisions with different cost structures — a common situation in a large retailer — a single decision-focused model tied to one specific optimisation is a poor fit, and a well-calibrated, generically accurate forecast, paired with per-decision post-processing, remains the more maintainable architecture.

Key references

  • Elmachtoub, A. & Grigas, P. (2022). Smart "Predict, then Optimize." Management Science 68(1).
  • Wilder, B., Dilkina, B. & Tambe, M. (2019). Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization. AAAI.
  • Amos, B. & Kolter, J. Z. (2017). OptNet: Differentiable Optimization as a Layer in Neural Networks. ICML.

What to learn next