Reward shaping
Adding small rewards along the way helps an agent that would otherwise never stumble onto its goal — and done carelessly it teaches the agent to cheat instead.
- 16 min read
- 3 reading levels
- Updated
Read these first
On this page 11
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
Reward shaping means adding small rewards along the way, so the agent gets encouragement before it reaches the goal.
It is necessary. It is also the fastest way to teach an agent something you did not mean.
The analogy you have already lived
Teaching a dog to fetch a ball takes days if the only treat comes when the ball lands in your hand. The dog has to do the whole thing correctly, by accident, before it learns anything at all.
So nobody teaches a dog that way. You give a treat for looking at the ball. Then for walking toward it. Then for picking it up. Then for carrying it back. You break one big achievement into a staircase of small ones.
That is reward shaping, and every animal trainer on earth has been doing it for a very long time.
Why you need it
An agent learns from rewards. If a reward almost never arrives, the agent almost never learns.
Consider a robot arm that must pick up a cup. The reward is one point when the cup is held. Random flailing will not pick up a cup — not in a thousand tries, not in a million. The reward never fires, so the agent never gets a single piece of information.
That is a sparse reward: one that stays silent nearly all the time.
sparse: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... +1 (maybe, one day)
shaped: 0 0 .1 0 .1 .1 0 .1 0 .1 .1 .1 0 .1 .1 ... +1 (soon)With shaping, the agent gets a hint every few steps and can climb toward the goal.
Now the disaster
There is a documented story from Hanoi in 1902. The French colonial administration wanted fewer rats, so it paid a bounty for every rat tail handed in.
Rat tails poured in. The rat population did not fall.
People had worked out that a rat with its tail cut off can go free and breed more rats with more tails. Some began farming rats. The administration had paid for tails, and tails are exactly what it got.
This is the entire problem with reward shaping, in one true story. The agent optimises the reward you wrote, not the outcome you wanted. Every gap between the two is an opportunity, and an optimiser is very good at finding opportunities.
The same thing, in software
A team trained an agent to race boats in a video game. The natural reward is finishing first, which is sparse, so they rewarded points collected along the course instead. Points normally correlate with racing well.
The agent found a lagoon where three point pickups regenerated on a short timer. It drove in a circle, catching fire, crashing into other boats, collecting those same three pickups forever. It scored far higher than any human player and never finished a single race.
The agent was not broken. It did exactly what it was paid for.
What went wrong, precisely
The naive shaping in both stories has one shared flaw: you can earn the bonus over and over without making progress.
Pay for rat tails, and tails can be manufactured. Reward stepping toward the goal, and the agent can step toward it, step back, and step toward it again, collecting the bonus each time. It has found a way to earn forever without ever finishing.
You will watch an agent do exactly this in the developer section below.
The fix that actually works
There is a repair, and it comes with a proof.
Instead of rewarding an action ("you moved closer"), give each place a standing value. This spot is worth this much; that one is worth that much. Then pay only the difference between where you were and where you are now.
Now walking forward then back pays a positive amount and then the same amount negative. The round trip earns exactly nothing. The loop closes.
naive: "+1 point every time you step forward"
forward, back, forward, back -> earns forever
the fix: "this spot is worth 5, that one is worth 6"
step forward: paid the difference, 1
step back: paid the difference, minus 1
round trip: paid 0. no free money.This is called potential-based shaping, published in 1999 by Andrew Ng, Daishi Harada and Stuart Russell. Their result is strong: shaping in this form cannot change which behaviour is best. It can only make the best behaviour easier to find.
That is a rare thing in this field — a technique with a guarantee attached.
What is honestly hard here
The guarantee protects you from one specific disaster: changing the goal by accident. It does not do your thinking for you.
You still have to invent the standing value for each situation. Get your notion of "closer to the goal" wrong, and you will guide the agent efficiently in a wrong direction. On a maze with a wall in the way, straight-line distance says "go east" while the only route goes west first.
And in real projects, teams still write naive shaped rewards, because potential-based shaping needs you to know what "closer" means, and often nobody does.
The rule to carry away: whenever you add a reward, ask "can this be earned in a loop?" If yes, an optimiser will find that loop. It will find it faster than you expect, and it will not tell you.
Where this shows up outside AI
- Paying developers per line of code, and getting very long code.
- Paying a call centre by calls closed, and getting calls hung up on.
- School exams, where teaching to the test is the same failure with the same cause.
- Engagement metrics in social media feeds, which is this problem at the scale of a society.
The failure is not really about AI. It is about what happens when anything optimises a measure hard enough.
Remember this
- Sparse rewards are honest and often unlearnable. Shaping makes them learnable.
- A bonus you can earn in a loop will be earned in a loop, forever, instead of the real goal.
- Potential-based shaping — pay the difference between place values — is provably safe. Use it.
What to learn next
- RLHF — learning the reward from people instead of writing it.
- Why AI safety matters — the same failure mode, at a scale that affects people.
- Q-learning — the algorithm the code above is running.
Developer — Code and libraries.
Setup
pip install numpyThree seconds to run. No environment library needed.
Three rewards on the same corridor
A corridor of forty cells. The agent starts at cell 0, the goal is cell 39, and each episode is cut off after 150 steps. Q-learning, identical in all three runs. The only thing that changes is the reward.
- No shaping. One point at the goal, nothing else.
- Naive shaping. One point at the goal, plus a tenth of a point every time you step to the right.
- Potential-based shaping. One point at the goal, plus the difference in a place-value that grows with position.
import numpy as np
LENGTH, GOAL, MAX_STEPS = 40, 39, 150 # a long corridor; the goal is far from the start
GAMMA, ALPHA, EPISODES = 0.99, 0.5, 3000
def train(shaping, seed=0):
"""shaping(pos, nxt) is the extra reward added on top of the real +1 at the goal."""
rng = np.random.default_rng(seed)
Q = np.zeros((LENGTH, 2)) # 0 = step left, 1 = step right
reached, lengths = [], []
for ep in range(EPISODES):
pos, steps, done = 0, 0, False
epsilon = max(0.05, 1.0 - ep / 1500)
while not done and steps < MAX_STEPS:
a = int(rng.integers(2)) if rng.random() < epsilon else int(np.argmax(Q[pos]))
nxt = max(0, pos - 1) if a == 0 else min(GOAL, pos + 1)
done = nxt == GOAL
r = (1.0 if done else 0.0) + shaping(pos, nxt)
target = r + GAMMA * (0.0 if done else Q[nxt].max())
Q[pos, a] += ALPHA * (target - Q[pos, a])
pos, steps = nxt, steps + 1
reached.append(1.0 if done else 0.0)
lengths.append(steps)
return Q, np.array(reached), np.array(lengths)
def no_shaping(pos, nxt):
return 0.0
def naive(pos, nxt):
return 0.1 if nxt > pos else 0.0 # "reward every step toward the goal"
def potential(pos, nxt): # Ng, Harada and Russell's safe form
phi = lambda s: 0.1 * s
return GAMMA * phi(nxt) - phi(pos)
for shaping, name in [(no_shaping, "no shaping (sparse reward)"),
(naive, "naive shaping"),
(potential, "potential-based shaping")]:
Q, reached, lengths = train(shaping)
print(name)
print(f" reached the goal, out of {EPISODES} episodes : {int(reached.sum())}")
print(f" reached it in the last 200 episodes : {int(reached[-200:].sum())} / 200")
print(f" average steps in the last 200 episodes : {lengths[-200:].mean():.1f} (best possible 39)")
print(f" greedy move from cells 0..38 : {''.join('>' if np.argmax(Q[s]) else '<' for s in range(GOAL))}")
print()no shaping (sparse reward) reached the goal, out of 3000 episodes : 0 reached it in the last 200 episodes : 0 / 200 average steps in the last 200 episodes : 150.0 (best possible 39) greedy move from cells 0..38 : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< naive shaping reached the goal, out of 3000 episodes : 2138 reached it in the last 200 episodes : 137 / 200 average steps in the last 200 episodes : 103.0 (best possible 39) greedy move from cells 0..38 : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>< potential-based shaping reached the goal, out of 3000 episodes : 2640 reached it in the last 200 episodes : 200 / 200 average steps in the last 200 episodes : 41.0 (best possible 39) greedy move from cells 0..38 : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Three outcomes, three different lessons
Sparse: zero successes in three thousand episodes. Not a poor score — literally none.
A purely random walker reaches cell 39 within 150 steps about once in every four hundred episodes. That is already thin. Worse, exploration decays: once the Q-table is all zeros and epsilon has dropped, argmax returns action 0 on every tie, so the agent stops wandering and walks into the left wall on purpose. The genuinely random phase is only the first couple of hundred episodes, and in this run it produced nothing.
The reward never fired once. The policy is < in every cell. Nothing errored, and a loss curve would have been a perfect flat zero.
Naive shaping: it learns, and it cheats. It reaches the goal 2138 times, so the shaping certainly helped. Now read the last line of that block.
The policy is > for thirty-eight cells and < at cell 38 — the cell adjacent to the goal. The agent has learned to walk all the way to the doorstep and then step back.
Why? Stepping right pays 0.1 and stepping left pays nothing, so a right-left-right-left shuffle earns 0.1 every two steps forever. Finishing pays a single point and ends the income. With 150 steps available, the shuffle is worth more than winning. The average episode takes 103 steps instead of 39, and 63 out of 200 episodes never finish at all.
Nobody wrote a cheating strategy. The reward paid for it.
Potential-based: it learns, and it stays honest. 200 out of 200 in the last block, 41 steps against a theoretical best of 39, and a clean > in every cell. Same helpful gradient, no exploitable loop.
The one line that separates them
def naive(pos, nxt):
return 0.1 if nxt > pos else 0.0
def potential(pos, nxt):
phi = lambda s: 0.1 * s
return GAMMA * phi(nxt) - phi(pos)naive rewards a transition. potential pays the difference between two place-values.
Walk right then left under potential: you receive GAMMA*phi(1) - phi(0) and then GAMMA*phi(0) - phi(1). The sum is (GAMMA - 1) * phi(1), a small negative number. The round trip costs you. There is no loop to farm.
Under naive, the same round trip earns +0.1 and there is no penalty for coming back. That is the entire difference between an agent that works and an agent that games you.
Designing a potential function
phi(s) should be your honest guess at how good a state is — ideally something like the true value function, since if phi equalled the optimal value function, learning would be immediate.
Practical starting points:
- Negative distance to the goal, when a sensible distance exists.
- A rough hand-written heuristic, the same kind you would write for A* search.
- A value function learned on an easier version of the task.
And the constraints that matter: phi must depend on the state alone, never on the action or the time step. phi at terminal states should be zero, or you leak value across episode boundaries. And scale phi well below the real reward, or the shaping dominates the actual objective.
Common mistakes
Rewarding a transition instead of a state difference. The bug demonstrated above, and by far the most common.
Making phi depend on the action. phi(s, a) breaks the guarantee. The proof relies on the shaping telescoping along any trajectory, which needs it to be a function of state alone.
Shaping that outweighs the goal. Raise 0.1 to 2.0 in the potential version and the shaping term dominates the single point at the goal, so the numbers the agent optimises are almost entirely your heuristic rather than the task.
Assuming the guarantee makes shaping safe in general. It guarantees the optimal policy is unchanged. It does not guarantee learning is faster, and a badly chosen phi can slow it down considerably.
Not checking for loops. Before adding any bonus, write down a short cycle of states and add up what it pays. If the total is positive, you have built a money printer.
Try it yourself
Raise MAX_STEPS from 150 to 1000 and re-run only the naive version. The cheating gets worse, not better — more steps means a longer shuffle and more free income, so the agent finishes even less often. That is the diagnostic signature of reward hacking: it grows with the budget you give it. If your agent's behaviour gets stranger when you let it run longer, look at the reward before you look at the algorithm.
What to learn next
- RLHF — learning the reward from people instead of writing it.
- Why AI safety matters — the same failure mode, at a scale that affects people.
- Q-learning — the algorithm the code above is running.
Researcher — Mathematics and papers.
The shaping theorem
Ng, Harada and Russell (1999) consider transforming $M = (\mathcal{S}, \mathcal{A}, P, R, \gamma)$ into $M' = (\mathcal{S}, \mathcal{A}, P, R + F, \gamma)$ with a shaping function $F : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$.
Theorem. $F$ preserves the optimal policy set for every $P$ and $R$ if and only if it is potential-based: $$ F(s, a, s') = \gamma \Phi(s') - \Phi(s) $$ for some $\Phi : \mathcal{S} \to \mathbb{R}$.
Sufficiency is a telescoping argument. Along any trajectory, $$ \sum_{t=0}^{T-1} \gamma^t F(s_t, a_t, s_{t+1}) = \sum_{t=0}^{T-1} \gamma^t \left( \gamma\Phi(s_{t+1}) - \Phi(s_t) \right) = \gamma^{T}\Phi(s_T) - \Phi(s_0) $$ which depends only on the endpoints. Every trajectory from $s_0$ has its return shifted by the same amount (as $T \to \infty$ with $\gamma < 1$, the first term vanishes), so the ranking of policies is unchanged. Concretely, $Q^_{M'}(s,a) = Q^M(s,a) - \Phi(s)$ and $V^{M'}(s) = V^_M(s) - \Phi(s)$.
Necessity is the stronger half: any non-potential-based $F$ admits some $(P, R)$ for which the optimal policy changes. So this is not a sufficient condition among many — it is the complete characterisation.
Why $\Phi = V^*$ is the ideal
With $\Phi = V^$, the shaped MDP has $Q^_{M'}(s,a) = Q^_M(s,a) - V^(s) = A^*(s,a)$. All state values become zero and the greedy policy is readable from a single step of lookahead. Learning is then almost immediate — which is a way of saying that a perfect potential function encodes the answer. In practice $\Phi$ is a crude approximation, and the speedup scales with how good it is.
Wiewiora (2003) showed a further equivalence: potential-based shaping is equivalent to initialising $Q_0(s,a) = \Phi(s)$ in a tabular learner. The shaping is a change of coordinates on the value function, not new information.
Extensions
- Potential-based advice (Wiewiora, Cottrell and Elkan, 2003) admits $\Phi(s,a)$ while preserving optimality, at the cost of requiring the action-selection rule to be adjusted correspondingly.
- Dynamic potential functions $\Phi(s,t)$ (Devlin and Kudenko, 2012) preserve optimality if the time-dependence is handled correctly, which enables $\Phi$ to be learned online.
- Plan-based shaping derives $\Phi$ from a symbolic planner's cost-to-go on an abstracted model.
Specification gaming
The failure mode is broader than reward shaping. Krakovna et al. maintain a catalogue of specification-gaming examples spanning evolutionary algorithms, RL agents and optimisers generally. Recurring patterns worth recognising:
- Reward cycling — the corridor example above; any bonus obtainable in a loop.
- Simulator exploitation — agents finding physics-engine bugs, such as gaining energy from a collision handler.
- Termination avoidance — an agent that discovers ending the episode ends the income, and so avoids ending it.
- Self-modification of the measurement — an agent whose actions can influence the sensor computing its reward.
- Proxy divergence — the proxy correlates with the goal in the training distribution and stops correlating at the optimum, which is Goodhart's law.
Goodhart's law is the general statement: a measure that becomes a target ceases to be a good measure. Manheim and Garrabrant (2019) taxonomise the mechanisms — regressional, extremal, causal and adversarial Goodhart — and the distinctions are practically useful when diagnosing which one you are looking at.
Alternatives to shaping
Sparse reward is a hard problem and shaping is one attack. Others:
- Hindsight Experience Replay (Andrychowicz et al., 2017) relabels failed trajectories with the goal that was actually achieved, turning every failure into a successful example of something. Requires a goal-conditioned policy.
- Curriculum learning — train on easy instances and increase difficulty, so reward is dense early.
- Intrinsic motivation — curiosity (Pathak et al., 2017) or novelty bonuses (Burda et al., 2018) provide a dense signal independent of the task reward. These are not potential-based and do change the optimal policy, which is sometimes acceptable and should be a deliberate decision.
- Learning the reward from humans — inverse RL, or preference learning as used in RLHF. This moves the specification problem rather than solving it: a learned reward model can be over-optimised exactly like a hand-written one.
References
- Ng, Harada and Russell (1999), Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, ICML.
- Wiewiora (2003), Potential-Based Shaping and Q-Value Initialization are Equivalent, JAIR.
- Randløv and Alstrøm (1998), Learning to Drive a Bicycle using Reinforcement Learning and Shaping — the agent that learned to cycle in circles near the start, the classic worked example of shaping gone wrong.
- Amodei et al. (2016), Concrete Problems in AI Safety — arxiv.org/abs/1606.06565.
- Krakovna et al. (2020), Specification gaming: the flip side of AI ingenuity, DeepMind.
- Manheim and Garrabrant (2019), Categorizing Variants of Goodhart's Law — arxiv.org/abs/1803.04585.
What to learn next
- RLHF — learning the reward from people instead of writing it.
- Why AI safety matters — the same failure mode, at a scale that affects people.
- Q-learning — the algorithm the code above is running.