Interactive explainer
Reward design in a small gridworld
The short route damages an object. The safe route takes longer. Change the reward and inspect the exact planning result.
Reward design in a small gridworld
The gridworld calculates an optimal route and its side effect. Both results follow directly from the selected settings.
What is calculated
An agent starts on the left of the middle row in a 3×3 grid and wants to reach the right-hand cell. Every step costs one unit, and reaching the goal pays ten once. The central cell contains a fragile object. Entering it for the first time incurs the adjustable model penalty.
The procedure knows the world completely and calculates the best route using dynamic programming. This is not a training animation. Alongside the optimised model score, an audit score evaluates the same route with fixed damage costs of six units. The audit score does not influence route selection. This makes it possible to see whether the reward accounts for an important consequence.
Three experiments
- Missing costs: Set the damage penalty to zero and allow six steps. The direct route has model score eight but causes damage and receives audit score two.
- Include the side effect: Increase the penalty to three. The safe four-step route becomes preferable to the shortcut. Inspect the route, damage and both evaluations.
- Too little time: Restrict planning to two steps. The safe detour no longer fits the available time. A penalty cannot create that missing option.
Reading the result
The state records whether the object has already been damaged, so damage is charged only once. Equal scores are resolved using a fixed order. The example is deterministic; resetting restores the initial settings.
The article on Reward design explains the calculation, the limitations of rewards and the distinction between penalties and hard constraints.