Interactive explainer

Reward design in a small gridworld

The short route damages an object. The safe route takes longer. Change the reward and inspect the exact planning result.

Interactive

Reward design in a small gridworld

The gridworld calculates an optimal route and its side effect. Both results follow directly from the selected settings.

What is calculated

An agent starts on the left of the middle row in a 3×3 grid and wants to reach the right-hand cell. Every step costs one unit, and reaching the goal pays ten once. The central cell contains a fragile object. Entering it for the first time incurs the adjustable model penalty.

The procedure knows the world completely and calculates the best route using dynamic programming. This is not a training animation. Alongside the optimised model score, an audit score evaluates the same route with fixed damage costs of six units. The audit score does not influence route selection. This makes it possible to see whether the reward accounts for an important consequence.

Three experiments

  1. Missing costs: Set the damage penalty to zero and allow six steps. The direct route has model score eight but causes damage and receives audit score two.
  2. Include the side effect: Increase the penalty to three. The safe four-step route becomes preferable to the shortcut. Inspect the route, damage and both evaluations.
  3. Too little time: Restrict planning to two steps. The safe detour no longer fits the available time. A penalty cannot create that missing option.

Reading the result

The state records whether the object has already been damaged, so damage is charged only once. Equal scores are resolved using a fixed order. The example is deterministic; resetting restores the initial settings.

The article on Reward design explains the calculation, the limitations of rewards and the distinction between penalties and hard constraints.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

AI workflows at NyxAI

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.