Machine learning & feedback

Reward design: what should an agent optimise?

An agent optimises its specified reward. Whether that calculation captures the intended outcome is a separate question. A small gridworld makes the distinction between model score and independent evaluation visible.

  • 4 min read
  • Updated on
  • smartlytics editorial team

The short answer

Reward design specifies which outcomes and transitions an agent should prefer. Missing costs can make unwanted side effects attractive. The reward should therefore be checked against a separate evaluation of the behaviour actually chosen.

Open interactive exampleFree · in your browser · no sign-up

At a glance

  1. A reward function is a model of the objective, not automatically the complete objective.
  2. A side effect must be represented in the state and evaluation before it can be accounted for.
  3. Independent evaluation can reveal that a high model score comes from unwanted behaviour.

Turning an intended outcome into a number

“Complete the task quickly” sounds clear. It still leaves open which damage, disruption or downstream costs are acceptable. Once a decision procedure maximises a number, that ambiguity must be resolved. A reward function assigns values to transitions or outcomes. The agent uses those values regardless of any additional expectations the human designer had in mind.

Unwanted side effects and exploiting incomplete objectives are recognised problems in learning-system design (Amodei et al. 2016). They do not require a particularly complex model. Even a tiny, fully known decision space can show how a reasonable intention becomes an unsuitable objective. Our example therefore separates optimisation from the evaluation performed afterwards.

A finite grid with a fragile object

The world contains three rows and three columns. The start is on the left of the middle row, and the goal is on the right. A fragile object occupies the central cell. Each move up, down, left or right costs one unit. Reaching the goal for the first time gives ten reward units and ends the episode.

Entering the object’s cell damages it once. The state therefore includes both position and whether damage has already occurred. Without this extra state component, the model could repeatedly charge for the same damage. The adjustable model penalty λ\lambda is deducted only when damage first occurs. For a route taking nn steps, the score is

G=10 1goal reached−n−λ 1damage.G=10\,\mathbf{1}_{\text{goal reached}}-n-\lambda\,\mathbf{1}_{\text{damage}}.

A maximum number of steps bounds the planning problem. A backward calculation considers every allowed next move for each position, damage state and remaining time. It finds the optimum of this finite model. It neither learns from experience nor uses a neural network.

A worked trade-off

The direct route takes two steps and damages the object. With no model penalty, it scores 10−2=810-2=8. The safe detour through the upper or lower row takes four steps and scores 10−4=610-4=6. Optimisation therefore initially prefers the route that causes damage.

A separate audit always evaluates the chosen route using damage costs of six units. Under that evaluation, the direct route scores only 10−2−6=210-2-6=2, while the detour still scores six. This second evaluation is not used for planning. It exposes the consequence omitted by the first objective.

With a model penalty of three, the direct route drops to five points. The detour wins with six. At a penalty of exactly two, they tie; a fixed search order displays the first route found. Stable output under a tie therefore does not imply a substantive preference. If the step limit is only two, there is no safe route that reaches the goal in time.

Change the decision yourself

Interactive

Reward design in a small gridworld

This gridworld calculates the optimal route for the chosen reward and separately evaluates it using fixed audit costs.

Start with no damage penalty and compare the model score with the audit score. Then increase the penalty to three. The route changes even though the environment, goal reward and movement costs stay the same. Next, reduce the available time to two steps. A question of preference becomes a restriction on the available possibilities.

The chart displays the calculated route. The text also reports steps, damage and whether the goal was reached. These separate quantities are more informative than a single overall score: they show whether an apparent improvement comes from greater benefit, lower costs or a changed definition of the objective.

Reward assumptions and limitations

All movements succeed, the model is known, and there is only one side effect. Real systems often have uncertain transitions, incomplete observations and multiple affected parties. The fixed audit penalty is itself an assumption, not a universal definition of correct behaviour. A useful audit model needs independent justification.

Additional rewards are not automatically harmless learning aids. Potential-based reward shaping can preserve the optimal policy under specific conditions (Ng et al. 1999). Our damage penalty, in contrast, is intended to change which policy is preferred. These purposes should remain distinct. When an action must be prohibited outright, an explicit constraint is clearer than hoping a sufficiently large penalty will always discourage it.

Reward as a reviewable process decision

For a practical process, record the objective, side effects and hard limits separately. Examine concrete behaviours: what could maximise the metric without meeting the underlying need? Which costs are missing? Which behaviour would remain unacceptable even with a high total score?

Reinforcement learning explains how a policy can be learned. Rules and optimisation shows how hard conditions are checked before optimisation. Both perspectives complement the question of which reward an agent should receive.

Frequently asked questions

Is the demo a trained reinforcement-learning agent?

No. It solves the known finite model exactly using dynamic programming. This isolates the effect of the reward function from training noise.

Is every extra reward harmless reward shaping?

No. Additional rewards can change the optimal policy. Potential-based shaping has a special policy-preservation property under appropriate assumptions; an arbitrary penalty does not.

Are the audit costs of six units objectively correct?

No. They are an explicitly specified second evaluation assumption. Real costs, limits and affected interests would need justification.

Sources and further reading

  1. PaperAmodei, D. et al. (2016): Concrete Problems in AI Safety. arXiv.
  2. PaperNg, A. Y., Harada, D. and Russell, S. (1999): Policy invariance under reward transformations: Theory and application to reward shaping. ICML 1999.

Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.