Interactive explainer

Gridworld: reinforcement learning you can explore

A small world with a goal, traps and walls: watch value iteration work backwards from the goal, and a Q-learning agent discover the same world through trial and error.

Interactive

Gridworld: reinforcement learning you can explore

This explainer requires JavaScript. Each grid cell is a state; the agent can move up, right, down or left. The Bellman equation connects a cell’s value to the values of its neighbours.

How the explainer works

The world is a Markov decision process with a 6 × 5 grid. In each cell, the agent chooses one of four directions. Hitting a wall or boundary leaves it in place. Each step costs 0.1, the goal yields +10 and a trap −10; the goal and traps end the episode. Two traps interrupt the bottom row between start and goal, so the shortest route runs directly alongside them.

On slippery ground, an action moves in the chosen direction with only 80% probability, or sideways with 10% probability on each side. Every step next to a trap then carries risk.

Value iteration knows the transitions P(s′∣s,a)P(s' \mid s, a) and rewards r(s,a,s′)r(s, a, s'). Each sweep recalculates every cell:

Vk+1(s)=max⁡a∑s′P(s′∣s,a) [ r(s,a,s′)+γ Vk(s′) ]V_{k+1}(s) = \max_{a} \sum_{s'} P(s' \mid s, a)\,\bigl[\,r(s, a, s') + \gamma\, V_k(s')\,\bigr]

It stops once no value changes by more than 0.0001. Arrows indicate actions achieving the maximum; the outline marks the route obtained by following them from the start.

Q-learning does not know the model. The agent chooses a random direction with probability ε\varepsilon, otherwise the direction with the largest Q-value, and adjusts its estimate after each step:

Q(s,a)←Q(s,a)+α [ r+γmax⁡a′Q(s′,a′)−Q(s,a) ]Q(s, a) \leftarrow Q(s, a) + \alpha\,\bigl[\,r + \gamma \max_{a'} Q(s', a') - Q(s, a)\,\bigr]

Cells show max⁡aQ(s,a)\max_a Q(s, a). The chart shows each episode’s return, the sum of its rewards, and a moving average.

Three things to try

  1. Make it slippery. Enable “Slippery”. The best route moves one row upwards: two extra steps cost less than the risk of slipping into a trap. The starting cell’s value falls from 4.85 to 2.55.
  2. Short or long horizon. Lower γ\gamma to 0.5. Distant rewards barely count; values near the start become close to zero and spread only weakly through the grid.
  3. Learning with risk. In Q-learning mode, run 100 episodes several times. The agent learns the short route along the edge, but exploration with ε=0,2\varepsilon = 0{,}2 still sends it into a trap in roughly one episode out of seven. Later, setting ε=0\varepsilon = 0 prevents this exploration risk, but it may remain on a detour because it no longer tries anything new.

The reinforcement learning explained article covers Markov decision processes, the Bellman equation and temporal-difference learning.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

AI workflows at NyxAI

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.