Interactive explainer
Gridworld: reinforcement learning you can explore
A small world with a goal, traps and walls: watch value iteration work backwards from the goal, and a Q-learning agent discover the same world through trial and error.
Gridworld: reinforcement learning you can explore
This explainer requires JavaScript. Each grid cell is a state; the agent can move up, right, down or left. The Bellman equation connects a cell’s value to the values of its neighbours.
How the explainer works
The world is a Markov decision process with a 6 × 5 grid. In each cell, the agent chooses one of four directions. Hitting a wall or boundary leaves it in place. Each step costs 0.1, the goal yields +10 and a trap −10; the goal and traps end the episode. Two traps interrupt the bottom row between start and goal, so the shortest route runs directly alongside them.
On slippery ground, an action moves in the chosen direction with only 80% probability, or sideways with 10% probability on each side. Every step next to a trap then carries risk.
Value iteration knows the transitions and rewards . Each sweep recalculates every cell:
It stops once no value changes by more than 0.0001. Arrows indicate actions achieving the maximum; the outline marks the route obtained by following them from the start.
Q-learning does not know the model. The agent chooses a random direction with probability , otherwise the direction with the largest Q-value, and adjusts its estimate after each step:
Cells show . The chart shows each episode’s return, the sum of its rewards, and a moving average.
Three things to try
- Make it slippery. Enable “Slippery”. The best route moves one row upwards: two extra steps cost less than the risk of slipping into a trap. The starting cell’s value falls from 4.85 to 2.55.
- Short or long horizon. Lower to 0.5. Distant rewards barely count; values near the start become close to zero and spread only weakly through the grid.
- Learning with risk. In Q-learning mode, run 100 episodes several times. The agent learns the short route along the edge, but exploration with still sends it into a trap in roughly one episode out of seven. Later, setting prevents this exploration risk, but it may remain on a detour because it no longer tries anything new.
The reinforcement learning explained article covers Markov decision processes, the Bellman equation and temporal-difference learning.