Machine learning & feedback
Reinforcement Learning explained
Reinforcement Learning describes how a system learns through actions and feedback to make good decisions over many steps. The core concepts, the Bellman equation and Q-learning—with a small example you can calculate yourself.
At a glance
- Reinforcement Learning learns from rewards rather than correct answers: an agent acts, observes the consequences and gradually improves its strategy.
- The Bellman equation divides a decision’s value into its immediate reward and the discounted value of the next state. Value iteration and Q-learning build on this relationship.
- In practice, success depends on reward design, the amount of data, simulation quality and safety constraints—not on the algorithm alone.
Many decisions have effects beyond the present moment. Building inventory today leaves less capital tomorrow; a move in chess changes all the possibilities that follow. Reinforcement Learning is the branch of machine learning that deals with precisely such sequences of decisions: an agent learns from feedback how to act so that its total rewards are as large as possible over the long term (Sutton & Barto 2018).
Learning by acting
In supervised learning, every example comes with the correct answer. In Reinforcement Learning, no such answer is provided. The agent only learns how good or bad the consequences of its actions were—often many steps later. This creates two tasks that supervised methods do not face: determining which earlier actions deserve credit for a later reward, and trying actions whose consequences are not yet known.
Formally, this is a feedback loop. At each time , the agent observes a state , chooses an action , and receives a reward and the next state from the environment.
The model: a Markov decision process
The mathematical framework is the Markov decision process (MDP) (Puterman 1994). It consists of
- a set of states and a set of actions ,
- transition probabilities : how likely is action in state to lead to state ?
- a reward function , and
- a discount factor between 0 and 1.
“Markov” means that the current state contains everything that matters for the future. The history provides no additional relevant information.
The agent’s objective is not the next reward, but the return: the discounted sum of all future rewards:
A close to 0 makes the agent short-sighted; a close to 1 makes it far-sighted. Like an interest rate, expresses the idea that a reward today is worth more than the same reward later. Roughly speaking, rewards much further away than steps carry little weight—for , that is about ten steps.
The agent’s behavior is described by its strategy, the policy : the probability of choosing action in state . The goal is a policy that maximizes expected return.
Values and the Bellman equation
Two value functions make this search manageable. The state value is the expected return when starting in and then following policy . The action value is the expected return when first choosing action in and then following .
Richard Bellman showed that such values can be written recursively (Bellman 1957). For a fixed policy, the Bellman expectation equation is:
In words: a state’s value is the expected immediate reward plus the discounted value of what follows. For the best possible policy, averaging over actions is replaced by a maximum—the Bellman optimality equation:
Once the optimal values are known, the optimal policy is simple: choose the action that achieves the maximum in every state. The same equation exists for action values, . It is the basis of Q-learning.
An example to calculate yourself: the corridor
An agent stands in a corridor with three cells: A, B and C. To the right of C is the goal, which ends the episode. In each cell, the agent can move left or right; moving left from A leaves it in place. Reaching the goal gives a reward of 10; every other step gives 0. The discount factor is .
Value iteration solves the Bellman optimality equation by repeatedly applying it as an update rule. Start with for all cells and, in each iteration, recalculate every cell:
| Iteration | |||
|---|---|---|---|
| 0 | 0 | 0 | 0 |
| 1 | 0 | 0 | 10 |
| 2 | 0 | 9 | 10 |
| 3 | 8.1 | 9 | 10 |
| 4 | 8.1 | 9 | 10 |
In the first iteration, only C recognizes that moving right earns 10. In the second, B learns about it: to the right is C with value 10, discounted to . In the third, the information reaches A with . After that, nothing changes—the values have converged. They flow backwards from the goal through the state space, and is precisely the reward of 10 discounted twice by a factor of 0.9. The optimal policy is immediately apparent: move right everywhere.
A related method is policy iteration: it alternates between calculating the values of a fixed policy and improving that policy by choosing the best action in each state according to those values. Both methods belong to dynamic programming and assume that transitions and rewards are known (Puterman 1994).
Gridworld: reinforcement learning you can explore
The interactive explainer demonstrates value iteration and Q-learning in a larger world with a goal, traps and walls. Without JavaScript, the corridor example above still provides the complete calculation.
Learning without a model: TD learning and Q-learning
In most interesting applications, and are unknown. The agent must learn from experience: observed transitions of the form (state, action, reward, next state). Temporal-difference learning estimates values by moving each estimate a little towards a target that itself contains an estimate—the observed reward plus the estimated value of the next state (Sutton 1988).
Q-learning applies this idea to action values (Watkins & Dayan 1992). After each step, the value of the chosen action is updated:
The expression in brackets is the TD error: the difference between what experience suggests and the previous estimate. The learning rate determines how far to move towards it.
Q-learning has a remarkable property: it learns the values of the best policy even while the agent behaves differently—for example, because it is still exploring. This is called off-policy learning. In the tabular case, the estimates converge to if every state–action pair is visited infinitely often and the learning rate decreases appropriately (Watkins & Dayan 1992).
This also explains why exploration is necessary. An agent that always selects the action currently believed best never learns whether another action would have been better. The simplest solution is ε-greedy: usually choose the best known action, but select a random one with small probability . Multi-Armed Bandits address the same exploration–exploitation dilemma, without states and with more sophisticated solutions.
From tables to neural networks
Tables work only while every state can be enumerated. Even an image with a few pixels has more possible states than could ever be visited. A function then replaces the table, calculating values from features of the state—linearly or using a neural network. Combining this with deep networks is called Deep Reinforcement Learning. One milestone was the Deep Q-Network, which learned 49 Atari games directly from screen pixels and, among other techniques, stored and reused earlier experiences (Mnih et al. 2015).
A second major family optimizes the policy directly. Policy-gradient methods change a policy’s parameters in the direction that increases expected return; the classic starting point is the REINFORCE algorithm (Williams 1992). Actor–critic methods connect both approaches: one component learns the policy and another learns its values.
Board games show how well learning and planning can work together. AlphaZero combined a network trained through self-play with a tree search that looks ahead in each position, achieving very high playing strength in chess, shogi and Go (Silver et al. 2018). The search uses a model of the game rules—a luxury rarely available in real-world applications.
Which method fits which question?
Reinforcement Learning is powerful, but it is not always the best choice. A simpler method can often solve the same task with less data and more understandable reasoning:
| Situation | Natural approach |
|---|---|
| One-off decision; probabilities and consequences can be estimated | Decision theory with expected utility |
| Repeated choice between options, without subsequent states | Multi-Armed Bandit |
| Known model, many constraints, planning over a horizon | Optimization or dynamic programming |
| Consequences across many steps, unknown model, experience or simulation available | Reinforcement Learning |
Reinforcement Learning in practice
Several common hurdles stand between textbook examples and production use:
- Reward design. An agent optimizes exactly what is rewarded, not what is intended. If a reward is incompletely specified, it finds shortcuts that formally earn points while missing the real objective. This risk is called reward hacking and is a known safety problem in learning systems (Amodei et al. 2016).
- Data requirements. Many methods need millions of interactions. Agents therefore often learn in simulations—and then have to bridge the gap between simulation and reality.
- Learning from existing data. Offline Reinforcement Learning learns from recorded decisions without trying actions itself. This is safer but difficult, because the data contains only the actions that were actually chosen (Levine et al. 2020).
- Constraints and oversight. Real processes impose hard constraints, and bad decisions cost real money or more. Firm boundaries, gradual approvals, continuous monitoring and the ability to reverse decisions are useful safeguards.
Conclusion
Reinforcement Learning rests on a simple structure: states, actions, rewards and the Bellman equation connecting the value of a decision to the value of its consequences. Value iteration uses that equation when the model is known; Q-learning learns the same values from experience. The route into practice requires carefully designed rewards, realistic simulations and clear boundaries. For a deeper treatment, Sutton and Barto provide a comprehensive account (Sutton & Barto 2018).
Frequently asked questions
What is the difference between Reinforcement Learning and supervised learning?
In supervised learning, every example comes with a correct answer. In Reinforcement Learning, the agent receives only a reward for the consequences of its actions, often many steps later. It must work out which actions contributed to that reward and also try things it does not yet know.
What is Q-learning?
Q-learning estimates the long-term discounted reward for each state–action pair. After every step, the estimate moves a little towards the immediate reward plus the best estimated value of the next state. It does not require a model of the environment.
What does the Bellman equation say?
It says that a state’s value equals the expected immediate reward plus the discounted value of the next state. For the optimal strategy, the action with the highest value is selected in each state. Almost all Reinforcement Learning methods derive from this recursive relationship.
Does Reinforcement Learning need neural networks?
No. Small problems can be solved using tables that assign a separate value to every state or state–action pair. Neural networks or other function approximators are needed only when there are too many states for a table—for example, when images are used as inputs.
Sources and further reading
- BookSutton, R. S. and Barto, A. G. (2018): Reinforcement Learning: An Introduction (2nd edition). MIT Press.The standard textbook; the authors make a PDF version freely available.
- BookBellman, R. (1957): Dynamic Programming. Princeton University Press.
- BookPuterman, M. L. (1994): Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.
- PaperSutton, R. S. (1988): Learning to predict by the methods of temporal differences. Machine Learning 3(1), 9–44.
- PaperWatkins, C. J. C. H. and Dayan, P. (1992): Q-learning. Machine Learning 8(3–4), 279–292.
- PaperWilliams, R. J. (1992): Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8(3–4), 229–256.
- PaperMnih, V., Kavukcuoglu, K., Silver, D. et al. (2015): Human-level control through deep reinforcement learning. Nature 518(7540), 529–533.
- PaperSilver, D., Hubert, T., Schrittwieser, J. et al. (2018): A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362(6419), 1140–1144.
- PaperAmodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. and Mané, D. (2016): Concrete Problems in AI Safety. arXiv:1606.06565.
- PaperLevine, S., Kumar, A., Tucker, G. and Fu, J. (2020): Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643.
Content last reviewed on 25 September 2026. Please let us know about any errors: impressum@nyxai.com.