Machine learning & feedback

Reinforcement Learning explained

Reinforcement Learning describes how a system learns through actions and feedback to make good decisions over many steps. The core concepts, the Bellman equation and Q-learning—with a small example you can calculate yourself.

  • 10 min read
  • Updated on
  • smartlytics editorial team

At a glance

  1. Reinforcement Learning learns from rewards rather than correct answers: an agent acts, observes the consequences and gradually improves its strategy.
  2. The Bellman equation divides a decision’s value into its immediate reward and the discounted value of the next state. Value iteration and Q-learning build on this relationship.
  3. In practice, success depends on reward design, the amount of data, simulation quality and safety constraints—not on the algorithm alone.

Many decisions have effects beyond the present moment. Building inventory today leaves less capital tomorrow; a move in chess changes all the possibilities that follow. Reinforcement Learning is the branch of machine learning that deals with precisely such sequences of decisions: an agent learns from feedback how to act so that its total rewards are as large as possible over the long term (Sutton & Barto 2018).

Learning by acting

In supervised learning, every example comes with the correct answer. In Reinforcement Learning, no such answer is provided. The agent only learns how good or bad the consequences of its actions were—often many steps later. This creates two tasks that supervised methods do not face: determining which earlier actions deserve credit for a later reward, and trying actions whose consequences are not yet known.

Formally, this is a feedback loop. At each time tt, the agent observes a state sts_t, chooses an action ata_t, and receives a reward rt+1r_{t+1} and the next state st+1s_{t+1} from the environment.

Agent Environment Action at State st+1 and reward rt+1
The Reinforcement Learning feedback loop: the agent chooses an action; the environment responds with a new state and a reward.

The model: a Markov decision process

The mathematical framework is the Markov decision process (MDP) (Puterman 1994). It consists of

  • a set of states SS and a set of actions AA,
  • transition probabilities P(s′∣s,a)P(s' \mid s, a): how likely is action aa in state ss to lead to state s′s'?
  • a reward function r(s,a,s′)r(s, a, s'), and
  • a discount factor γ\gamma between 0 and 1.

“Markov” means that the current state contains everything that matters for the future. The history provides no additional relevant information.

The agent’s objective is not the next reward, but the return: the discounted sum of all future rewards:

Gt=rt+1+γ rt+2+γ2 rt+3+⋯=∑k=0∞γk rt+k+1G_t = r_{t+1} + \gamma\, r_{t+2} + \gamma^2\, r_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k\, r_{t+k+1}

A γ\gamma close to 0 makes the agent short-sighted; a γ\gamma close to 1 makes it far-sighted. Like an interest rate, γ\gamma expresses the idea that a reward today is worth more than the same reward later. Roughly speaking, rewards much further away than 1/(1−γ)1/(1-\gamma) steps carry little weight—for γ=0,9\gamma = 0{,}9, that is about ten steps.

The agent’s behavior is described by its strategy, the policy π(a∣s)\pi(a \mid s): the probability of choosing action aa in state ss. The goal is a policy that maximizes expected return.

Values and the Bellman equation

Two value functions make this search manageable. The state value Vπ(s)V^\pi(s) is the expected return when starting in ss and then following policy π\pi. The action value Qπ(s,a)Q^\pi(s, a) is the expected return when first choosing action aa in ss and then following π\pi.

Richard Bellman showed that such values can be written recursively (Bellman 1957). For a fixed policy, the Bellman expectation equation is:

Vπ(s)=∑aπ(a∣s)∑s′P(s′∣s,a) [ r(s,a,s′)+γ Vπ(s′) ]V^\pi(s) = \sum_{a} \pi(a \mid s) \sum_{s'} P(s' \mid s, a)\,\bigl[\,r(s, a, s') + \gamma\, V^\pi(s')\,\bigr]

In words: a state’s value is the expected immediate reward plus the discounted value of what follows. For the best possible policy, averaging over actions is replaced by a maximum—the Bellman optimality equation:

V∗(s)=max⁡a∑s′P(s′∣s,a) [ r(s,a,s′)+γ V∗(s′) ]V^*(s) = \max_{a} \sum_{s'} P(s' \mid s, a)\,\bigl[\,r(s, a, s') + \gamma\, V^*(s')\,\bigr]

Once the optimal values are known, the optimal policy is simple: choose the action that achieves the maximum in every state. The same equation exists for action values, Q∗(s,a)=∑s′P(s′∣s,a) [ r(s,a,s′)+γmax⁡a′Q∗(s′,a′) ]Q^*(s, a) = \sum_{s'} P(s' \mid s, a)\,[\,r(s, a, s') + \gamma \max_{a'} Q^*(s', a')\,]. It is the basis of Q-learning.

An example to calculate yourself: the corridor

An agent stands in a corridor with three cells: A, B and C. To the right of C is the goal, which ends the episode. In each cell, the agent can move left or right; moving left from A leaves it in place. Reaching the goal gives a reward of 10; every other step gives 0. The discount factor is γ=0,9\gamma = 0{,}9.

Value iteration solves the Bellman optimality equation by repeatedly applying it as an update rule. Start with V0=0V_0 = 0 for all cells and, in each iteration, recalculate every cell:

Vk+1(s)=max⁡a∑s′P(s′∣s,a) [ r(s,a,s′)+γ Vk(s′) ]V_{k+1}(s) = \max_{a} \sum_{s'} P(s' \mid s, a)\,\bigl[\,r(s, a, s') + \gamma\, V_k(s')\,\bigr]
Iteration kk Vk(A)V_k(\text{A}) Vk(B)V_k(\text{B}) Vk(C)V_k(\text{C})
0 0 0 0
1 0 0 10
2 0 9 10
3 8.1 9 10
4 8.1 9 10

In the first iteration, only C recognizes that moving right earns 10. In the second, B learns about it: to the right is C with value 10, discounted to 0,9⋅10=90{,}9 \cdot 10 = 9. In the third, the information reaches A with 0,9⋅9=8,10{,}9 \cdot 9 = 8{,}1. After that, nothing changes—the values have converged. They flow backwards from the goal through the state space, and V∗(A)=8,1V^*(\text{A}) = 8{,}1 is precisely the reward of 10 discounted twice by a factor of 0.9. The optimal policy is immediately apparent: move right everywhere.

A related method is policy iteration: it alternates between calculating the values of a fixed policy and improving that policy by choosing the best action in each state according to those values. Both methods belong to dynamic programming and assume that transitions and rewards are known (Puterman 1994).

Interactive

Gridworld: reinforcement learning you can explore

The interactive explainer demonstrates value iteration and Q-learning in a larger world with a goal, traps and walls. Without JavaScript, the corridor example above still provides the complete calculation.

Learning without a model: TD learning and Q-learning

In most interesting applications, PP and rr are unknown. The agent must learn from experience: observed transitions of the form (state, action, reward, next state). Temporal-difference learning estimates values by moving each estimate a little towards a target that itself contains an estimate—the observed reward plus the estimated value of the next state (Sutton 1988).

Q-learning applies this idea to action values (Watkins & Dayan 1992). After each step, the value of the chosen action is updated:

Q(s,a)←Q(s,a)+α [ r+γmax⁡a′Q(s′,a′)−Q(s,a) ]Q(s, a) \leftarrow Q(s, a) + \alpha\,\bigl[\,r + \gamma \max_{a'} Q(s', a') - Q(s, a)\,\bigr]

The expression in brackets is the TD error: the difference between what experience suggests and the previous estimate. The learning rate α\alpha determines how far to move towards it.

Q-learning has a remarkable property: it learns the values of the best policy even while the agent behaves differently—for example, because it is still exploring. This is called off-policy learning. In the tabular case, the estimates converge to Q∗Q^* if every state–action pair is visited infinitely often and the learning rate decreases appropriately (Watkins & Dayan 1992).

This also explains why exploration is necessary. An agent that always selects the action currently believed best never learns whether another action would have been better. The simplest solution is ε-greedy: usually choose the best known action, but select a random one with small probability ε\varepsilon. Multi-Armed Bandits address the same exploration–exploitation dilemma, without states and with more sophisticated solutions.

From tables to neural networks

Tables work only while every state can be enumerated. Even an image with a few pixels has more possible states than could ever be visited. A function then replaces the table, calculating values from features of the state—linearly or using a neural network. Combining this with deep networks is called Deep Reinforcement Learning. One milestone was the Deep Q-Network, which learned 49 Atari games directly from screen pixels and, among other techniques, stored and reused earlier experiences (Mnih et al. 2015).

A second major family optimizes the policy directly. Policy-gradient methods change a policy’s parameters in the direction that increases expected return; the classic starting point is the REINFORCE algorithm (Williams 1992). Actor–critic methods connect both approaches: one component learns the policy and another learns its values.

Board games show how well learning and planning can work together. AlphaZero combined a network trained through self-play with a tree search that looks ahead in each position, achieving very high playing strength in chess, shogi and Go (Silver et al. 2018). The search uses a model of the game rules—a luxury rarely available in real-world applications.

Which method fits which question?

Reinforcement Learning is powerful, but it is not always the best choice. A simpler method can often solve the same task with less data and more understandable reasoning:

Situation Natural approach
One-off decision; probabilities and consequences can be estimated Decision theory with expected utility
Repeated choice between options, without subsequent states Multi-Armed Bandit
Known model, many constraints, planning over a horizon Optimization or dynamic programming
Consequences across many steps, unknown model, experience or simulation available Reinforcement Learning

Reinforcement Learning in practice

Several common hurdles stand between textbook examples and production use:

  • Reward design. An agent optimizes exactly what is rewarded, not what is intended. If a reward is incompletely specified, it finds shortcuts that formally earn points while missing the real objective. This risk is called reward hacking and is a known safety problem in learning systems (Amodei et al. 2016).
  • Data requirements. Many methods need millions of interactions. Agents therefore often learn in simulations—and then have to bridge the gap between simulation and reality.
  • Learning from existing data. Offline Reinforcement Learning learns from recorded decisions without trying actions itself. This is safer but difficult, because the data contains only the actions that were actually chosen (Levine et al. 2020).
  • Constraints and oversight. Real processes impose hard constraints, and bad decisions cost real money or more. Firm boundaries, gradual approvals, continuous monitoring and the ability to reverse decisions are useful safeguards.

Conclusion

Reinforcement Learning rests on a simple structure: states, actions, rewards and the Bellman equation connecting the value of a decision to the value of its consequences. Value iteration uses that equation when the model is known; Q-learning learns the same values from experience. The route into practice requires carefully designed rewards, realistic simulations and clear boundaries. For a deeper treatment, Sutton and Barto provide a comprehensive account (Sutton & Barto 2018).

Frequently asked questions

What is the difference between Reinforcement Learning and supervised learning?

In supervised learning, every example comes with a correct answer. In Reinforcement Learning, the agent receives only a reward for the consequences of its actions, often many steps later. It must work out which actions contributed to that reward and also try things it does not yet know.

What is Q-learning?

Q-learning estimates the long-term discounted reward for each state–action pair. After every step, the estimate moves a little towards the immediate reward plus the best estimated value of the next state. It does not require a model of the environment.

What does the Bellman equation say?

It says that a state’s value equals the expected immediate reward plus the discounted value of the next state. For the optimal strategy, the action with the highest value is selected in each state. Almost all Reinforcement Learning methods derive from this recursive relationship.

Does Reinforcement Learning need neural networks?

No. Small problems can be solved using tables that assign a separate value to every state or state–action pair. Neural networks or other function approximators are needed only when there are too many states for a table—for example, when images are used as inputs.

Sources and further reading

  1. BookSutton, R. S. and Barto, A. G. (2018): Reinforcement Learning: An Introduction (2nd edition). MIT Press.The standard textbook; the authors make a PDF version freely available.
  2. BookBellman, R. (1957): Dynamic Programming. Princeton University Press.
  3. BookPuterman, M. L. (1994): Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.
  4. PaperSutton, R. S. (1988): Learning to predict by the methods of temporal differences. Machine Learning 3(1), 9–44.
  5. PaperWatkins, C. J. C. H. and Dayan, P. (1992): Q-learning. Machine Learning 8(3–4), 279–292.
  6. PaperWilliams, R. J. (1992): Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8(3–4), 229–256.
  7. PaperMnih, V., Kavukcuoglu, K., Silver, D. et al. (2015): Human-level control through deep reinforcement learning. Nature 518(7540), 529–533.
  8. PaperSilver, D., Hubert, T., Schrittwieser, J. et al. (2018): A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362(6419), 1140–1144.
  9. PaperAmodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. and Mané, D. (2016): Concrete Problems in AI Safety. arXiv:1606.06565.
  10. PaperLevine, S., Kumar, A., Tucker, G. and Fu, J. (2020): Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643.

Content last reviewed on 25 September 2026. Please let us know about any errors: impressum@nyxai.com.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.