Machine learning & feedback
Understanding contextual bandits
An overall success rate can hide meaningful differences. Contextual bandits learn which action fits which situation while continuing to gather new evidence.
The short answer
A contextual bandit observes a context before choosing an action. In this simple discrete model, each context–action combination has its own beta posterior. The preferred action can therefore differ between situations.
Open interactive exampleFree · in your browser · no sign-up
At a glance
- Context is information available before choosing an action.
- Separate models can learn different preferences but need enough observations in each context.
- Changing environments require adaptation; an ordinary posterior does not forget old data.
From one best action to the right action
A support process can offer a short guide or a guided conversation. A guide may be enough for a straightforward question, while a complex problem benefits from the conversation. Looking only at average success could lead to assigning the same option to every request. A contextual bandit instead uses information known beforehand, such as the type of request, to choose an action for the situation.
The sequence is simple: observe context, choose an action, record the chosen action’s outcome and update the relevant model. The outcome of the unchosen action remains unknown. This partial feedback distinguishes the problem from an ordinary dataset containing complete target values. Contextual recommendation is a classic application (Li et al. 2010). A learned preference, however, does not establish why the difference exists.
The discrete model and its update
Our simulation has contexts 1 and 2 and actions A and B. It maintains a separate success probability for every combination. With binary outcomes and a uniform prior, observing successes and failures produces the posterior
For the current context, Thompson sampling draws one value from each action’s posterior and chooses the larger value. It then updates only the combination that actually produced an observation. The comparison policy uses the same calculation but maintains just two global posteriors. It ignores context entirely.
This is deliberately a small table-based model. Research on contextual Thompson sampling includes linear relationships between features and rewards (Agrawal & Goyal 2013). Such models can share information across situations. Our separate tables do not: a new context category starts with its own prior and must accumulate its own evidence.
A worked example
In context 1, action A succeeds with probability 0.8 and B with probability 0.2. Context 2 reverses these rates. If both contexts occur equally often, either fixed action has an overall success probability of . A policy that already knows the better action for each context reaches instead.
Over 100 decisions, that gives 50 expected successes for a fixed action and 80 for the informed contextual policy. The difference of 30 is a theoretical comparison under the specified rates, not the observed result of a particular random run. A learning algorithm must gather information first and cannot immediately match the informed benchmark.
After three successes and one failure for A in context 1, its posterior is , with mean . Observations in context 2 do not change this posterior. That separation makes different decisions possible, but it also creates a need for more observations. A small minority context may remain uncertain even when the total number of decisions looks substantial.
Explore regret and changing environments
Compare contextual bandits interactively
This simulation computes two Thompson sampling policies using reproducible random numbers.
The curve shows cumulative expected loss relative to the best action in the current context. With these rates, choosing the worse action incurs regret. A random failure after choosing the better action does not add regret. Consequently, realised success counts and regret can tell different stories in a single run.
First, change only the share of context 1. A less frequent context 2 gives its model fewer observations. Then enable the shift: halfway through the run, all success rates swap. Old observations remain in the posteriors. An action that previously seemed reliable may stay preferred long after the environment has changed. This demonstrates the absence of forgetting; it does not compare advanced methods designed for nonstationary environments.
Assumptions and limitations
Contexts are drawn independently. Conditional on context and action, outcomes follow Bernoulli distributions, and feedback arrives immediately. Context must be available before the decision. Information created by the chosen action cannot simply be treated as an input feature available beforehand.
A fixed random seed makes comparisons repeatable. It does not replace repeated runs, uncertainty intervals or evaluation on independent data. Large context spaces, delayed feedback and rare events make learning harder. Unrestricted exploration is also inappropriate when some actions have unacceptable consequences. In that case, the allowed set of actions must be constrained before selection takes place.
From the experiment to a process
Before deployment, settle three concrete questions: what outcome matters, which features are available in time, and which actions are permitted? Rewarding a click can produce different choices from rewarding a resolved request. Reward design explores this distinction. For a starting point without context, Multi-armed bandits explains the underlying balance between learning and using current knowledge.
Frequently asked questions
Is a contextual bandit reinforcement learning?
It is a simplified decision problem with feedback for the chosen action. Unlike a general Markov decision process, this model does not represent long-term effects of actions on future states.
Does context have to describe a person?
No. Time of day, process state, device type or available capacity can also provide context. The information must be available when the decision is made.
Does this simulation learn linear relationships?
No. It uses two discrete contexts and independent beta–Bernoulli models. It does not share information between similar contexts.
Sources and further reading
- PaperLi, L., Chu, W., Langford, J. and Schapire, R. E. (2010): A Contextual-Bandit Approach to Personalized News Article Recommendation. WWW 2010.
- PaperAgrawal, S. and Goyal, N. (2013): Thompson Sampling for Contextual Bandits with Linear Payoffs. PMLR 28(3), 127–135.
Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.