Language models & knowledge

Understand transformer attention with numbers

Attention combines information from several positions in a sequence. Four artificial tokens are enough to make the calculation visible: calculate scores, normalise them and mix values.

  • 4 min read
  • Updated on
  • smartlytics editorial team

The short answer

Scaled dot-product attention calculates scores from a query and keys, scales them and uses softmax to produce weights. Its output is the weighted sum of value vectors. A causal mask excludes later positions.

Open interactive exampleFree · in your browser · no sign-up

At a glance

  1. The query and keys determine the weights; values supply the content being mixed.
  2. Softmax produces nonnegative weights that sum to one, not probabilities that statements are true.
  3. A causal mask blocks access to later positions independently of temperature.

Which information fits the current position?

A sequence contains multiple positions whose information may have different relevance for the next processing step. Attention calculates how strongly one position incorporates content from other positions. The querying position has a query vector. Each available position supplies a key for comparison and a value containing the information to combine.

Attention was used in neural translation before the transformer (Bahdanau et al. 2015). The transformer combines multiple attention heads with further processing steps (Vaswani et al. 2017). Our example isolates exactly one head. It uses artificial tokens A, B, C and D. These letters identify positions and have no learned linguistic meaning here. That keeps the entire calculation inspectable.

Query, key and value work together

For a query qq and key kik_i, first calculate their dot product. Scores are divided by dk\sqrt{d_k}, where dkd_k is the key dimension. Our additional adjustable temperature τ>0\tau>0 controls the concentration of the weights:

zi=q⋅kidk τ,wi=exp⁡(zi)∑jexp⁡(zj),o=∑iwivi.z_i=\frac{q\cdot k_i}{\sqrt{d_k}\,\tau},\qquad w_i=\frac{\exp(z_i)}{\sum_j\exp(z_j)},\qquad o=\sum_i w_i v_i.

Softmax converts the scores into nonnegative weights that sum to one. For numerical calculation, we subtract the largest score before exponentiation. This leaves the weights unchanged while avoiding unnecessarily large intermediate values.

The roles are distinct: keys and the query determine how much weight a position receives. Values determine what that position contributes to the output vector. A high score therefore does not by itself tell you the final number in a particular output component. You need the values as well. In this example, the output remains a weighted average of the visible value vectors.

A worked example with four tokens

Choosing token C gives query qC=(1,1)q_C=(1, 1). The keys are kA=(1,0)k_A=(1, 0), kB=(0,1)k_B=(0, 1), kC=(1,1)k_C=(1, 1) and kD=(−1,0)k_D=(-1, 0). Their dot products with the query are 1,1,2,−11, 1, 2,-1. At temperature one and dimension two, the scores are approximately 0.7070.707, 0.7070.707, 1.4141.414 and −0.707-0.707.

The corresponding rounded weights are 0.2340.234, 0.2340.234, 0.4750.475 and 0.0570.057. The third token receives the largest weight, but the other values still contribute. With vA=(1,0)v_A=(1, 0), vB=(0,2)v_B=(0, 2), vC=(2,1)v_C=(2, 1) and vD=(−1,1)v_D=(-1, 1), the output is approximately o=(1.127,1.000)o=(1.127, 1.000). The display calculates these quantities without the intermediate rounding shown here.

The calculation becomes particularly simple for token A with a causal mask. Only the first position is allowed. Its weight is exactly one, and every other weight is zero. The output is therefore precisely vA=(1,0)v_A=(1, 0), regardless of the positive temperature setting.

Change temperature and masking

Interactive

Attention calculator: query, key and value

The calculator computes scores, softmax weights and the output vector of an artificial attention head.

Lower temperature amplifies differences between allowed scores. Higher temperature spreads weight more evenly. It does not change which positions are visible in the first place. A causal mask effectively sets later positions’ scores to negative infinity, giving them zero weight after softmax.

Start with C, vary the temperature and then enable masking. D is now excluded completely. The remaining weights are normalised again. This redistribution matters: masking does more than delete one contribution from an already completed sum; it also changes the other weights. The displayed table makes both the excluded position and the updated values visible.

What this representation does not explain

The vectors were chosen by hand. A trained transformer instead calculates Q, K and V using learned transformations of its current representations. Our example has no positional encoding, training, multiple heads, residual connections or output distribution over a vocabulary.

Attention weights are not probabilities that statements are true, nor are they a complete explanation of a model decision. They describe mixing within a single calculation. The attention temperature adjusted here is also distinct from the temperature used later to sample output tokens from a language model. The two use related mathematical operations but act on different quantities at different stages.

From representations to supported information

Attention can combine information within a supplied context. Whether that context contains current, relevant information is a separate question. Retrieval-augmented generation therefore examines the preceding search for sources and checks on their validity. Rules and optimisation then shows why a fluent description does not prove that a plan is feasible.

Frequently asked questions

Do attention weights fully explain a model answer?

No. They describe one calculation. Multiple heads, layers, residual connections and other transformations also affect the result.

Is this temperature the same as text-generation temperature?

No. Here it scales attention scores within one head. A temperature used when sampling output tokens acts at another point in the model.

Were the demo vectors learned?

No. All Q, K and V vectors were deliberately chosen by hand. The demo performs actual matrix calculations but is not a trained language model.

Sources and further reading

  1. PaperVaswani, A. et al. (2017): Attention Is All You Need. NeurIPS 2017.
  2. PaperBahdanau, D., Cho, K. and Bengio, Y. (2015): Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.

Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.