Interactive explainer

Attention calculator: query, key and value

One attention head with four artificial tokens: all vectors remain visible and every output is calculated again.

Interactive

Attention calculator: query, key and value

Four artificial tokens expose the complete calculation of a single attention head.

What is calculated

Each token has a predefined query, key and value vector with two components. For the selected query token, the calculator takes dot products with all keys, divides by the square root of the dimension and by temperature, and calculates softmax. The resulting weights then mix the value vectors.

The table shows the keys and values used, together with scores and weights. The result text reports the output vector and the sum of weights. Every number is calculated in the browser. Tokens A to D identify artificial positions; their vectors were not learned from language.

Three experiments

  1. Change the query: Select A and then C. Observe which keys receive more weight and how the output vector changes.
  2. Concentrate the weights: Keep C selected and reduce temperature. The largest allowed score gains relative weight. Higher temperature spreads weights more evenly.
  3. Hide the future: Enable the causal mask and select A. Only A remains visible; its weight is one and the output equals its value vector exactly.

Reading the result

The sum of weights stays at one. A masked position receives zero, and allowed positions are normalised again. The output is a weighted mixture, not a selection of one token and not generated text. Attention temperature is also distinct from the temperature used later when sampling text from output probabilities.

The article on Transformer attention develops the formula and a numerical example, and explains how this small calculator differs from a trained language model.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

AI workflows at NyxAI

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.