Skip to visualization
←visualization / Linear Attention

minGPT · Linear Attention

feature maps and recurrent state
◆ LINEAR STATE
T=8
iLinear

🔍 Variable Inspector

click any code line
Click a code line to inspect its variables.
About this lesson
Quick answer

What Linear Attention explains

Reorder feature-mapped attention into causal KᵀV and normalization states whose decoding memory does not grow with sequence length.

Key concepts

Primary research

Transformers are RNNs · Katharopoulos et al.