Skip to visualization
← visualization / RoPE

minGPT · RoPE

rotary, applied in attention
RoPE (Rotary)
i
i
i
iT=8
i

🔍 Variable Inspector

click any code line
Click any line of code to inspect its variables, shapes, and meanings.
About this lesson
Quick answer

What Rotary Position Embedding explains

Rotate query and key pairs by position so relative distance emerges inside the dot product.

Key concepts

Primary research

RoFormer · Su et al.