Skip to visualization
←visualization / AttnRes

minGPT · Attention Residuals

selective retrieval across depth
◆ DEPTH ATTENTION
T=8
iAttnRes
i4
i2

🔍 Variable Inspector

click any code line
Click a code line to inspect its variables.
About this lesson
Quick answer

What Attention Residuals explains

Replace fixed accumulation across model depth with learned softmax selection over earlier block representations.

Key concepts

Primary research

Attention Residuals · Kimi Team