←visualization / Flash Attention

minGPT · Flash Attention

tiled exact attention

◆ FLASH

n_embd

n_head

n_layer

seq_lenT=8

block_qi8

block_ki8

🔍 Variable Inspector

click any code line

Click a code line to inspect its variables.

About this lesson

Quick answer

Tile exact attention with online softmax so the full score matrix is never materialized in high-bandwidth memory.