←visualization / Modern Block

minGPT · Modern Transformer Block

RMSNorm and SwiGLU

◆ MODERN BLOCK

n_embd

n_head

n_layer

seq_lenT=8

block styleiRMSNorm + SwiGLU

🔍 Variable Inspector

click any code line

Click a code line to inspect its variables.

About this lesson

Quick answer

Compare the classic LayerNorm and GELU block with the RMSNorm and SwiGLU design used by many modern decoder models.