YouTube
MinGPT with Attention
Attention Is All You Need · Vaswani et al.
A minimal decoder-only Transformer with plain causal attention, learned position lookup, residual blocks, and next-token prediction.
foundation · complete flowopen →
Foundation and research timeline · 2017–2026
Visualizations to understand
LLM Internals with Code
The LLM Improvement Visualizer is a free collection of interactive, code-linked walkthroughs of transformer internals. Each lesson pairs readable PyTorch with a live flow diagram so you can inspect tensor shapes, compute costs, memory savings, and the primary research.
Sinusoidal PE
Attention Is All You Need · Vaswani et al.
Fixed sine and cosine frequencies give token vectors an explicit, non-learned sense of position.
position: added to the token embedding
position · foundationalopen →
Mixture of Experts
Sparsely-Gated Mixture-of-Experts · Shazeer et al.
A learned router activates only a small subset of expert networks for each token.
feed-forward · sparse computeopen →
KV Cache
Fast Transformer Decoding · Shazeer
Reuse previously computed keys and values during autoregressive decoding instead of recomputing the past.
inference · reuseopen →
MQA · GQA
MQA 2019 · GQA extension 2023
Share key/value heads across many query heads to reduce cache bandwidth and memory.
attention · shared KVopen →
Modern Transformer Block
RMSNorm 2019 · SwiGLU 2020
Compare the classic LayerNorm and GELU block with the RMSNorm and SwiGLU design used by many modern decoder models.
block · normalization · gated MLPopen →
Linear Attention
Transformers are RNNs · Katharopoulos et al.
Reorder feature-mapped attention into causal KᵀV and normalization states whose decoding memory does not grow with sequence length.
attention · recurrent stateopen →
Sliding Window
Longformer · Beltagy et al.
Restrict each token to a local causal band. This page uses readable dense reference code and explains the optimized O(T·W) kernel separately.
attention · local contextopen →
RoPE
RoFormer · Su et al.
Rotate query and key pairs by position so relative distance emerges inside the dot product.
position: applied to queries and keys in attention
position · rotationopen →
DeltaNet
Fast Weight Programmers 2021 · Parallel DeltaNet 2024
Read an old key association, correct its value with a learned Delta update, and schedule the recurrence sequentially or in chunks.
fast weights · delta ruleopen →
Flash Attention
FlashAttention · Dao et al.
Tile exact attention with online softmax so the full score matrix is never materialized in high-bandwidth memory.
attention · IO-awareopen →
Speculative Decoding
Speculative Decoding · Leviathan et al.
Use a small draft model to propose several tokens, then verify them together with the target model while preserving the target distribution.
decoding · draft and verifyopen →
Quantization
SmoothQuant · Xiao et al.
Compare FP16, INT8, and INT4 weight storage, SmoothQuant outlier migration, integer matrix multiplication, and quantization error.
compression · integer inferenceopen →
PagedAttention
vLLM · Kwon et al. 2023
Map logical KV blocks onto non-contiguous GPU pages to reduce fragmentation, batch requests efficiently, and share cached prefixes.
KV cache · paging · servingopen →
Latent · MLA
DeepSeek-V2 · DeepSeek-AI
Cache a compact latent per block and reconstruct keys and values only when attention needs them.
attention · compressed cacheopen →
Gated DeltaNet · KDA
Gated Delta Networks 2024 · Kimi Linear 2025
Combine global or channel-wise forgetting with targeted Delta replacement and chunk-wise recurrent-state computation.
linear attention · gating · KDAopen →
AttnRes
Attention Residuals · Kimi Team
Replace fixed accumulation across model depth with learned softmax selection over earlier block representations.
residual stream · depth attentionopen →
Kimi K3
Kimi K3: Open Frontier Intelligence · Kimi Team
Explore a reduced educational Kimi K3 macrocycle combining KDA, Gated MLA, Stable LatentMoE, SiTU, and blockwise AttnRes.
hybrid architecture · KDA · MoEopen →
a decoder-only LLM (GPT)
Embedding
Position
Attention
MLP
Norm
Output