←visualization / Mixture of Experts

minGPT · Mixture of Experts

sparse routed feed-forward

◆ MOE

n_embd

n_head

n_layer

seq_lenT=8

expertsi8

shared expertsi1

top_ki2

MoE styleiLatent

latent widthi64

🔍 Variable Inspector

click any code line

Click a code line to inspect its variables.

About this lesson

Quick answer

What Mixture of Experts explains

A learned router activates only a small subset of expert networks for each token.

Key concepts

feed-forward
sparse compute

Primary research

Sparsely-Gated Mixture-of-Experts · Shazeer et al.