A systems journey · one prompt to a production cluster

Understand what makes
LLM inference fast

Follow one request through memory, model compression, scheduling, benchmarking, and distributed serving. Every lesson starts with a bottleneck and ends with the metric it changes.

5 chapters14 lessons≈ 101 minutes0 / 14 completed

New to inference systems?

Start with one request, then widen the system

Each lesson follows the same prompt and answers three questions: what hurts, what changes, and which metric moves.

00FoundationHow does generation work?
01MemoryWhat occupies the GPU?
02ModelCan the artifact shrink?
03RuntimeCan requests share better?
04ProductionCan the service scale safely?

Begin with Autoregressive Generation

CH. 00

What happens before we optimize anything?

Inference Foundations

Build the serving mental model first: watch a causal transformer generate one token at a time, decide when running locally is useful, and learn the latency, throughput, and quality objectives used to judge the system.

LESSON 00.1Live

Autoregressive Generation

How does a transformer turn one prompt into a streamed answer?

Bottleneck: Serial token dependency
Primary metric: TTFT + ITL

LESSON 00.2Live

Why Local Models + SLOs

When should we run locally, and how do we prove the service is good enough?

Bottleneck: Deployment choice + success criteria
Primary metric: Cost + latency + goodput

CH. 01

What is consuming the machine?

Memory Model

Begin with one request. See why saved attention work makes decoding faster, why the cache grows with every token, and why fitting a model is not the same as serving it efficiently.

LESSON 01Live

KV Cache

Why does faster decoding create a growing memory bill?

Bottleneck: Repeated decode compute
Primary metric: ITL + memory

LESSON 02Live

GPU Memory

Why can a model fit numerically and still fail or run slowly?

Bottleneck: Capacity + bandwidth
Primary metric: Fit + utilization

CH. 02

Can the model itself become cheaper?

Model Optimization

Change the artifact before serving begins. Store fewer bits, remove selected weights, and keep quality, kernel support, and real wall-clock performance visible as separate concerns.

LESSON 03Live

Quantization

How can the same model values occupy fewer bytes?

Bottleneck: Weight storage + traffic
Primary metric: Memory + throughput

LESSON 04Live

Sparsification

If many weights contribute little, can we skip them?

Bottleneck: Unnecessary parameters
Primary metric: Compute + quality

CH. 03

Can requests share work and hardware better?

Inference Optimization

Keep the accelerator useful under uneven traffic. Schedule at token boundaries, allocate cache in pages, reuse exact prefixes, and verify multiple draft tokens per target-model pass.

LESSON 05Live

Continuous Batching

Why wait for the slowest request in a fixed batch?

Bottleneck: Idle batch slots
Primary metric: Throughput + queueing

LESSON 06Live

PagedAttention

How can variable KV caches grow without large contiguous regions?

Bottleneck: KV fragmentation
Primary metric: Capacity + batching

LESSON 07Live

Prefix Caching

Why recompute the same system prompt for every user?

Bottleneck: Repeated prefill
Primary metric: TTFT + throughput

LESSON 08Live

Speculative Decoding

Can one expensive target pass accept several output tokens?

Bottleneck: Serial target passes
Primary metric: ITL

CH. 04

Where do these mechanisms live in real software?

Implementation Details

Finish the journey at the software boundary. Prepare a compressed artifact, serve it efficiently, measure it under realistic load, and coordinate replicas across a cluster.

LESSON 09Live

LLM Compressor

How does a full-precision checkpoint become deployment-ready?

Bottleneck: Oversized artifact
Primary metric: Size + quality

LESSON 10Live

vLLM & SGLang

How do two modern runtimes build a model server?

Bottleneck: End-to-end execution
Primary metric: Goodput

LESSON 11Live

GuideLLM

How do we know whether a deployment is fast enough?

Bottleneck: Unreliable measurement
Primary metric: SLO goodput

LESSON 12Live

llm-d

How do model servers become one production cluster?

Bottleneck: Cluster coordination
Primary metric: Tail latency + scale

Course synthesis

Different techniques solve different bottlenecks

They compose; they are not interchangeable.

Repeated token workKV Cache · Speculative Decoding

Reuse past state or commit more useful output per expensive target pass.

Model bytesQuantization · Sparsification

Change the artifact before serving; kernel support decides whether smaller also means faster.

Shared GPU capacityContinuous Batching · PagedAttention

Keep execution slots useful and allocate growing KV state without large reservations.

Repeated promptsPrefix Caching

Skip prefill only when the exact token prefix and model state are compatible.

Production operationvLLM · GuideLLM · llm-d

Serve one model, measure its SLO-safe capacity, then coordinate multiple replicas.