This course examines the systems that make modern AI possible, and the systems that agents are now forcing us to rethink. We start with LLMs and agents themselves — using them, then building one — so that the rest of the course has something concrete to optimize. We then work down the stack that sits between a model and the hardware it runs on: attention kernels and the roofline, then the serving path, where batching, KV-cache and prefix reuse, quantization, speculative decoding, and routing decide what inference actually costs. Two tracks are provided as optional content rather than covered in lecture: GPU architecture and Triton kernel programming, and the training side of the stack — distributed training, fine-tuning, and data pipelines. We close by turning the relationship around and asking what LLMs can do for systems research. At every layer the question is the same: where do the time and the money actually go?
Tentative and subject to change. Class meets Tuesday and Thursday. Dates follow the FAS Fall 2026 calendar; the Thanksgiving recess and the last day of classes should be confirmed against the registrar's calendar. Rows with Students as the speaker are student-led paper discussions — see the paper discussion page. Readings marked required should be read before class; everything else is optional depth. Training-side topics are not lectured on — their materials live in Optional Content below. Every reading below, plus supplementary papers on each topic, is collected on the readings page.
| Week | Date | Speaker | Topic | Course Materials | Notes |
|---|---|---|---|---|---|
| Part I — Introduction to LLMs and Agents | |||||
| 1 | 2026-09-03 | Juncheng Yang | Course overview and logistics | notes; optional readings Transformer, scaling laws, sustainable AI | |
| 2 | 2026-09-08 | Juncheng Yang | Modern ML basics: transformers, training, and where inference cost comes from | notes; required reading efficiently scaling inference; optional readings Transformer, PyTorch, TensorFlow, Chinchilla | |
| 2 | 2026-09-10 | Juncheng Yang | Agents from a user’s perspective | notes; optional readings MemGPT, RAG, ReAct | Assignment 1 out (use an agent) |
| 3 | 2026-09-15 | Juncheng Yang | Agents from a designer’s perspective I: the loop, tools, and context | notes; required reading Parrot; optional readings SGLang, Toolformer | Assignment 2 out (design an agent) |
| 3 | 2026-09-17 | Juncheng Yang | Agents from a designer’s perspective II: what agent workloads do to the serving system | notes; optional readings Ray, InferCept, Teola, agentic AI workload characteristics | |
| Part II — Systems for LLMs | |||||
| 4 | 2026-09-22 | Juncheng Yang | LLM serving basics | required reading vLLM / PagedAttention; optional readings FlashInfer, vLLM docs | Paper presentation sign-up opens |
| 4 | 2026-09-24 | Juncheng Yang | Efficient LLM computing: GPU kernels | required readings FlashAttention, roofline; optional readings FlashAttention-2, FlashAttention-3, making DL go brrr | |
| 5 | 2026-09-29 | Juncheng Yang | Efficient LLM serving: batching and scheduling I | required reading Orca; optional readings vLLM / PagedAttention, LoongServe, NanoFlow | Assignment 1 due, 11:59pm |
| 5 | 2026-10-01 | Students | Student sharing I — what you built with an agent (Assignment 1) | no assigned reading; bring your Assignment 1 agent transcript — one task it handled well, one it failed, and your read on why | Presentation paper choice due Oct 2 |
| 6 | 2026-10-06 | Juncheng Yang | Efficient LLM serving: batching and scheduling II — disaggregation | required reading Sarathi-Serve; optional readings DistServe, Splitwise, Mooncake, MemServe, P/D-Serve | |
| 6 | 2026-10-08 | Students | Efficient LLM serving: KV-cache optimization | required readings Mooncake, KIVI; optional readings InfiniGen, ring attention, SnapKV, Quest, HiSparse | Assignment 2 due, 11:59pm; Assignment 3 out (serve your own agent) |
| 7 | 2026-10-13 | Students | Efficient LLM serving: prefix cache | required readings SGLang / RadixAttention, Preble; optional readings Parrot, ChunkAttention, Mooncake, MemServe, Marconi | |
| 7 | 2026-10-15 | Students | Student sharing II — agent designs and what broke (Assignment 2) | no assigned reading; bring your Assignment 2 design — the loop you wrote, the tool interface you chose, and the failure mode you did not anticipate | |
| 8 | 2026-10-20 | Juncheng Yang | Efficient LLM serving: pruning and quantization I | required reading LLM.int8(); optional readings SmoothQuant, QServe, mixed precision | |
| 8 | 2026-10-22 | Students | Efficient LLM serving: pruning and quantization II | required readings GPTQ, QServe; optional readings AWQ, SmoothQuant, QuIP#, QuaRot, SpinQuant | |
| 9 | 2026-10-27 | Students | Efficient LLM serving: speculative decoding | required readings speculative decoding, EAGLE; optional readings Medusa, Sequoia, EAGLE-2, MagicDec | Assignment 3 due Oct 27, 11:59pm; Assignment 4 out (optimize the full stack) |
| 9 | 2026-10-29 | Students | Student sharing III — serving your own model (Assignment 3) | no assigned reading; bring your Assignment 3 stack — the quality gap you closed against the frontier API, and what it cost you to close it | |
| 10 | 2026-11-03 | Juncheng Yang | Efficient LLM serving: routing and load balancing | required reading Llumnix; optional readings AlpaServe, Preble, MuxServe, DynamoLLM, Clipper, GShard, Switch Transformer | |
| 10 | 2026-11-05 | Juncheng Yang | Efficient agent serving systems I: declared structure and the request DAG | required reading Parrot; optional readings SGLang, InferCept, Autellix | |
| 11 | 2026-11-10 | Juncheng Yang | Efficient agent serving systems II: tool stalls, interception, and the sandbox | required reading InferCept; optional readings parallelizing tool execution, SpecBox, DeltaBox, Firecracker, Sandlock | |
| 11 | 2026-11-12 | Juncheng Yang | Efficient agent serving systems III: session state and agent memory | required reading Continuum; optional readings Mooncake, CacheGen, prediction-based KV-cache management, KV-cache management survey, MemGPT | Assignment 4 due Nov 12, 11:59pm |
| 12 | 2026-11-17 | Juncheng Yang | Efficient agent serving systems IV: multi-agent workloads and performance optimization | required reading TokenCake; optional readings TokenDance, ScaleSim, ForkKV, large language monkeys, KernelBench | |
| 12 | 2026-11-19 | Students | Efficient agent serving systems V: scheduling agentic programs | required readings Autellix, Teola; optional readings SAGA, Helium, agentic AI workload characteristics, Continuum | cache competition opens |
| 13 | 2026-11-24 | Students | Student sharing IV — what you optimized and what it cost (Assignment 4) | no assigned reading; bring your Assignment 4 numbers — one profile, one prediction, one measurement, and the gap between the last two | |
| Part III — LLMs for Systems | |||||
| 14 | 2026-12-01 | Students | LLM for Systems Research I | required readings SWE-agent, KernelBench; optional readings LLMs for compiler optimization, OpenHands, SWE-bench Multimodal, BountyBench, measure one level deeper | |
| 14 | 2026-12-03 | Students | LLM for Systems Research II | required readings MLGym, AI Scientist-v2; optional readings automated statistical model discovery, MLE-bench, SWE-bench, OSWorld | cache competition closes Dec 3, 11:59pm |
The topics below are not covered in lecture and are not examinable. There is no class meeting, assignment, or paper discussion attached to any of them. They fall in two tracks.
GPU programming. How the hardware is built and how you write code against it directly. Lecture covers attention kernels and the roofline (Sep 24) at the level needed to reason about serving cost, but stops short of the architecture underneath and of writing kernels yourself. Work through this track if you want to write your own Triton kernels — the Assignment 4 optimization path and the kernel-level optional project both assume it. Start with the CUDA guide's programming-model and hardware-implementation chapters, then run the Triton tutorials.
The training side of the stack. How a model gets built, rather than how it gets served. Work through it if you want the background on distributed training, fine-tuning, or checkpointing. The Ultra-Scale Playbook is the best single entry point; the papers below go deeper, and the readings page collects them in one place with more on each.
| Topic | Materials |
|---|---|
| GPU programming for MLSys I: architecture and the memory hierarchy | start with the CUDA guide (programming model, hardware implementation, and the coalescing and occupancy guidelines); then dissecting Volta, dissecting Hopper, TPU |
| GPU programming for MLSys II: Triton, tiling, and fusion | start by running the Triton docs tutorials 01–03 (vector add, fused softmax, matmul); then Triton, CUDA matmul, ThunderKittens, TileLang |
| Distributed training and fine-tuning I: parallelism and ZeRO | start with Megatron-LM, ZeRO; then FSDP, GPipe, PipeDream, Alpa, LoRA, QLoRA, TorchTitan |
| Distributed training and fine-tuning II: memory, failures, and scale | start with Llama 3 (§3, infrastructure); then MegaScale, DeepSeek-V3, ByteCheckpoint, FP8-LM, ZeRO-Infinity, sublinear memory |
| Data pipelines and checkpointing | Data-Juicer, Dolma, DataComp-LM, CheckFreq, GEMINI, 3FS |
Compute for the assignments, the cache competition, and the optional project is expected to come from a combination of the following. Access instructions will be posted on the optional project page once confirmed.