Every reading in the course, in one place and grouped by the class meeting it belongs to. 257 works across 26 meetings.
The schedule on the home page remains the authoritative date-by-date view: it is the one to check the week before class, and if the two pages ever disagree, it wins. This page is the same axis with more on it. Each meeting below lists not just what is assigned but the optional depth for that day and the supplementary papers on the same problem — so one section is everything the course has to say about prefix caching, or speculative decoding, or tool stalls. Use it to prepare for a meeting, to choose a paper to present, or to find the background an assignment or a project needs.
A paper assigned on more than one date carries an also note pointing at the other meetings, and is listed under each of them.
You are not expected to read all of this, and no one will. A realistic plan: the required papers, plus one meeting’s section in depth for whatever you end up going deep on. If a topic is new to you, start from Surveys and Long-Form References rather than from the most recent paper in the list. The last group on this page, Not Tied to a Class Meeting, holds the self-study tracks and the reference material that no single meeting owns.
Presenting. Seven meetings are student-led paper discussions, marked student-led below. Presentation papers are normally drawn from the required and optional entries for that date, but a well-argued substitution from the same meeting’s section is usually fine — ask first. The paper discussion page has the presenter guide, the rubric, and a three-pass reading strategy; Reading and Evaluating Systems Papers below has the two short pieces worth reading before your first paper.
The short list: what has to be read before which meeting. Follow a topic to jump to that meeting’s full section below. Rows marked student-led are paper discussions.
Nothing is required for the first meeting. These three optional papers frame why the course exists: the architecture everything later optimizes, the scaling argument that made models large, and the energy and carbon bill that arrives afterward. The supplementary entries are the capability results people usually have in mind when they say “LLM” — useful context, never examinable.
Lecture notes for this meeting →
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Transformer — Attention Is All You Need | 2017 | optional also optional Sep 8 |
The architecture the whole course optimizes. Read it for shapes, not for BLEU scores: which operations are matmuls, which are attention, and what has to be kept around from one token to the next. |
| scaling laws — for Neural Language Models | 2020 | optional | Loss as a smooth power law in parameters, data, and compute. The reason anyone believed making models bigger would keep working. |
| sustainable AI — Environmental Implications, Challenges and Opportunities | 2021 | optional | Carbon and energy accounting for the full lifecycle, including the parts usually left out. |
| GPT-3 — Language Models are Few-Shot Learners | 2020 | extra | Where in-context learning came from — and therefore why prompts got long enough that caching them pays. |
| InstructGPT — Training language models to follow instructions with human feedback | 2022 | extra | How a base model becomes an assistant. The step that makes chat and tool use possible at all. |
| emergent abilities — of Large Language Models | 2022 | extra | The claim that some capabilities appear discontinuously with scale — and a good exercise in asking whether an effect is real or an artifact of the metric. |
Where inference cost comes from. Read the required paper closely: it is the analytic model the whole of Part II argues with, and the partitioning discussion sets up the serving lectures. The rest of this section fills in the model itself — what a token is, which tensors get multiplied, what has to be kept between tokens, which architecture choices decide how big that state is, and which open-weight families the assignments run on.
Lecture notes for this meeting →
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| efficiently scaling inference — Efficiently Scaling Transformer Inference | 2022 | required | The paper to read closely. Analytic model of inference cost, partitioning strategies, and where latency actually goes at scale. |
| Transformer — Attention Is All You Need | 2017 | optional also optional Sep 3 |
The architecture the whole course optimizes. Read it for shapes, not for BLEU scores: which operations are matmuls, which are attention, and what has to be kept around from one token to the next. |
| PyTorch — An Imperative Style, High-Performance Deep Learning Library | 2019 | optional | Eager execution as a deliberate design choice, and what it costs. |
| TensorFlow — A system for large-scale machine learning | 2016 | optional | The dataflow-graph alternative. Read it against PyTorch: the tradeoff is debuggability versus whole-program optimization. |
| Chinchilla — Training Compute-Optimal Large Language Models | 2022 | optional | The correction: models were badly undertrained for their size. It moved the optimum toward smaller models on more data, which is exactly why 7B-class models are worth serving. |
| BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding | 2018 | extra | The encoder-only counterpart, and a useful contrast: no autoregressive decode, so no KV cache and none of the serving problems in Part II. |
| GPT-4 — Technical Report | 2023 | extra | Almost no systems detail, which is itself the point: read it next to Llama 3 and DeepSeek-V3 to see how much closed reports withhold. |
| LLaMA — Open and Efficient Foundation Language Models | 2023 | extra | The open-weight line the assignments run on. |
| Llama 2 — Open Foundation and Fine-Tuned Chat Models | 2023 | extra | Adds the chat/RLHF half and the license that made open-weight serving mainstream. |
| Mistral 7B | 2023 | extra | A 7B-class model in exactly the size range Assignment 3 targets; sliding-window attention. |
| Qwen2.5 — Technical Report | 2024 | extra | A strong open-weight family with a wide size ladder — convenient when a project needs to sweep model size. |
| OLMo — Accelerating the Science of Language Models | 2024 | extra | Fully open: data, code, checkpoints, logs. The one to pick if a project needs to see inside training. |
| DeepSeek-R1 — Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | 2025 | extra | Reasoning models emit far more decode tokens per request, which shifts every tradeoff in Part II toward the decode side. |
| RoPE — RoFormer: Enhanced Transformer with Rotary Position Embedding | 2021 | extra | Rotary embeddings — why KV entries are position-encoded before caching, which constrains how cached prefixes can be reused. |
| ALiBi — Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | 2021 | extra | An alternative positional scheme aimed at extrapolating past the training context length. |
| SwiGLU — GLU Variants Improve Transformer | 2020 | extra | Why the MLP block has three weight matrices instead of two — it changes the FLOP and parameter accounting. |
| RMSNorm — Root Mean Square Layer Normalization | 2019 | extra | A cheaper normalization; a small paper that shows up in every fusion discussion. |
| Mamba — Linear-Time Sequence Modeling with Selective State Spaces | 2023 | extra | State-space models: constant state per sequence instead of a growing KV cache. Read it for what a serving stack would look like without a KV cache at all. |
What an agent is, before any of it becomes a systems problem: a loop that calls the model many times, retrieval to get facts in, and memory that has to outlive the context window. Read these for the shape of the workload rather than the prompt tricks — every one of these patterns turns into repeated, nearly identical prefixes later. The evaluation entries matter for Assignment 1: they decide what “it worked” means.
Lecture notes for this meeting →
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| MemGPT — Towards LLMs as Operating Systems | 2023 | optional also optional Nov 12 |
Treats the context window as a memory hierarchy with explicit paging. The most systems-flavored agent paper on the list. |
| RAG — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | 2020 | optional | Retrieval-augmented generation — the origin of the long, partly repeated prompts that make prefix caching worth building. |
| ReAct — Synergizing Reasoning and Acting in Language Models | 2022 | optional | The interleaved reason/act loop that nearly every agent framework is a variation on. |
| lost in the middle — How Language Models Use Long Contexts | 2023 | extra | Long context is not free accuracy. Relevant when a project's answer to a problem is "put more in the prompt". |
| chain-of-thought — Prompting Elicits Reasoning in Large Language Models | 2022 | extra | Buying accuracy with decode tokens — the first place quality and cost trade off explicitly. |
| self-consistency — Improves Chain of Thought Reasoning in Language Models | 2022 | extra | Sample many chains, vote. Linear cost increase for a bounded accuracy gain; a clean target for batching. |
| Tree of Thoughts — Deliberate Problem Solving with Large Language Models | 2023 | extra | Search over reasoning states. Branching turns one request into a tree with a shared prefix — exactly the structure RadixAttention exploits. |
| Reflexion — Language Agents with Verbal Reinforcement Learning | 2023 | extra | Self-critique across attempts, which makes trajectories long and stateful. |
| Self-Refine — Iterative Refinement with Self-Feedback | 2023 | extra | The same idea at single-response granularity: generate, critique, revise. |
| WebArena — A Realistic Web Environment for Building Autonomous Agents | 2023 | extra | Agents in a reproducible web environment with real applications behind it. |
| AgentBench — Evaluating LLMs as Agents | 2023 | extra | A multi-environment agent benchmark; useful mostly as a survey of what people think agents should be tested on. |
| GAIA — a benchmark for General AI Assistants | 2023 | extra | Questions that are easy for people and hard for assistants, which is a sharper test than it sounds. |
The first paper that treats an agent as a program the serving system can see. Read Parrot for its central claim — if the system knows the structure of a request, it can schedule it better — and note how little of that structure a plain chat API exposes. The optional and supplementary entries are the tool-use line: how a model learns to call an API, and what breaks when there are thousands of them.
Lecture notes for this meeting →
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Parrot — Efficient Serving of LLM-based Applications with Semantic Variable | 2024 | required also required Nov 5; optional Oct 13 |
Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency. |
| SGLang — Efficient Execution of Structured Language Model Programs | 2023 | optional also required Oct 13; optional Nov 5 |
A front-end language plus a runtime, co-designed. Read it here for the programming model; the RadixAttention half is in prefix caching. |
| Toolformer — Language Models Can Teach Themselves to Use Tools | 2023 | optional | Teaching a model when to call a tool, rather than prompting it to. |
| Voyager — An Open-Ended Embodied Agent with Large Language Models | 2023 | extra | A long-running agent that accumulates a skill library — open-ended sessions rather than one-shot requests. |
| HuggingGPT — Solving AI Tasks with ChatGPT and its Friends in Hugging Face | 2023 | extra | An LLM as a planner that dispatches to other models — heterogeneous inference behind one request. |
| Gorilla — Large Language Model Connected with Massive APIs | 2023 | extra | Tool calling when the tool catalog is far too large to fit in the prompt. |
| ToolLLM — Facilitating Large Language Models to Master 16000+ Real-world APIs | 2023 | extra | Scaling tool use to thousands of APIs, with the retrieval step that implies. |
| CoALA — Cognitive Architectures for Language Agents | 2023 | extra | A framework for comparing agent designs instead of collecting them. Useful when writing Assignment 2. |
The seam where Part I meets Part II. Three assumptions of classic serving break at once: a request is a DAG of dependent calls rather than one prompt, generation stalls whenever a tool runs, and consecutive prompts are nearly identical. The multi-agent entries are here because they make all three worse, and because they are the workloads the last block of Part II is about.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Ray — A Distributed Framework for Emerging AI Applications | 2017 | optional | The distributed task model underneath a lot of agent and RL infrastructure. |
| InferCept — Efficient Intercept Support for Augmented Large Language Model Inference | 2024 | optional also required Nov 10; optional Nov 5 |
What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out. |
| Teola — Towards End-to-End Optimization of LLM-based Applications | 2024 | optional also required Nov 19 |
End-to-end optimization across the whole application dataflow rather than the model call alone. |
| agentic AI workloads — Agentic AI Workload Characteristics | 2026 | optional also optional Nov 19 |
A characterization of agentic workloads — the trace-level view of what these systems actually receive. |
| generative agents — Interactive Simulacra of Human Behavior | 2023 | extra | Many agents, each with memory and reflection. Read it for the workload shape: bursty, highly repetitive prompts. |
| AutoGen — Enabling Next-Gen LLM Applications via Multi-Agent Conversation | 2023 | extra | Multi-agent conversation as a programming model — and the scheduling problem it hands the serving system. |
| MetaGPT — Meta Programming for A Multi-Agent Collaborative Framework | 2023 | extra | Role-structured multi-agent pipelines with explicit hand-offs. |
The core of Part II. Two ideas carry most of the weight: manage KV memory in pages rather than contiguously, and batch at the granularity of a token rather than a request. Nearly every later paper assumes both, so read vLLM closely enough to explain why fragmentation was the binding constraint. The supplementary entries are the offload and preemption alternatives that lost.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| vLLM / PagedAttention — Efficient Memory Management for Large Language Model Serving with PagedAttention | 2023 | required also optional Sep 29 |
Virtual memory for the KV cache. The single most important systems paper on the list — fragmentation, not compute, was the binding constraint. |
| FlashInfer — Efficient and Customizable Attention Engine for LLM Inference Serving | 2025 | optional | A customizable attention engine underneath the serving stacks — where the kernel layer and the scheduler layer meet. |
| vLLM docs — vLLM documentation | 2022 | optional | The serving stack most projects will build on. |
| FlexGen — High-Throughput Generative Inference of Large Language Models with a Single GPU | 2023 | extra | Throughput-first offloading to CPU and disk. The opposite end of the latency spectrum from everything else here. |
| FastServe — Fast Distributed Inference Serving for Large Language Models | 2023 | extra | Preemptive scheduling with multi-level feedback queues, borrowed straight from OS scheduling. |
| DeepSpeed-FastGen — High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference | 2024 | extra | Dynamic SplitFuse — another take on mixing prefill and decode in one batch. |
Where the arithmetic actually happens. The recurring theme: the win comes from moving less data, not from doing less math, so tiling, fusion, and where a tensor lives matter more than the FLOP count. Read the roofline paper first — it gives you the units for the rest of the course — then FlashAttention as the canonical application. The hands-on kernel-writing material is in the self-study GPU track below.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| FlashAttention — Fast and Memory-Efficient Exact Attention with IO-Awareness | 2022 | required | Tile attention so the N×N score matrix is never written to HBM. The clearest demonstration in the course that IO, not FLOPs, is the budget. |
| roofline — An Insightful Visual Performance Model for Multicore Architectures | 2009 | required | The model that tells you whether you are compute- or bandwidth-bound before you optimize anything. The right first step in Assignment 4. |
| FlashAttention-2 — Faster Attention with Better Parallelism and Work Partitioning | 2023 | optional | Better work partitioning and fewer non-matmul FLOPs. |
| FlashAttention-3 — Fast and Accurate Attention with Asynchrony and Low-precision | 2024 | optional | Asynchrony and FP8 on Hopper — the same algorithm rewritten for a new memory and execution model. |
| making DL go brrr — Making Deep Learning Go Brrrr From First Principles | 2022 | optional | Compute, bandwidth, or overhead — decide which one you are fighting before you touch anything. |
| TVM — An Automated End-to-End Optimizing Compiler for Deep Learning | 2018 | extra | Compiling tensor programs to hardware: the schedule/compute split that Triton and TileLang later reuse. |
| Ansor — Generating High-Performance Tensor Programs for Deep Learning | 2020 | extra | Search-based schedule generation — the auto-tuning half of the compiler story. |
| cuDNN — Efficient Primitives for Deep Learning | 2014 | extra | The vendor-library layer that hand-written kernels compete against. |
| online softmax — Online normalizer calculation for softmax | 2018 | extra | Two pages, and the numerical trick that makes FlashAttention possible. Read it first. |
| FlashDecoding++ — Faster Large Language Model Inference on GPUs | 2023 | extra | Decode-side attention, where the batch is small and the kernel is bandwidth-bound. |
| Hidet — Task-Mapping Programming Paradigm for Deep Learning Tensor Programs | 2022 | extra | Pushes scheduling into the programming model instead of the search space. |
| Mirage — A Multi-Level Superoptimizer for Tensor Programs | 2024 | extra | Superoptimization across kernel, thread-block, and thread levels at once. |
Prefill is compute-bound and batches well; decode is memory-bound and does not. Orca’s answer, iteration-level scheduling, is the baseline every later serving paper is measured against, so read it for the mechanism and for how cleanly the result is isolated. The supplementary entries are about predicting how long a request will run, which is what makes any of this schedulable in the first place.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Orca — A Distributed Serving System for Transformer-Based Generative Models | 2022 | required | Continuous (iteration-level) batching: let requests join and leave a batch every token instead of every request. Also a model of a well-argued systems evaluation. |
| vLLM / PagedAttention — Efficient Memory Management for Large Language Model Serving with PagedAttention | 2023 | optional also required Sep 22 |
Virtual memory for the KV cache. The single most important systems paper on the list — fragmentation, not compute, was the binding constraint. |
| LoongServe — Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism | 2024 | optional | Elastic sequence parallelism — vary the parallelism degree with the request, not with the deployment. |
| NanoFlow — Towards Optimal Large Language Model Serving Throughput | 2024 | optional | Intra-device parallelism: overlap compute, memory, and network within a single device to approach a throughput bound. |
| length prediction — Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction | 2024 | extra | Scheduling wants the output length, which is exactly what is unknown. Predict it with a cheap proxy model. |
| response length perception — Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline | 2023 | extra | Ask the model itself how long its answer will be, then schedule on that. |
| VTC fairness — Fairness in Serving Large Language Models | 2023 | extra | What fairness even means when requests cost wildly different amounts — and a scheduler that achieves it. |
No assigned reading. Bring your Assignment 1 agent transcript — one task it handled well, one it failed, and your read on why.
If prefill and decode want different hardware and different batching, you can either interleave them carefully on one machine or split them across machines and ship the KV cache between them. Sarathi-Serve is the first answer; the optional papers are the second, and they disagree about what the split costs. Watch for what lands on the critical path: a large, latency-sensitive transfer that did not exist before.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Sarathi-Serve — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve | 2024 | required | Chunked prefill plus stall-free batching: split long prefills so decodes are never blocked behind them. |
| DistServe — Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving | 2024 | optional | The clean statement of the idea: disaggregate the two phases, optimize goodput under both TTFT and TPOT constraints. |
| Splitwise — Efficient generative LLM inference using phase splitting | 2023 | optional | The same split, argued from power and hardware heterogeneity — and backed by a production trace. |
| Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | optional also required Oct 8; optional Oct 13, Nov 12 |
A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling. |
| MemServe — Context Caching for Disaggregated LLM Serving with Elastic Memory Pool | 2024 | optional also optional Oct 13 |
An elastic memory pool that makes cached context a first-class, shareable resource across instances. |
| P/D-Serve — Serving Disaggregated Large Language Model at Scale | 2024 | optional | Disaggregation as actually operated at scale, including the parts that are unglamorous. |
The KV cache grows linearly with context and batch size, and it is read in full for every token generated — so it is simultaneously the capacity limit and the bandwidth limit. The readings shrink it, quantize it, tier it, or drop parts of it, and the sparse-attention entries attack the same cost from the algorithm side. The two architecture papers are here because MQA and GQA change the size of the problem before any system touches it.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | required also optional Oct 6, Oct 13, Nov 12 |
A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling. |
| KIVI — A Tuning-Free Asymmetric 2bit Quantization for KV Cache | 2024 | required | 2-bit KV quantization, asymmetric by axis: per-channel for keys, per-token for values. Tuning-free, which is why it is deployable. |
| InfiniGen — Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management | 2024 | optional | Speculate which KV entries the next layer will actually attend to, and prefetch only those from CPU memory. |
| ring attention — with Blockwise Transformers for Near-Infinite Context | 2023 | optional | Distribute a long sequence across devices and overlap the KV transfer with compute. |
| SnapKV — LLM Knows What You are Looking for Before Generation | 2024 | optional | The attention pattern over the prompt predicts which entries matter, before generation starts. |
| Quest — Query-Aware Sparsity for Efficient Long-Context LLM Inference | 2024 | optional | Query-aware page selection: pick which KV pages to attend to per query, per layer. |
| HiSparse — Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management | 2026 | optional | Hierarchical KV cache management for sparse-attention decoding. |
| MQA — Fast Transformer Decoding: One Write-Head is All You Need | 2019 | extra | Multi-query attention: one KV head instead of many. The single largest constant-factor cut to KV cache size. |
| GQA — Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | 2023 | extra | The compromise between multi-head and multi-query that nearly every current open model ships. |
| H2O — Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models | 2023 | extra | Heavy hitters: a small set of tokens receives most of the attention mass, so the rest can be evicted. |
| StreamingLLM — Efficient Streaming Language Models with Attention Sinks | 2023 | extra | Attention sinks — keeping the first few tokens is what makes a sliding window not collapse. A genuinely surprising empirical result. |
| Scissorhands — Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time | 2023 | extra | Importance persists across steps, so an eviction decision made once stays roughly right. |
| FastGen — Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs | 2023 | extra | Different attention heads deserve different eviction policies; profile them and specialize. |
| PyramidKV — Dynamic KV Cache Compression based on Pyramidal Information Funneling | 2024 | extra | Budget the cache by layer rather than uniformly — deeper layers need less. |
| KVQuant — Towards 10 Million Context Length LLM Inference with KV Cache Quantization | 2024 | extra | Pushes KV quantization far enough that context length, not memory, becomes the limit again. |
| AttentionStore — Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention | 2024 | extra | A cache hierarchy for multi-turn conversations, where the reuse is across turns and sessions. |
| ShadowKV — KV Cache in Shadows for High-Throughput Long-Context LLM Inference | 2024 | extra | Keep a low-rank key cache on GPU and offload values, reconstructing on demand. |
| MInference — 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention | 2024 | extra | Long-prompt prefill is the bottleneck in RAG and agent workloads; exploit the structure in the attention pattern. |
| DuoAttention — Efficient Long-Context LLM Inference with Retrieval and Streaming Heads | 2024 | extra | Only some heads need the full context; give the others a streaming window. |
| NSA — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention | 2025 | extra | Sparse attention designed to be trainable and hardware-aligned, rather than bolted on at inference time. |
| MoBA — Mixture of Block Attention for Long-Context LLMs | 2025 | extra | Block-level routing of attention, in the spirit of mixture-of-experts. |
| Star Attention — Efficient LLM Inference over Long Sequences | 2024 | extra | Two-phase block-sparse attention aimed at distributed long-context inference. |
| LServe — Efficient Long-sequence LLM Serving with Unified Sparse Attention | 2025 | extra | Unifies static and dynamic sparsity in one serving system rather than one kernel. |
Agent and chat workloads send prompts that share long prefixes: a system prompt, a document, the conversation so far. Recomputing that prefix is pure waste, and the two required papers are the two halves of the fix — how to index shared prefixes, and how to schedule requests so the sharing actually pays. This is also the subject of the cache competition: everything here is cheap to compute and expensive to store, so the interesting question is what to evict. The eviction-policy background is in Background: Caching and Eviction.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| SGLang / RadixAttention — SGLang: Efficient Execution of Structured Language Model Programs | 2023 | required also optional Sep 15, Nov 5 |
A radix tree over cached prefixes with LRU eviction, plus a language whose structure exposes the sharing. |
| Preble — Efficient Distributed Prompt Scheduling for LLM Serving | 2024 | required also optional Nov 3 |
Prefix sharing across a distributed cluster, which turns cache reuse into a scheduling and placement problem. |
| Parrot — Efficient Serving of LLM-based Applications with Semantic Variable | 2024 | optional also required Sep 15, Nov 5 |
Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency. |
| ChunkAttention — Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition | 2024 | optional | A prefix-aware KV structure with a two-phase kernel, so sharing does not cost attention throughput. |
| Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | optional also required Oct 8; optional Oct 6, Nov 12 |
A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling. |
| MemServe — Context Caching for Disaggregated LLM Serving with Elastic Memory Pool | 2024 | optional also optional Oct 6 |
An elastic memory pool that makes cached context a first-class, shareable resource across instances. |
| Marconi — Prefix Caching for the Era of Hybrid LLMs | 2024 | optional | Prefix caching when the model is a hybrid (attention plus SSM) and cached state is no longer a simple per-token array. |
| Prompt Cache — Modular Attention Reuse for Low-Latency Inference | 2023 | extra | Reuse at the level of declared, reusable prompt modules rather than raw token prefixes. |
| Hydragen — High-Throughput LLM Inference with Shared Prefixes | 2024 | extra | Decompose attention into shared-prefix and per-sequence parts so the shared half becomes a dense matmul. |
| CacheBlend — Fast Large Language Model Serving for RAG with Cached Knowledge Fusion | 2024 | extra | RAG chunks share no common prefix. Recompute selectively so non-prefix reuse stays correct. |
| RAGCache — Efficient Knowledge Caching for Retrieval-Augmented Generation | 2024 | extra | A multilevel cache of retrieved-document KV state, informed by how skewed retrieval actually is. |
No assigned reading. Bring your Assignment 2 design — the loop you wrote, the tool interface you chose, and the failure mode you did not anticipate.
Fewer bits per weight or per KV entry buys memory and bandwidth, and the whole game is where the error goes. LLM.int8() is the outlier paper: read it for why naïve INT8 fails on large models specifically. The pruning entries are here for contrast — quantization shrinks every weight, pruning removes weights outright, and only structured sparsity is something the hardware can actually exploit.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| LLM.int8() — 8-bit Matrix Multiplication for Transformers at Scale | 2022 | required | The outlier problem, discovered: a few feature dimensions have huge magnitudes and wreck naive INT8. Mixed-precision decomposition as the fix. |
| SmoothQuant — Accurate and Efficient Post-Training Quantization for Large Language Models | 2022 | optional also optional Oct 22 |
Migrate quantization difficulty from activations to weights with a per-channel rescale. |
| QServe — W4A8KV4 Quantization and System Co-design for Efficient LLM Serving | 2024 | optional also required Oct 22 |
W4A8KV4 with the system co-designed alongside the algorithm — the paper that takes dequantization overhead seriously. |
| mixed precision — Training | 2017 | optional | The original FP16 training recipe: loss scaling and an FP32 master copy. |
| FP8 formats — for Deep Learning | 2022 | extra | Why E4M3 and E5M2 exist and where each is used. |
| ZeroQuant — Efficient and Affordable Post-Training Quantization for Large-Scale Transformers | 2022 | extra | An early end-to-end PTQ pipeline with fused kernels; useful as a baseline. |
| SparseGPT — Massive Language Models Can Be Accurately Pruned in One-Shot | 2023 | extra | One-shot pruning to 50% sparsity without retraining, using the same second-order machinery as GPTQ. |
| Wanda — A Simple and Effective Pruning Approach for Large Language Models | 2023 | extra | Weights times input activations, and nothing else. The baseline any pruning method has to beat. |
| LLM-Pruner — On the Structural Pruning of Large Language Models | 2023 | extra | Structural pruning, which is the kind that actually makes a GPU faster. |
The post-training quantization line, and the rotation trick that dodges outliers instead of special-casing them. Read GPTQ for the method and QServe for what it costs to serve a quantized model at speed — the gap between a good compression ratio and a good token rate is the whole point of pairing them. The supplementary entries are the extreme end: 4-bit, 2-bit, and ternary.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| GPTQ — Accurate Post-Training Quantization for Generative Pre-trained Transformers | 2022 | required | One-shot post-training quantization with second-order error compensation, layer by layer. |
| QServe — W4A8KV4 Quantization and System Co-design for Efficient LLM Serving | 2024 | required also optional Oct 20 |
W4A8KV4 with the system co-designed alongside the algorithm — the paper that takes dequantization overhead seriously. |
| AWQ — Activation-aware Weight Quantization for LLM Compression and Acceleration | 2023 | optional | Not all weights matter equally; protect the salient 1% using activation statistics. |
| SmoothQuant — Accurate and Efficient Post-Training Quantization for Large Language Models | 2022 | optional also optional Oct 20 |
Migrate quantization difficulty from activations to weights with a per-channel rescale. |
| QuIP# — Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks | 2024 | optional | Hadamard incoherence plus lattice codebooks — the strong 2-bit result. |
| QuaRot — Outlier-Free 4-Bit Inference in Rotated LLMs | 2024 | optional | Rotate the model so outliers cannot exist in any basis, then quantize everything to 4 bits. |
| SpinQuant — LLM quantization with learned rotations | 2024 | optional | Learn the rotation instead of fixing it. |
| Atom — Low-bit Quantization for Efficient and Accurate LLM Serving | 2023 | extra | Low-bit quantization evaluated as a serving system, in end-to-end throughput rather than perplexity. |
| OmniQuant — Omnidirectionally Calibrated Quantization for Large Language Models | 2023 | extra | Learn the clipping ranges and equivalent transforms rather than hand-tuning them. |
| SqueezeLLM — Dense-and-Sparse Quantization | 2023 | extra | Dense low-bit weights plus a sparse outlier set kept at full precision. |
| BitNet b1.58 — The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | 2024 | extra | Ternary weights trained from scratch. Read it for the claim that quantization belongs in training, not after. |
Decode is memory-bound, so verifying several tokens costs barely more than generating one: guess ahead cheaply, then verify in parallel. The two required papers sit at opposite ends of the design space — an independent draft model versus a head trained on the target model’s own features. The variants differ mainly in where guesses come from and how they are verified; read two or three and the pattern is clear.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| speculative decoding — Fast Inference from Transformers via Speculative Decoding | 2022 | required | The original formulation, with the rejection-sampling argument for why the output distribution is preserved exactly. |
| EAGLE — Speculative Sampling Requires Rethinking Feature Uncertainty | 2024 | required | Draft in feature space rather than token space, one autoregressive head on top of the target model. |
| Medusa — Simple LLM Inference Acceleration Framework with Multiple Decoding Heads | 2024 | optional | Multiple decoding heads instead of a separate draft model — no second model to serve. |
| Sequoia — Scalable, Robust, and Hardware-aware Speculative Decoding | 2024 | optional | Treats the draft tree shape as an optimization problem, hardware-aware. |
| EAGLE-2 — Faster Inference of Language Models with Dynamic Draft Trees | 2024 | optional | Context-dependent dynamic draft trees. |
| MagicDec — Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding | 2024 | optional | Speculative decoding for long context and large batch, where the usual assumptions invert. |
| speculative sampling — Accelerating Large Language Model Decoding with Speculative Sampling | 2023 | extra | The concurrent DeepMind version; read alongside the above for the same idea argued differently. |
| blockwise parallel decoding — for Deep Autoregressive Models | 2018 | extra | The 2018 ancestor of the whole idea. |
| lookahead decoding — Break the Sequential Dependency of LLM Inference Using Lookahead Decoding | 2024 | extra | Draft with Jacobi iteration — no draft model and no training at all. |
| SpecInfer — Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification | 2023 | extra | Tree-based speculation with a token-tree verifier, framed as a serving system. |
| self-speculative — Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding | 2023 | extra | Draft by skipping layers of the model itself. |
| REST — Retrieval-Based Speculative Decoding | 2023 | extra | Retrieve draft continuations from a corpus instead of generating them. |
| SpecExec — Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices | 2024 | extra | Very large speculative trees when the target model is offloaded and each step is expensive. |
| online spec decoding — Online Speculative Decoding | 2023 | extra | Keep updating the draft model on the live query distribution. |
| spec decoding survey — Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding | 2024 | extra | Read this before picking a presentation paper in this area. |
No assigned reading. Bring your Assignment 3 stack — the quality gap you closed against the frontier API, and what it cost you to close it.
Above a single replica: which instance should serve this request, what happens when the fleet goes imbalanced, and how to share GPUs across many models or adapters. Llumnix is the migration argument — read it for why rescheduling a live request is worth the copy. The mixture-of-experts entries are here because sparse activation hands the same load-balancing problem to the layer below: the weights you need depend on the token.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Llumnix — Dynamic Scheduling for Large Language Model Serving | 2024 | required | Live migration of running requests between instances — scheduling as a continuous, not one-shot, decision. |
| AlpaServe — Statistical Multiplexing with Model Parallelism for Deep Learning Serving | 2023 | optional | Statistical multiplexing with model parallelism: parallelism is a latency tool under bursty load, not just a capacity tool. |
| Preble — Efficient Distributed Prompt Scheduling for LLM Serving | 2024 | optional also required Oct 13 |
Prefix sharing across a distributed cluster, which turns cache reuse into a scheduling and placement problem. |
| MuxServe — Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving | 2024 | optional | Spatial-temporal multiplexing of several LLMs on shared GPUs. |
| DynamoLLM — Designing LLM Inference Clusters for Performance and Energy Efficiency | 2024 | optional | Energy as a first-class objective for an inference cluster. |
| Clipper — A Low-Latency Online Prediction Serving System | 2016 | optional | Pre-LLM prediction serving. Worth reading for how much of the modern stack is a rediscovery, and how much genuinely is not. |
| GShard — Scaling Giant Models with Conditional Computation and Automatic Sharding | 2020 | optional | Conditional computation with automatic sharding — where the modern MoE recipe starts. |
| Switch Transformer — Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | 2021 | optional | Top-1 routing, and a careful account of the instability that comes with it. |
| Mixtral — of Experts | 2024 | extra | An open-weight sparse MoE, which is what makes MoE serving experiments feasible for a course project. |
| DeepSeekMoE — Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models | 2024 | extra | Fine-grained experts plus shared experts — the architecture behind DeepSeek-V3. |
| expert choice routing — Mixture-of-Experts with Expert Choice Routing | 2022 | extra | Invert the assignment: experts pick tokens, which fixes load balance by construction. |
| MegaBlocks — Efficient Sparse Training with Mixture-of-Experts | 2022 | extra | Block-sparse kernels that remove the token-dropping hack entirely. |
| Tutel — Adaptive Mixture-of-Experts at Scale | 2022 | extra | Adaptive parallelism and pipelining for MoE, switchable at runtime. |
| DeepSpeed-MoE — Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale | 2022 | extra | MoE inference as a systems problem: expert placement, communication, and distillation. |
| Fiddler — CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models | 2024 | extra | Run some experts on the CPU when they will not fit on the GPU. |
| MoE-Infinity — Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache | 2024 | extra | Expert activation is skewed and repetitive, so cache experts the way you would cache anything else. |
| pre-gated MoE — An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference | 2023 | extra | Decide the routing one layer early so the expert weights can be prefetched. |
| S-LoRA — Serving Thousands of Concurrent LoRA Adapters | 2023 | extra | Thousands of adapters over one base model: unified paging for both KV cache and adapter weights. |
| Punica — Multi-Tenant LoRA Serving | 2023 | extra | A batched kernel that lets requests using different adapters share one batch. |
| ServerlessLLM — Low-Latency Serverless Inference for Large Language Models | 2024 | extra | Cold start is the whole problem when models are gigabytes; make checkpoint loading fast. |
Parrot returns, now from the serving side. An agent request is really a DAG of dependent model calls, and a system that can see the DAG can schedule it, deduplicate its prefixes, and stop optimizing each call in isolation. This is the premise of the largest single block of lectures in Part II, so read it as a design proposal rather than as a set of numbers.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Parrot — Efficient Serving of LLM-based Applications with Semantic Variable | 2024 | required also required Sep 15; optional Oct 13 |
Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency. |
| SGLang — Efficient Execution of Structured Language Model Programs | 2023 | optional also required Oct 13; optional Sep 15 |
A front-end language plus a runtime, co-designed. Read it here for the programming model; the RadixAttention half is in prefix caching. |
| InferCept — Efficient Intercept Support for Augmented Large Language Model Inference | 2024 | optional also required Nov 10; optional Sep 17 |
What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out. |
| Autellix — An Efficient Serving Engine for LLM Agents as General Programs | 2025 | optional also required Nov 19 |
Treats an agent program, not a request, as the scheduling unit — and shows what head-of-line blocking costs when it is not. |
An agent request stops generating whenever it calls a tool, and the tool has to run somewhere isolated. That creates two problems the classic serving stack never had: what to do with GPU state during a stall — hold it, discard it, or swap it — and how to start and checkpoint a sandbox fast enough that the sandbox is not the new bottleneck.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| InferCept — Efficient Intercept Support for Augmented Large Language Model Inference | 2024 | required also optional Sep 17, Nov 5 |
What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out. |
| parallelizing tool execution — Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving | 2026 | optional | Overlap tool execution with generation instead of serializing them. |
| SpecBox — Speculative Sandbox Scheduling for Efficient LLM Agent Serving | 2026 | optional | Speculative sandbox scheduling — start the sandbox before you are certain it will be needed. |
| DeltaBox — Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback | 2026 | optional | Millisecond-level sandbox checkpoint and rollback for stateful agents. |
| Firecracker — Lightweight Virtualization for Serverless Applications | 2020 | optional | The microVM that most agent sandboxes are built on, and the case for why a VM boundary can still be cheap. |
| Sandlock — Confining AI Agent Code with Unprivileged Linux Primitives | 2026 | optional | Confining agent-generated code using unprivileged Linux primitives rather than a VM. |
Agents are stateful and long-lived, and their context comes back nearly unchanged on the next turn. The KV cache stops being a per-request scratchpad and becomes shared, persistent state with a lifetime policy — which makes it a storage problem more than an attention problem. Read Continuum for the session abstraction, then the optional entries for where that state physically goes.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Continuum — Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live | 2025 | required also optional Nov 19 |
Multi-turn agent scheduling with an explicit time-to-live on cached state: how long is a session's KV cache worth keeping? |
| Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | optional also required Oct 8; optional Oct 6, Oct 13 |
A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling. |
| CacheGen — KV Cache Compression and Streaming for Fast Large Language Model Serving | 2023 | optional | If cached KV has to cross a network, its encoding is a bandwidth problem, not just a memory problem. |
| prediction-based KV management — Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management | 2026 | optional | Predict what a dynamic agent workflow will need next and manage the cache accordingly. |
| KV cache management survey — From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving | 2026 | optional | From a tensor buffer to a distributed memory hierarchy — the survey to read before picking a presentation paper in this area. |
| MemGPT — Towards LLMs as Operating Systems | 2023 | optional also optional Sep 10 |
Treats the context window as a memory hierarchy with explicit paging. The most systems-flavored agent paper on the list. |
Many agents running over nearly identical context, and the measurement problem that comes with them. Every number in Part II is conditional on a workload, so the trace and simulator entries are here rather than in a corner: they are what Assignment 4 and the project evaluation are built on. Read TokenCake for the sharing opportunity and ScaleSim for how to sweep a design you cannot afford to build.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| TokenCake — A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications | 2025 | required | A KV-cache-centric serving framework built for multi-agent applications. |
| TokenDance — Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing | 2026 | optional | Collective KV cache sharing across agents — scaling by exploiting how much they have in common. |
| ScaleSim — Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management | 2026 | optional | Large-scale multi-agent simulation, scheduled by invocation distance. |
| ForkKV — Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache | 2026 | optional | Copy-on-write over a disaggregated KV cache for multi-LoRA agent serving. The OS analogy is exact and deliberate. |
| large language monkeys — Scaling Inference Compute with Repeated Sampling | 2024 | optional | Scaling inference compute by repeated sampling: coverage keeps rising with the number of samples. The cleanest statement of "spend more decode tokens, get more correct answers", and therefore of why throughput is a quality knob. |
| KernelBench — Can LLMs Write Efficient GPU Kernels? | 2025 | optional also required Dec 1 |
Can a model write a fast GPU kernel? Directly relevant to Assignment 4 and the exam. |
| Vidur — A Large-Scale Simulation Framework For LLM Inference | 2024 | extra | Simulate configurations you cannot afford to benchmark. The right tool for a large sweep on a small allocation. |
| Etalon — Holistic Performance Evaluation Framework for LLM Inference Systems | 2024 | extra | Argues that mean TTFT and TPOT hide what users experience, and proposes fluidity-based metrics instead. Read it before choosing your project's metric. |
| BurstGPT — A Real-world Workload Dataset to Optimize LLM Serving Systems | 2024 | extra | A real request trace, including the burstiness that synthetic Poisson arrivals leave out. |
| Mooncake trace — Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | extra also required Oct 8; optional Oct 6, Oct 13, Nov 12 |
The Mooncake paper ships a production trace; useful for cache and scheduling experiments. |
| Splitwise trace — Splitwise: Efficient generative LLM inference using phase splitting | 2023 | extra also optional Oct 6 |
Production inference traces from Azure, split by phase. |
| MLPerf — Benchmarks (MLCommons) | — | extra | How the industry defines an inference benchmark, including the rules about what may be tuned. |
Scheduling when the unit of work is a program rather than a request. The scheduler has to reason about dependencies between calls, about which program is closest to finishing, and about what is worth keeping in cache between steps — and it has to do it without the program telling it anything. Read the two required papers against each other: one infers structure, the other asks the program to declare it.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Autellix — An Efficient Serving Engine for LLM Agents as General Programs | 2025 | required also optional Nov 5 |
Treats an agent program, not a request, as the scheduling unit — and shows what head-of-line blocking costs when it is not. |
| Teola — Towards End-to-End Optimization of LLM-based Applications | 2024 | required also optional Sep 17 |
End-to-end optimization across the whole application dataflow rather than the model call alone. |
| SAGA — Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters | 2026 | optional | Workflow-atomic scheduling: schedule the whole agent workflow as a unit on a GPU cluster. |
| Helium — Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective | 2026 | optional | Serving agentic workflows from a data-systems angle. |
| agentic AI workloads — Agentic AI Workload Characteristics | 2026 | optional also optional Sep 17 |
A characterization of agentic workloads — the trace-level view of what these systems actually receive. |
| Continuum — Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live | 2025 | optional also required Nov 12 |
Multi-turn agent scheduling with an explicit time-to-live on cached state: how long is a session's KV cache worth keeping? |
No assigned reading. Bring your Assignment 4 numbers — one profile, one prediction, one measurement, and the gap between the last two.
Part III, and the premise behind the final exam: models that write, optimize, and evaluate systems code. Read these as systems papers — ask what the harness is, what the environment lets the agent touch, and what the benchmark actually scores. SWE-agent is the agent-computer interface argument; KernelBench is the one whose task is closest to what you did in Assignment 3.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| SWE-agent — Agent-Computer Interfaces Enable Automated Software Engineering | 2024 | required | The agent-computer interface is the contribution: what the tools look like matters more than which model is behind them. |
| KernelBench — Can LLMs Write Efficient GPU Kernels? | 2025 | required also optional Nov 17 |
Can a model write a fast GPU kernel? Directly relevant to Assignment 4 and the exam. |
| compiler optimization — Large Language Models for Compiler Optimization | 2023 | optional | An LLM predicting optimization passes, evaluated against a real compiler. |
| OpenHands — An Open Platform for AI Software Developers as Generalist Agents | 2024 | optional | An open platform for coding agents — the practical reference implementation. |
| SWE-bench Multimodal — Do AI Systems Generalize to Visual Software Domains? | 2024 | optional | Does any of it generalize outside Python? |
| BountyBench — Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | 2025 | optional | Agents on real security tasks, scored in dollars. |
| measure one level deeper — Always Measure One Level Deeper | 2018 | optional | Ousterhout on why end-to-end numbers alone will mislead you, and what to measure instead. Read it before Assignment 4. |
| Codex — Evaluating Large Language Models Trained on Code | 2021 | extra | Where code models and pass@k came from. |
| AlphaCode — Competition-Level Code Generation with AlphaCode | 2022 | extra | Massive sampling plus filtering. A useful reminder of how much compute early results cost. |
| learned index structures — The Case for Learned Index Structures | 2017 | extra | The ancestor of "replace a systems component with a model", and still the clearest statement of the tradeoff. |
The same question one level up: can the loop run a research process rather than a coding task. Read the benchmarks for what they cannot see — a score that rises while the underlying claim goes unchecked is the failure mode both of these papers are trying to avoid, with mixed success. Useful directly for the exam and for judging your own project’s evaluation.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| MLGym — A New Framework and Benchmark for Advancing AI Research Agents | 2025 | required | A gym-style environment for AI research agents, with tasks and an evaluation protocol. |
| AI Scientist-v2 — The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search | 2025 | required | Agentic tree search over the whole research loop. Read it skeptically and decide what you believe. |
| statistical model discovery — Automated Statistical Model Discovery with Language Models | 2024 | optional | LLMs proposing and revising statistical models. |
| MLE-bench — Evaluating Machine Learning Agents on Machine Learning Engineering | 2024 | optional | Agents on Kaggle competitions, with human baselines to compare against. |
| SWE-bench — Can Language Models Resolve Real-World GitHub Issues? | 2023 | optional | The benchmark that reframed code generation as repository-scale issue resolution. |
| OSWorld — Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments | 2024 | optional | Agents driving a real desktop. |
| AI Scientist — The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery | 2024 | extra | The first version, and the better target for criticism of the framing. |
Material no single meeting owns: the three self-study optional content tracks, the caching background behind the cache competition, and the surveys, tools, and paper-reading advice worth having open all term. Nothing here is examinable.
You cannot reason about a kernel without knowing the machine, and you cannot write one from the vendor documentation alone. The microbenchmarking papers measure what that documentation leaves out — real latencies, real cache sizes, real tensor-core behavior — and the rest is the tooling you would actually write a kernel in. This is the optional GPU programming track: not lectured on and not examinable, but the background for the kernel-level path in Assignment 4. The lectured kernel material is Sep 24.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| dissecting Volta — Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking | 2018 | optionalself-study | Microbenchmarking a GPU until its actual memory hierarchy falls out. The method matters more than the specific numbers. |
| dissecting Hopper — Benchmarking and Dissecting the Nvidia Hopper GPU Architecture | 2024 | optionalself-study | The same treatment for Hopper, including the tensor cores and features the LLM stack depends on. |
| TPU — In-Datacenter Performance Analysis of a Tensor Processing Unit | 2017 | optionalself-study | A machine designed for one workload. Read it for the argument about what a datacenter accelerator is for. |
| CUDA programming guide — CUDA Programming Guide | — | optionalself-study | Reference, not reading. Know where the memory-hierarchy and occupancy sections are. |
| Triton — An Intermediate Language and Compiler for Tiled Neural Network Computations | 2019 | optionalself-study | The tile-level abstraction: write blocks, let the compiler handle threads. |
| ThunderKittens — Simple, Fast, and Adorable AI Kernels | 2024 | optionalself-study | Small tile primitives that make hand-written kernels tolerable to write. |
| TileLang — A Composable Tiled Programming Model for AI Systems | 2025 | optionalself-study | Composable tiled programming, separating dataflow from schedule. |
| CUDA matmul walkthrough — How to Optimize a CUDA Matmul Kernel step by step | 2022 | optionalself-study | Ten iterations from naive to near-cuBLAS, each with the reason it helped. The best single hands-on introduction. |
| Triton docs — Triton documentation | — | optionalself-study | Tutorials first; the fused-softmax and matmul examples cover most of what Assignment 4 needs. |
Not lectured on and not examinable — this is the optional content track, kept here because projects that touch fine-tuning or distributed training need it. The Ultra-Scale Playbook is the best single entry point; these go deeper.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Llama 3 — The Llama 3 Herd of Models | 2024 | optionalself-study | §3 is the one to read: 16K GPUs, real failure rates, and what actually breaks at scale. |
| DeepSeek-V3 — Technical Report | 2024 | optionalself-study | Mixture-of-experts at frontier scale with an unusually candid systems section (FP8 training, custom communication kernels, hardware co-design). |
| Megatron-LM — Training Multi-Billion Parameter Language Models Using Model Parallelism | 2019 | optionalself-study | Tensor parallelism: split individual matmuls across devices. Start here. |
| ZeRO — Memory Optimizations Toward Training Trillion Parameter Models | 2019 | optionalself-study | Shard optimizer state, gradients, and parameters instead of replicating them. Start here. |
| Megatron on GPU clusters — Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM | 2021 | extra | How tensor, pipeline, and data parallelism compose — the 3D-parallelism accounting. |
| FSDP — PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | optionalself-study | The PyTorch-native version, and an honest account of what was hard to make work. |
| GPipe — Efficient Training of Giant Neural Networks using Pipeline Parallelism | 2018 | optionalself-study | Pipeline parallelism with micro-batches, and the bubble that comes with it. |
| PipeDream — Fast and Efficient Pipeline Parallel DNN Training | 2018 | optionalself-study | Asynchronous pipelining with weight stashing — fills the bubble at the cost of staleness. |
| zero bubble — Pipeline Parallelism | 2023 | extra | Splitting the backward pass to remove the pipeline bubble almost entirely. |
| Alpa — Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning | 2022 | optionalself-study | Search the parallelization strategy instead of hand-picking it. |
| TorchTitan — One-stop PyTorch native solution for production ready LLM pre-training | 2024 | optionalself-study | A current, readable reference implementation of all of the above composed together. |
| activation recomputation — Reducing Activation Recomputation in Large Transformer Models | 2022 | extra | Selective recomputation: the memory/compute knob, tuned properly. |
| sublinear memory — Training Deep Nets with Sublinear Memory Cost | 2016 | optionalself-study | The original gradient-checkpointing result. Short and worth reading. |
| ZeRO-Infinity — Breaking the GPU Memory Wall for Extreme Scale Deep Learning | 2021 | optionalself-study | Offload to CPU and NVMe when the model does not fit in aggregate GPU memory. |
| Ulysses — DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models | 2023 | extra | Sequence parallelism for very long contexts. |
| MegaScale — Scaling Large Language Model Training to More Than 10,000 GPUs | 2024 | optionalself-study | 10,000+ GPUs, with the diagnosis and fault-tolerance machinery that scale requires. |
| FP8-LM — Training FP8 Large Language Models | 2023 | optionalself-study | FP8 training end to end, and which tensors cannot tolerate it. |
| Horovod — fast and easy distributed deep learning in TensorFlow | 2018 | extra | Ring all-reduce as a usable library. Where data-parallel scaling became routine. |
| LoRA — Low-Rank Adaptation of Large Language Models | 2021 | optionalself-study | Low-rank adapters: fine-tune a small delta, keep the base frozen. Also what makes multi-tenant adapter serving possible. |
| QLoRA — Efficient Finetuning of Quantized LLMs | 2023 | optionalself-study | LoRA on a 4-bit frozen base — fine-tuning a large model on one GPU. |
Also optional content. Included because the RL loop is a genuinely interesting systems workload: it runs generation and training in the same job, with the sampler feeding the trainer.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| InstructGPT — Training language models to follow instructions with human feedback | 2022 | extra | The RLHF pipeline, stated plainly. |
| PPO — Proximal Policy Optimization Algorithms | 2017 | extra | The optimizer underneath most RLHF. Read the algorithm box, skip the rest if you like. |
| DPO — Direct Preference Optimization: Your Language Model is Secretly a Reward Model | 2023 | extra | Removes the reward model and the RL loop entirely — a large systems simplification. |
| HybridFlow / verl — HybridFlow: A Flexible and Efficient RLHF Framework | 2024 | extra | The systems paper of the group: how to place and schedule generation and training together. |
| OpenRLHF — An Easy-to-use, Scalable and High-performance RLHF Framework | 2024 | extra | A practical open implementation built on Ray and vLLM. |
The third optional-content track. Checkpointing in particular is a storage-systems problem that gets solved badly surprisingly often.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Data-Juicer — A One-Stop Data Processing System for Large Language Models | 2023 | optionalself-study | Data processing treated as a system with operators and a pipeline, not a pile of scripts. |
| Dolma — an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research | 2024 | optionalself-study | An open pretraining corpus with the curation decisions documented. |
| DataComp-LM — In search of the next generation of training sets for language models | 2024 | optionalself-study | Hold the training recipe fixed and vary only the data — a controlled experiment on data quality. |
| CheckFreq — Frequent, Fine-Grained DNN Checkpointing | 2021 | optionalself-study | Pick the checkpoint interval from first principles, and overlap the write with compute. |
| GEMINI — Fast Failure Recovery in Distributed Training with In-Memory Checkpoints | 2023 | optionalself-study | Checkpoint into the memory of other machines. Recovery time drops by orders of magnitude. |
| ByteCheckpoint — A Unified Checkpointing System for Large Foundation Model Development | 2024 | optionalself-study | A unified checkpointing system, including resharding when the parallelism configuration changes. |
| 3FS — Design Notes (DeepSeek) | 2025 | optionalself-study | Design notes for the parallel file system behind DeepSeek's training and inference pipelines. |
Direct background for the cache competition. A prefix cache is a cache: the objects have wildly different sizes and costs, the workload is skewed and non-stationary, and the eviction policy decides most of the hit ratio. These are the papers on how that problem is normally attacked, and what the standard mistakes are.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| libCacheSim — a high-performance cache simulator and library | — | extra | The simulator the competition is built on. Read the docs before writing a policy. |
| S3-FIFO — FIFO Queues are All You Need for Cache Eviction | 2023 | extra | Three FIFO queues beat LRU on hit ratio and on throughput. The clearest demonstration that "one-hit wonders" dominate real cache workloads. |
| SIEVE — is Simpler than LRU: an Efficient Turn-Key Eviction Algorithm for Web Caches | 2024 | extra | Lazy promotion in about ten lines of diff over FIFO. A good model for how simple a competitive policy can be. |
| LRB — Learning Relaxed Belady for Content Distribution Network Caching | 2020 | extra | Learn to approximate the offline optimum. Read it for the framing of eviction as prediction, and for how much machinery that costs. |
| Segcache — A Memory-Efficient and Scalable In-Memory Key-Value Cache for Small Objects | 2021 | extra | Metadata overhead is the hidden cost when cached objects are small — which is exactly the regime of a per-block prefix cache. |
| CacheLib — The CacheLib Caching Engine: Design and Experiences at Scale | 2020 | extra | What a production cache has to handle beyond the eviction policy: sizing, admission, flash, and warm restarts. |
| TinyLFU — A Highly Efficient Cache Admission Policy | 2015 | extra | Admission, not eviction, is often the lever — and a sketch is enough to make the decision. |
| Marconi — Prefix Caching for the Era of Hybrid LLMs | 2024 | optionalOct 13 | Prefix-cache eviction when entries are not uniform. The closest published analogue to the competition task. |
| Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving | 2024 | requiredOct 8 also Oct 6, Oct 13, Nov 12 |
The production view: a distributed KV cache pool with real capacity limits and real hit-ratio consequences. |
When a topic is new to you, start with one of these rather than with a random recent paper. They are also the fastest way to find a presentation paper you actually want to read.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| Ultra-Scale Playbook — The Ultra-Scale Playbook: Training LLMs on GPU Clusters | 2025 | extra | The best single entry point to distributed training, with runnable intuition instead of only equations. |
| efficient inference survey — A Survey on Efficient Inference for Large Language Models | 2024 | extra | A map of the whole Part II landscape: model-level, system-level, and hardware-level. |
| efficient LLMs survey — Efficient Large Language Models: A Survey | 2023 | extra | Broader still, covering training as well as inference. |
| full-stack inference survey — Full Stack Optimization of Transformer Inference: a Survey | 2023 | extra | Works down from the model to the hardware; good for seeing where the layers meet. |
| spec decoding survey — Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding | 2024 | extra | Taxonomy of drafting and verification strategies. |
| OSTEP — Operating Systems: Three Easy Pieces | 2023 | extra | Free, and the right refresher if virtual memory, paging, or scheduling feel distant. Several papers in this course are OS ideas applied to a GPU. |
Not readings — things to have open while working on the assignments, the project, and the exam.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| vLLM docs — vLLM documentation | — | optionalSep 22 | The serving stack most projects will build on. |
| SGLang — (source repository) | — | extra | The other stack, and the reference implementation of RadixAttention. |
| Nsight Systems — NVIDIA Nsight Systems | — | extra | Timeline profiling. The tool that shows you the gap between kernels you did not know was there. |
| PyTorch profiler | — | extra | Lower-friction profiling from inside Python; enough for most Assignment 4 measurements. |
| libCacheSim — (source repository) | — | extra | Cache simulator used by the competition. |
| ACM sigconf template — ACM Primary Article Template | — | extra | The format to use if you write up an optional project. |
Seven of the 26 meetings are student-led paper discussions, and the presentation is worth 5% of the grade. The paper discussion page has the presenter guide, the rubric, and a three-pass reading strategy; these are the two short pieces worth reading before your first paper.
| Paper | Year | In the course | Why read it |
|---|---|---|---|
| How to Read a Paper — (Keshav) | 2007 | extra | The three-pass method. Four pages, and it will save you many hours over the term. |
| Always Measure One Level Deeper | 2018 | optionalDec 1 | Ousterhout on performance measurement: the mistakes are predictable, and this lists them. |
| Orca — A Distributed Serving System for Transformer-Based Generative Models | 2022 | requiredSep 29 | Read again as a model of how to argue a systems result: one idea, cleanly isolated, honestly evaluated. |