This class sets up the question the rest of the term answers: for a modern AI workload, where do the time and the money actually go? We draw the stack we will spend fourteen weeks taking apart, then do the one calculation that explains why most of this course is about serving rather than training. Logistics come last, on purpose. By the end you should be able to estimate how many generated tokens it takes before inference has cost more than training ever did.
This is the first of 26 meetings. Part I (5 classes) introduces LLMs and agents from a systems point of view, so that the rest of the term has something concrete to optimize. Part II (19 classes) is the bulk of the course and works down the stack from framework to kernel to cluster to serving path, with four student sharing sessions interleaved after each assignment. Part III (2 classes) asks what LLMs can do for systems research and closes the term. Most of the numbers that motivate Part II descend from the estimate in §1.6. Tuesday (Sep 8) derives the 6ND rule that we use today on faith, and then goes on to split the 2N per generated token into prefill and decode and price them both.
75-minute class (11:15am – 12:30pm).
| Time | Segment | Notes |
|---|---|---|
| 0–5 | Open cold with the question | Write "where do the time and the money actually go?" on the board and leave it up all term. No self-introduction yet. |
| 5–13 | §1.1 What this is and is not | Say "this is not an ML course" out loud. Some students are in the wrong room and deserve to know in the first ten minutes. |
| 13–21 | §1.2 Why now | Three changes plus the dated arc. Keep it to three. |
| 21–28 | §1.3 The stack, top to bottom | Six layers on the left board, annotated with dates. Leave the drawing up all class. |
| 28–36 | §1.4 One request, end to end | Thread a single arrow down that drawing, naming the resource at each hop. No quantities. |
| — | §1.5 Reading an accelerator datasheet | Reading-only, not scheduled. Assign it with the before-Tuesday list. |
| 36–53 | §1.6 The break-even calculation | The centerpiece. Derive T = 3D live; no finished slide. |
| 53–59 | §1.7 Energy and carbon | Two conversions on the board, not a lecture on sustainability. |
| 59–69 | §1.8 How the course runs | Grading table on screen, not recited. Competition gets 90 seconds of its own. |
| 69–75 | §1.9 Expectations and this week | End on "what is it divided by?", then the before-Tuesday list. |
If running long: cut §1.7 to the kWh line, and compress §1.8 to the grading table plus "read the five pages". Never cut §1.6 — it is the only part of today that has to land.
By the end of this class you should be able to:
N parameters and D training tokens, compute where cumulative serving compute overtakes training compute, and explain why the answer is independent of N.This is a systems course. The workload happens to be large language models and the agents built on them, but the questions are ones you already know how to ask: what is the bottleneck resource, what is on the critical path, what is cached and what is recomputed, and what one unit of useful work costs. We care about time, memory capacity, bandwidth, money, and energy.
We do not ask whether a model is good. That disappoints a few people every year, so be blunt: we never ask whether a model is accurate, only what it costs to train and to serve. If you want architectures, objectives, or learning theory, this is the wrong course. It is also not a prompt-engineering course — you will write prompts in Assignment 1, but as a way to generate a workload you can measure.
By the end you should be able to read a serving paper and predict, before its evaluation section, which numbers must be true and which are suspicious; size a deployment's memory and bandwidth on an envelope; and instrument an agent so that every token and second is accounted for. You will not leave with an opinion about which model to use, because that is a question about quality.
The prerequisite is one of CS61, CS1610, or CS2620, but what matters is comfort with memory hierarchies, caches, and concurrency, plus Python and PyTorch. Machine-learning concepts get explained as they arrive; systems concepts get used as tools. A KV cache is a cache, context management is an eviction policy, prefill and decode are a two-phase pipeline, and an agent is dependent-task scheduling. That transfer is the design of the course.
Minutes: 8.
Board: Two columns, "we ask" / "we do not ask". Left: FLOPs, bytes moved, GB, ms, dollars, joules. Right: accuracy, MMLU, "is the model good". Leave it up through §1.2.
Ask the room: "How many of you came here to make a model better, as opposed to cheaper?" Let the hands go up, then say the course is entirely about the second thing.
Expect confusion: Students conflate "systems for ML" with "ML". Fixing sentence: "We treat the model as a fixed workload with known arithmetic, the way a database course treats a query."
If short on time: Keep paragraph three; prerequisites are on the site.
The arc has checkable dates. The shape of the workload was fixed in 2017, when Attention Is All You Need replaced recurrence with attention and made the transformer the computation every layer of the stack now serves. In 2020, Scaling Laws for Neural Language Models turned model size from a discovery into a line item — loss falls predictably with compute, so a bigger model became something you budget for rather than something you find. In 2022, Chinchilla corrected the recipe: the compute-optimal token budget is roughly D ≈ 20N, far more data per parameter than practice had assumed. Chat then industrialized inference, and agents multiplied the number of model calls each user action generates. That sequence is why this course exists in 2026 and would have had little to teach in 2019: the workload, the budget, and the request pattern all changed after the systems underneath them were designed.
Three things changed, and each is a systems problem rather than a modeling one.
Models outran the device. For most of deep learning's history the interesting model fit on one accelerator, and the software stack assumed it. At 2 bytes per parameter, a model with N above roughly 40 billion parameters no longer fits in the 80 GB of an A100 or an H100 — weights alone, before anything else. Past that point, sharding weights, splitting layers across devices, and moving activations over an interconnect are not optimizations; they are the only way to run at all (Oct 6, Oct 8).
Inference overtook training as the dominant lifetime cost. A training run is a large one-time expense with a known end. Serving is a bill that arrives every day for as long as anyone uses the model, and §1.6 shows how fast it wins. That inverts the priorities the field inherited from the era when a model was trained once and evaluated on a fixed test set, and it is why most of Part II is the serving path.
Agents turned one request into many dependent calls. A chatbot request was one prefill plus some decode steps. An agent request is tens or hundreds of model calls, each carrying a context that mostly repeats the previous call's, interleaved with tool invocations that have their own latency and failure modes. The unit of work became a trajectory rather than a request, so what systems people know about dependent tasks — critical paths, queueing, stragglers, caching of repeated work — now applies to a stack not built for it (Sep 15, Sep 17; Nov 12, Nov 17).
Minutes: 8.
Board: A timeline with three ticks: "model fits on one GPU" → "model does not" → "one request becomes a hundred". Anchor the left end with the publication years the body establishes — 2017 attention, 2020 scaling laws, 2022 Chinchilla — and label each tick with the lecture dates that own it.
Ask the room: "A 70-billion-parameter model in bf16 — does it fit on an 80 GB H100?" Get 140 GB said out loud, then note this is only the weights and Lecture 2 adds the rest.
Expect confusion: Students assume training is where the compute is, because that is what gets reported. Do not resolve it here; say "hold that thought" and let §1.6 do the work.
Common wrong answer: "Agents are chatbots with tools, so serving them is the same problem." Response: the context is re-sent on nearly every step, so the workload is dominated by repeated prefix processing, which chat barely exercises.
Here is the map we keep returning to: one request descending through six layers, each of which independently decides how much work gets done.
At the application layer, an agent decides how many model calls a task takes and how much context each carries — the layer with the largest available wins and the least attention paid to it. A loop that re-sends 20,000 tokens of unchanged context on each of forty steps has manufactured 800,000 tokens of prompt processing out of nothing (Sep 15, Sep 17). The framework layer — PyTorch and the engines built on it — turns a call into kernel launches and decides which tensors are materialized and which are recomputed (Sep 8, Sep 22). The kernel layer decides how many times each value crosses the boundary between on-chip memory and HBM; for attention the arithmetic did not change but the number of round trips did, and that was worth a large constant factor (Sep 29, Oct 1).
The accelerator sets a ceiling no software above it can exceed: an A100 80GB SXM offers 2,039 GB/s of memory bandwidth against 312 TFLOP/s of dense BF16 tensor throughput, an H100 SXM 3,350 GB/s against roughly 989 TFLOP/s dense. Across that generation, arithmetic got about 3.2× faster while memory got about 1.6× faster, so whether a piece of work benefits from the newer part depends on which of the two limits it (Sep 24, Oct 1). The interconnect and cluster layer carries what one device cannot hold, then handles failure at scale; its training-side material is now optional content, and the multi-replica half returns on Oct 29. The serving layer sits across all of it, deciding which requests share a batch, whether a prefix is already resident, which replica gets the request, and what precision the weights are in (Oct 6 through Nov 17).
Keep the map, because performance questions are almost always asked at the wrong layer. "Our inference is slow" is not a statement about a model: it has a different answer and a different fix at each of six levels.
Minutes: 7.
Board: Six layers as a vertical stack, top to bottom, with the units each is measured in beside it: tokens per task, kernel launches, bytes moved per FLOP, TFLOP/s and GB/s, GB/s across the link, requests/s at a latency target. Annotate with dates. Leave this up — §1.4 threads a request down it and §1.6 and §1.8 point back at it.
Ask the room: "A team says their agent is slow. Name three measurements that tell you which layer to blame." Push for tokens per completed task, time in tool calls versus model calls, achieved versus peak bandwidth.
Expect confusion: Students expect bottom-up ordering, as in an architecture course. Say that Part I goes top-down because you cannot judge a serving optimization before seeing the workload that motivates it.
Note: All TFLOP/s figures spoken aloud today are dense. The dense-versus-sparse caveat is written up in §1.5, which is reading rather than lecture; if it comes up, one sentence and point there.
If short on time: The six layers and their dates must survive; defer the A100/H100 ratio to Sep 24.
The map is six nouns. Here is what actually happens, in order, when you press enter on a chat request — one trip down the stack, naming at each hop the resource that gets consumed and the week that owns it. No quantities today; those are Sep 8's job.
A tokenizer turns your prompt into token IDs. This is CPU work, and it is the only hop in the entire trip with no accelerator in it. It costs microseconds, nobody optimizes it, and it nonetheless decides S — the prompt length in tokens — which is the unit every later hop is billed in, in memory, in arithmetic, and in money. Tokens are not words. Whitespace, punctuation, source code, and non-English text all convert at different rates, so two prompts that look equally long to you can differ substantially as far as the machine is concerned. Every estimate you make this term starts from a number this hop produced, so look at it rather than assume it.
The front end authenticates the request and puts it in a queue. No computation happens here and latency accrues anyway. How long the request waits is a policy decision rather than a physical one: how many requests the server admits at once, whether a short request may jump a long one, and which replica gets it at all. Those are Oct 6 and Oct 8, with replica choice on Oct 29. This is the part of a user's latency that has nothing to do with the model, and on a loaded server it is routinely the largest part: a serving system that is "slow" is very often a serving system that is queueing, and no kernel optimization touches that.
The scheduler admits it to a batch. The resource being allocated here is not compute — it is memory. Admitting a request means reserving room for the working state it will hold for its entire life, and that reservation, not the arithmetic, is what caps how many requests can run at once. Systems readers have met this before: it is admission control against a capacity constraint, the same shape as a connection pool or any server that keeps per-client state. Getting the policy right is Oct 15 and Oct 20; getting more requests into the same capacity is Oct 22.
Prefill. The prompt phase, to use the synonym once. The model runs one forward pass over all S prompt tokens at once. Because they all pass through the weights together, the work is large matrix-matrix multiplication, and the resource consumed is arithmetic: this is the compute-heavy phase of a request's life. The same pass writes the KV cache — for every layer, the keys and values that later attention steps will need to look back at. It is a cache in the ordinary systems sense, state kept so that work already done is not redone, and it is the single most recurring object in this course. Sep 8 prices this phase; Sep 29 and Oct 1 are about making the kernels that execute it move fewer bytes.
Decode. The generation phase. The model now emits one token per step per sequence, and each step must read essentially the entire set of weights out of HBM to produce it. The resource is memory bandwidth, not arithmetic — the same weights that prefill pushed thousands of tokens through are now read to produce one. Recall §1.2's 70-billion-parameter model, whose bf16 weights alone are 140 GB: every decode step re-reads more bytes than fit on one device. Meanwhile the KV cache grows every step, adding a key and a value in every layer, for as long as generation runs. A request therefore gets more expensive to hold the longer it runs, which is unlike most caches you have met, where the cost of an entry is fixed at insertion. Batching many sequences into a single pass over the weights is the lever that makes any of this affordable, and it is the reason the scheduler in the previous hop exists at all. Sep 8 quantifies the phase, Oct 6 and Oct 8 schedule it, Oct 13 handles the growth, and Oct 20 and Oct 22 change how many bytes "the weights" even is.
Detokenize and stream back. Each token is turned back into text and pushed to the client as it is produced, which is why output appears a piece at a time rather than all at once, and why serving has two latency metrics rather than one. Time to first token (TTFT) covers tokenization, queueing, and prefill; it is dominated by the length of your prompt and by how busy the server was when you arrived. Time per output token (TPOT) — inter-token latency, ITL, is the synonym — is the gap between the tokens that follow, and it is decode's bill, one step at a time. Nearly every decision in Part II trades one against the other: a larger batch raises tokens per second per GPU and makes somebody wait longer. Sep 8 puts numbers on both. Today only the split matters.
The request ends and gives its memory back. The KV cache is freed and the slot returns to the scheduler. Until then the request holds capacity whether or not it is doing anything, which is why an abandoned browser tab with an open stream is not free and why timeouts are a capacity mechanism rather than a courtesy.
Now run that whole trace forty times over, with a context that barely changes between calls, and you have an agent (Sep 10, Sep 15, Sep 17). Every hop repeats. S grows on each step instead of resetting; the scheduler sees a burst of related requests rather than one; and prefill re-processes a prefix it has already processed, again and again. That last observation — that most of the prompt tokens in an agent workload have been through the machine before — is the entire motivation for Oct 15 and for the cache competition. Assignment 2 makes you write that loop, and Assignment 3 makes you account for it.
Minutes: 8.
Board: Do not draw anything new. Thread a single arrow down the six-layer stack from §1.3, stopping at each hop to write the resource beside it: CPU, queue, memory reservation, arithmetic, bandwidth, network. Six labels and nothing else. If anyone asks how many milliseconds, the answer is "Sep 8".
Ask the room: "Between pressing enter and seeing the first token, how much of that time is the model computing?" Most of the room will say all of it. Queueing and prefill share it, and the split depends on load — which is why TTFT is as much a scheduling metric as a model one.
Expect confusion: Students hear decode as "prefill, repeated". Fixing sentence: prefill puts thousands of tokens through the weights in one pass, decode puts one through per pass — same weights, opposite bottleneck.
Common wrong answer: "The KV cache is an optimization you could switch off." Without it every step recomputes the whole prefix from scratch, so no practical server has the option. It is a cache in the sense that it stores reusable work, not in the sense that it is discretionary.
If short on time: Prefill versus decode and the TTFT/TPOT split must survive. The tokenizer and the teardown can be one sentence each.
This section is reading rather than lecture, and it teaches a skill you will use in every assignment and every time a paper reports a number that looks wrong. Datasheets are written to be quoted selectively. Read one in this order.
Memory bandwidth first. For the serving-dominated workloads of this course it is the number most likely to bind, because decode streams the weights past the arithmetic units once per generated token, and that is where nearly all tokens come from. The A100 80GB SXM offers 2,039 GB/s and the H100 SXM 3,350 GB/s. Bandwidth is also the hardest number on the page to inflate: it falls out of bus width and clock rather than out of a benchmark protocol or a choice of numeric format, so there is no favorable row to select. Sep 8 pairs it against the arithmetic figure to get the single number you will carry all term, and Oct 1 applies that number to real kernels.
Treat the headline throughput as marketing. The figure on page one is the 2:4 structured-sparsity number, which is exactly twice the dense figure on both reference parts: 624 against 312 TFLOP/s on the A100, 1,979 against roughly 989 on the H100. Structured sparsity requires that two of every four weights in a group be zero, in that pattern, so that the hardware can skip them — which a trained dense transformer's weights emphatically are not. Getting there means pruning and retraining and accepting whatever happens to model quality, so it is a modeling decision with a hardware payoff rather than a feature you switch on. Read the dense row. Then check which precision that row is: on the H100, FP8 dense is roughly 1,979 TFLOP/s, double the BF16 figure, and the A100 does not support FP8 at all. A comparison that quotes H100 FP8 against A100 BF16 is a 6.3× ratio (1,979 / 312) built from three changes at once, and it will be presented as one.
Check the form factor. The same silicon ships as an SXM module and as a PCIe card, and in practice they are not the same part: SXM gets the higher power envelope, the full NVLink fabric, and the memory bandwidth this course's canonical numbers assume, while the PCIe card runs at a lower power limit with reduced bandwidth and reaches its host over PCIe. Every number in these notes is for the SXM parts. When a paper's measurements look inexplicably low, form factor is a standard first thing to check. The interconnect line deserves the same suspicion — the A100's 600 GB/s (3rd-generation NVLink) and the H100's 900 GB/s (4th-generation) are useless until you know what they are per: per GPU or per link, one direction or the sum of both. A factor of two hides in each of those questions, and it starts to matter the moment a model is sharded across devices (Oct 6).
Read TDP for exactly what it is. The 400 W on the A100 and the 700 W on the H100 are board power limits — ceilings the parts will not exceed. They are not measured draws and they are not performance figures. TDP has two honest uses: multiply it by hours for a worst-case energy bound, exactly as §1.7 does with 0.7 kW, and use it to size power and cooling. What it cannot tell you is what your workload actually draws, which depends on which units are busy; a bandwidth-bound decode loop that leaves the tensor cores mostly idle does not draw what a compute-bound prefill draws. If you need the real figure, measure it, which is what the instrumentation section of Sep 17 is for.
TDP is also worth dividing into the other two headline numbers, because the result says something the datasheet does not intend to. Arithmetic per watt went from 312 / 400 = 0.78 TFLOP/s per watt on the A100 to 989 / 700 = 1.41 on the H100, a gain of about 1.8×. Bandwidth per watt went from 2,039 / 400 = 5.10 GB/s per watt to 3,350 / 700 = 4.79, a change of about −6%. One generation therefore made arithmetic dramatically cheaper in energy terms and moving bytes very slightly more expensive. If your workload is bandwidth-bound — and the decode hop of §1.4 is — the newer part buys you speed rather than efficiency. Hold on to that; it is one of the reasons Part II is as long as it is.
Finally, "80 GB" is the opening of a negotiation, not an allowance. Weights claim their share first, and §1.2's 140 GB for a 70-billion-parameter model in bf16 already exceeds one device, so that model has no single-GPU deployment at that precision — the fleet's unit of replication is at least two GPUs before any other consideration. Then subtract, in roughly this order: the driver context and the framework's own allocations, which are charged before your first tensor exists; activation workspace for whatever is currently executing, which grows with batch size and sequence length; the KV cache of every in-flight request, held for that request's entire life; and finally the blocks the allocator is holding but cannot hand out in the size the next request wants. That last one is external fragmentation in the OSTEP sense, and it is exactly why the serving systems of Oct 13 borrowed paging from operating systems. Plan against the usable figure rather than the printed one, and decide deliberately what fraction goes to KV cache, because that fraction sets the maximum batch size and the batch size sets your throughput. Sep 8 does the sizing.
One closing warning that applies to every line above. Every number on a datasheet is a ceiling under conditions the vendor chose, and none of them is a measurement of your workload. The distance between the ceiling and what you actually achieve is roughly the subject of this course, and you cannot read it off a page — you have to measure one level deeper than the layer you are trying to explain.
Minutes: 0 — reading-only, not scheduled. Assign it with the before-Tuesday list in §1.9.
Board: Nothing. If sparsity or SXM-versus-PCIe comes up during §1.3, give one sentence — "the headline number is the 2:4 sparse figure, and dense is what a transformer gets" — and point here.
Expect confusion: Students read 80 GB as a budget and size a deployment with it. In office hours, walk the subtraction in order: weights, then framework overhead, then activation workspace, then per-request KV cache, then fragmentation — and only then ask what batch size fits.
Common wrong answer: "The H100 is three times the A100, so it is three times better." It is 3.2× the arithmetic and 1.6× the bandwidth, and which one you get is a property of the workload rather than of the part. Exercise 2 makes them find that out with their own arithmetic.
If short on time: Nothing to cut — it is not on the clock. If the room is ahead of schedule, the per-watt pair is the 90 seconds worth spending.
Now the calculation that organizes the course. For one model over its deployed life, compare the compute spent training it against the compute spent serving it. Both sides reduce to a FLOP count. Training a dense transformer with N parameters on D tokens costs about 6ND FLOPs — roughly 2ND forward and 4ND backward, derived in Lecture 2. Generating one token in the decode phase costs about 2N FLOPs per sequence: one forward pass, each parameter contributing a multiply and an add.
T is the total number of tokens generated over the model's life.
Set the two equal: 2 · N · T = 6 · N · D. Divide both sides by 2N: T = 3 · D.
N cancels, so the break-even point does not depend on model size — both training and serving are linear in N. The factor of 3 has a plain reading: a training token pays for a forward pass and a backward pass, the backward being about twice the forward, so one training token costs about what three generated tokens cost.
Substitute the Chinchilla compute-optimal ratio D ≈ 20N (Chinchilla): T = 3 · 20N = 60 · N.
Sixty generated tokens per parameter. Cumulative serving compute passes training compute there, and every token after that only widens the gap.
Put numbers on it with N = 70 × 10⁹. Compute-optimal data is D = 20N = 1.4 × 10¹² tokens, so C_train = 6 × 70 × 10⁹ × 1.4 × 10¹² = 5.88 × 10²³ FLOPs, and break-even output is T = 3D = 4.2 × 10¹² tokens. Check from the other side: 2 × 70 × 10⁹ × 4.2 × 10¹² = 5.88 × 10²³ FLOPs, as required. At 300 output tokens per response, 4.2 trillion tokens is 14 billion responses; at 10 billion generated tokens per day it arrives in about 420 days. Substitute your own numbers — anything with real usage crosses this line, after which every further day of serving spends more than the training run ever did.
What the estimate ignores, and which way each omission pushes:
2N FLOPs, so counting it makes serving more expensive and break-even in generated tokens arrives sooner than 3D. Sep 8 separates prefill from decode properly.6ND for one successful run omits, and that pushes the other way.This is an order-of-magnitude argument, not an accounting statement. It survives the corrections because it is not close, and because the two largest push toward serving. That is why roughly two-thirds of this course is about inference.
Minutes: 17. The centerpiece; protect the time.
Board: Write C_train ≈ 6ND above C_serve ≈ 2NT, cancel N in front of the room, box T = 3D. Only then substitute D ≈ 20N and box T = 60N. Do the 70 × 10⁹ instantiation as a separate block — students copy numbers they watched appear. No slide with the answer on it.
Ask the room: "Before I substitute anything — does the answer depend on the model being large?" Wait for someone to see that N cancels. That moment is the lecture.
Expect confusion: Many students read T = 3D as "serving costs three times training". It is the opposite: serving is cheaper per token, which is exactly why it takes 3× as many tokens to catch up — and why catching up is inevitable. Say that out loud.
Common wrong answer: "Training dominates, because that is the number in the news." Answer with rates: training is a one-time integral, serving is a rate, and a rate beats an integral given time.
If short on time: Trim the caveats to prefill and utilization. T = 3D and T = 60N must be on the board, and 5.88 × 10²³ must survive because §1.7 reuses it.
The same FLOP count converts into hours and into joules with one multiplication each. Take the 5.88 × 10²³ FLOPs from §1.6. At 989 TFLOP/s dense BF16 and a 40% utilization assumption — an assumption, and a generous one — the effective rate is 3.96 × 10¹⁴ FLOP/s, so the run takes 5.88 × 10²³ / 3.96 × 10¹⁴ ≈ 1.49 × 10⁹ seconds, or 1.49 × 10⁹ / 3600 ≈ 4.1 × 10⁵ accelerator-hours. At $2 per GPU-hour, substitute your own rate, that is order $0.8M. An H100 SXM has a 700 W board power limit, so the same hours are about 4.1 × 10⁵ × 0.7 = 2.9 × 10⁵ kWh — roughly 290 MWh, and only the accelerator boards: no host CPUs, memory, network, storage, or cooling.
Multiply by your facility's overhead and your grid's carbon intensity — both lookupable for a specific deployment, and neither invented here — and you have an operational footprint. This belongs in a systems course rather than a policy discussion because of the framing in Sustainable AI: the footprint has an operational term from energy consumed while running and an embodied term from manufacturing the hardware, and the embodied share is too large to treat as a rounding error. One consequence people miss is that buying more accelerators to finish a job faster is not free even when energy per unit of work falls.
The practical point is simpler. Every optimization in Part II reduces FLOPs executed, bytes moved, or devices occupied per unit of delivered work, and each reduction is simultaneously a cost, a latency, and an energy reduction. Efficiency work here is sustainability work, not because we attach a moral frame to it, but because it is the same quantity over the same denominator.
Minutes: 6.
Board: 4.1 × 10⁵ h × 0.7 kW ≈ 2.9 × 10⁵ kWh. Write "boards only" beneath it and underline twice.
Ask the room: "What is missing from 290 MWh?" Collect host, network, storage, cooling, and the energy to build the machines.
Expect confusion: Students want a carbon number. Refuse, and explain why: it is a product of grid intensity, facility overhead, and hardware lifetime, none of which are properties of the model. Naming the factors is the teachable content; a fabricated total is not.
If short on time: Keep the kWh line and the last paragraph.
Twenty-six meetings: 15 instructor lectures, 7 student-led paper discussions, and 4 student sharing sessions. Lectures carry the fundamentals; the research-heavy topics — KV-cache optimization, prefix caching, quantization II, speculative decoding, agent serving V, and both Part III sessions — run as discussions, marked Students in the schedule. There is no midterm and no course project. The final is a hands-on session in the December exam period rather than a written exam: you bring your Assignment 4 agentic system, are given a task it has never seen, and adapt it live while the class competes on accuracy and speed — see the exam page.
| Component | Weight |
|---|---|
| Assignment 1 — use an agent | 15% |
| Assignment 2 — design an agent | 15% |
| Assignment 3 — serve your own agent | 15% |
| Assignment 4 — optimize the full stack | 15% |
| Final exam — live agentic optimization sprint | 20% |
| Cache competition | 10% |
| Paper presentation | 5% |
| Class participation | 5% |
| Bonus — course feedback 5%; problems current AI cannot solve, 2% each | up to 10% |
The four assignments are 60% of the grade and they tell one story. Assignment 1 (due Sep 29) builds something real with an existing coding agent and reports where it failed. Assignment 2 (due Oct 8) makes you write the loop yourself against a commercial API. Assignment 3 (due Oct 27) takes the frontier model away — you serve a 7–8B class open-weight model yourself and recover the quality you lost, in the harness. Assignment 4 (due Nov 12) holds that quality fixed and drives the cost down: profile it, commit to a prediction in writing before you touch the code, then optimize and measure against the same task set. That sequence is the course in miniature: build it, own it, understand where the cost is, then move the cost. The first two go out during Part I, within the next two weeks; take the release dates and the specs from the assignments page, which is the authoritative source for both. All four are individual.
The meeting after each assignment comes due — Oct 1, Oct 15, Oct 29, Nov 24 — is a student sharing session: you show the room what you built, what broke, and one number you did not expect. Nothing formal, nothing separately graded; it is where the 5% participation is earned alongside the discussions. Build with that session in mind.
The cache competition is worth 10% and is the one component where you compete on a shared metric. It opens Nov 19 and closes Dec 3 at 11:59pm. You submit an eviction policy for a serving prefix cache and are ranked on a held-out metric against staff baselines. Most of the credit is for doing the work well rather than for winning, so write up honestly what your policy does and where it loses. Take the rules, the metric, and the baselines from the competition page and nowhere else.
There is no graded project. If you want an open-ended build anyway, the optional project page has six worked examples, questions worth chasing, and access to compute — ungraded, no deadline, nothing depends on it. Two topics are likewise off the lecture schedule and available as optional self-study: GPU programming (architecture, Triton, tiling and fusion) and the training side of the stack. Both are listed under optional content. The GPU track is the one to work through if you want the kernel-level path in Assignment 4.
The final exam is worth 20% and is not a written exam. It is a three-hour hands-on session in the December exam period: you bring the system you built for Assignment 4, receive a task it has never seen, and spend the session adapting it while the room competes on a live leaderboard scored by accuracy and speed. Nothing about it can be memorised — you prepare by making that system easy to change and keeping its profiler switched on. Rules, scoring, and how to prepare are on the exam page.
Minutes: 10.
Board: Nothing. Grading table on screen. Do not read it aloud; point at it and name the three things students get wrong — that there is no project and the four assignments carry the weight instead, that the competition's write-up outweighs its ranking, and that the final exam is a live build session, not something you revise for. On the exam, say only this today: it is Assignment 4 against an unseen task, on a clock, and the way to prepare is to build Assignment 4 so it is easy to change.
Ask the room: "Which assignment do you think is hardest?" Most say Assignment 4. It is Assignment 3 — bring-up is where the term's schedule actually breaks. Say it now so it is not a surprise in week eight.
Expect confusion: Students assume the competition is graded on rank alone. Correct it — the write-up carries real weight — and send them to the competition page for the breakdown rather than quoting one from memory.
If short on time: Grading table, the four assignment deadlines, and "read the five pages this week".
Read before class. In the second half of the term that is not a suggestion — a discussion class with an unprepared room does not work, and everyone can tell within four minutes. "Read" means something specific here, and the paper discussion page spells it out as a three-pass strategy adapted from Keshav. Reading fewer papers more carefully beats skimming many. You are not expected to understand every detail; you are expected to arrive having formed an opinion — one thing you did not believe, one thing you would have done differently.
Participation is 5% and it is earned there, and at the four sharing sessions. One sharp question about how a baseline was configured is worth more than five comments; bring it written down, because that makes it far more likely you will ask it. AI tools are permitted throughout, under two conditions: disclose what you used, and take responsibility for what it produced. That, late days, and attendance are on the policy page, which also hosts the FAQ.
The single most useful habit to build this term: when you see a performance number, ask what it is divided by. Tokens per second, per what — per GPU, per dollar, per watt, at what batch size, at what sequence length, at what percentile? "3× faster" is not a claim until you know the denominator and the baseline. Most disagreements in this field, and most mistakes in write-ups, are a missing denominator.
Before Tuesday: read the assignments, competition, discussion, and policy pages once through — they are short, and they are the contract. The optional project page is worth a skim but commits you to nothing. Skim the Transformer paper if you have not read it. Compute is expected to come from Harvard FASRC (Cannon GPU partitions) and CloudLab, both to be confirmed; access instructions go on the optional project page once they are, so do not plan around an allocation that has not been announced. The TA and office hours are TBD and will be announced on the home page.
Minutes: 6.
Board: Write "÷ what?" in large letters beside the opening question and leave both up to the end.
Ask the room: "I tell you my kernel is 3× faster. What do you ask me?" Take answers until someone says "faster than what" and someone says "per what". Those two questions are the method of the course.
Expect confusion: Students hear "all readings optional" as "no work this week". Reframe: the reading is optional, the five pages of course mechanics are not.
If short on time: The "divided by what" habit and the before-Tuesday list.
6ND FLOPs; generating a token costs about 2N. Serving compute overtakes training compute at T = 3D generated tokens, or 60N under D ≈ 20N, independent of model size.| Quantity | Value | Source |
|---|---|---|
| A100 80GB SXM — memory bandwidth | 2,039 GB/s (≈2.0 TB/s) | NVIDIA datasheet |
| A100 80GB SXM — dense BF16 tensor throughput | 312 TFLOP/s | NVIDIA datasheet |
| H100 SXM — memory bandwidth | 3,350 GB/s (3.35 TB/s) | NVIDIA datasheet |
| H100 SXM — dense BF16 tensor throughput | ≈989 TFLOP/s | NVIDIA datasheet |
| Training compute | ≈ 6ND FLOPs | Derived in Lecture 2 |
| Decode compute | ≈ 2N FLOPs per generated token | §1.6 |
| Chinchilla compute-optimal token budget | D ≈ 20N | Chinchilla |
| Training/serving break-even | T = 3D ≈ 60N generated tokens | Derived in §1.6 |
D = 20N = 1.4 × 10¹¹, so T = 3D = 4.2 × 10¹¹ — about 420 billion generated tokens. Equivalently 60N = 60 × 7 × 10⁹ = 4.2 × 10¹¹.N?Both sides are linear in N — 6ND and 2NT — so N cancels. Doubling the model doubles the training cost and the per-token serving cost in the same proportion.5.88 × 10²³ FLOPs. What fraction of peak dense BF16 must you sustain?Peak is 1,000 × 989 × 10¹² = 9.89 × 10¹⁷ FLOP/s over 30 × 86,400 = 2.592 × 10⁶ s, so 2.56 × 10²⁴ FLOPs are available. Required: 5.88 × 10²³ / 2.56 × 10²⁴ ≈ 0.23, about 23% of peak.70 × 10⁹ × 2 = 140 GB of weights alone exceeds one device, and the driver context, the activation workspace, every in-flight request's KV cache, and allocator fragmentation are all still unpaid out of what remains. 80 GB is the opening of a subtraction, not a budget (§1.5).These want a calculator and ten minutes each. Show the intermediate quantities, not just the answer — that is the habit being trained.
D ≈ 20N to reach T = 60N. Real deployments frequently do the opposite, training a smaller model far past compute-optimal precisely so that serving it is cheap. Take a 7-billion-parameter model (N = 7 × 10⁹, as in Self-check 1) trained on D = 15 × 10¹² tokens. Where is break-even, in generated tokens and in tokens per parameter? By what factor is it further out than the Chinchilla answer, and why is T = 3D still exactly true when T = 60N is not?Solution sketch: T = 3D = 4.5 × 10¹³ generated tokens, which is 4.5 × 10¹³ / 7 × 10⁹ ≈ 6.4 × 10³ tokens per parameter against 60 — about 107× further out. That factor is just the token budget's own excess: D/N = 15 × 10¹² / 7 × 10⁹ ≈ 2,143 against 20, and 2,143 / 20 ≈ 107. T = 3D never assumed anything about D; only the 60N form did, by substituting one particular training recipe. The trade is deliberate — training is paid once, while the smaller N cuts the ≈2N cost of every token served forever.
5.88 × 10²³ FLOPs of the 70B training run into H100-hours at a 40% utilization assumption. Redo it for the A100 80GB SXM under the same assumption. How many A100-hours? Where does the ratio between the two figures come from, and why is it not the ratio of the memory bandwidths? Convert both to board-level energy and check the result against the per-watt figures in §1.5.Solution sketch: 312 TFLOP/s × 0.40 = 1.248 × 10¹⁴ FLOP/s, so 5.88 × 10²³ / 1.248 × 10¹⁴ ≈ 4.71 × 10⁹ s ≈ 1.31 × 10⁶ A100-hours. Against the H100's unrounded 4.13 × 10⁵ hours that is a factor of 3.17, which is exactly 989/312 — the dense arithmetic ratio, because this conversion never consults bandwidth at all. It assumes the run is arithmetic-limited and hides everything else inside the 40%. Energy: 1.31 × 10⁶ × 0.4 kW ≈ 5.2 × 10⁵ kWh against the H100's 2.9 × 10⁵ kWh, a factor of 1.81 — the arithmetic-per-watt ratio from §1.5, and necessarily so, since hours scale as 1/throughput and energy as hours × TDP. The load-bearing assumption is that 40% is the same number on both parts, and §1.5's ratios are a reason to doubt it.
2 × 10¹⁰ output tokens per day from a 70-billion-parameter model in bf16. Using decode's ≈2N FLOPs per generated token, how many H100s would that take if decode ran at 100% of dense peak? Name three effects from this lecture that make your answer a floor rather than an estimate, with directions. One of them is not about utilization at all — what does 140 GB of weights do to the shape of the fleet?Solution sketch: 2 × 70 × 10⁹ = 1.4 × 10¹¹ FLOPs per token; × 2 × 10¹⁰ = 2.8 × 10²¹ FLOPs per day; / 86,400 = 3.24 × 10¹⁶ FLOP/s; / 9.89 × 10¹⁴ = about 33 H100s. A floor, because prefill is uncounted (adds), decode achieves a small percentage of dense peak rather than 100% (multiplies, and Sep 8 shows this is the dominant correction by an order of magnitude), and a fleet sized to the daily mean carries no headroom for peaks (adds). The shape point: 140 GB does not fit in 80 GB, so the unit of deployment is at least two GPUs — roughly 17 two-GPU replicas rather than 33 independent ones (§1.5). The token count is known to a few percent and the utilization only to within a factor of ten, so a fleet estimate is a claim about utilization wearing a claim about tokens as a disguise.
Solution sketch: Quote the H100's FP8 dense row against the A100's BF16 dense row — 1,979 / 312 ≈ 6.3× of headroom to hide in, and the A100 cannot run FP8 at all, so the workload changed; quote a 2:4 sparse row on one side only; measure with the latency target removed, or at a batch size the A100's 80 GB could not hold; or report per node against per GPU. Ask instead for tokens per second per GPU at a fixed latency percentile, same precision, same batching policy, baseline tuned by someone who wanted it to win. The bound this lecture gives you for free: at equal precision neither peak ratio exceeds 3.17, so any per-GPU claim above that comes from something other than the two parts — a better-configured contender, a worse-configured baseline, or a changed workload — and the report should say which.
Attention Is All You Need — optional. Read §3, and within it §3.2 (attention) and §3.3 (the position-wise feed-forward network); essentially all of the FLOPs and the memory traffic live in those two. Table 1, on per-layer complexity and path lengths, is the one table a systems reader should linger on. Skip §6 — the BLEU scores are not why we read this. Question to hold: which tensors in a layer have to be kept around, and for how long?
Scaling Laws for Neural Language Models — optional. Read the introduction and the summary of findings, study the loss-versus-compute plots, ignore the appendices. Its relevance today is that these curves made compute something you budget rather than something you discover. Question to hold: in each curve, what is held fixed? The compute-optimal ratio was later revised — the D ≈ 20N we use comes from Chinchilla, not from this paper.
Sustainable AI — optional. Read for the framing rather than the figures: the split between operational and embodied footprint, and the argument that a life-cycle view changes which optimizations look worthwhile. Question to hold: which of these costs already appears in a metric your serving system reports, and which is invisible to it?
If you read only one: the Transformer paper, if you have never read it, because Lecture 2 assumes you know the shape of the computation. Otherwise the scaling laws paper, for the framing behind §1.6.
Tuesday (Sep 8) is the long one. It derives the 6ND rule from the forward and backward pass, does the training-memory accounting that shows why weights are the smallest part of a training footprint, then splits the 2N per token into prefill and decode, introduces the latency metrics and the roofline, and turns today's FLOP ratio into a serving cost model. Thursday (Sep 10) puts agents in front of you as a user — what they are, what they cost you, and how you would tell whether one worked. Sep 15 and Sep 17 turn that around to the designer's side: first the loop, the tool interfaces, and context management as a cache policy, then what agent workloads do to the serving system underneath them. Part II then descends the stack from §1.3: serving basics (Sep 22), GPU architecture and kernels (Sep 24, Sep 29, Oct 1), batching and scheduling (Oct 6, Oct 8), KV-cache and prefix caching (Oct 13, Oct 15), quantization (Oct 20, Oct 22), speculative decoding (Oct 27), routing (Oct 29), and then five straight classes on agent serving (Nov 3, 5, 10, 12, 17). Training-side topics — parallelism, ZeRO, failures at scale — are optional content rather than lectures. Part III, on Nov 19 and Nov 24, turns the relationship around.