Assignments
Four individual assignments. The first two run in Part I, before the systems material begins;
the last two come back to the same agent once you know enough to take its whole stack apart — first by
taking the frontier model away, then by making what is left fast and cheap — and the
final exam makes you adapt that system, live, to a task you have never seen. At 15%
each they are 60% of the grade — there is no course project, so these are the build.
Each assignment gets a class session. The meeting after each one comes due is a
student sharing session — Oct 1, Oct 15, Oct 29, and Nov 24 — where you show the
room what you built, what broke, and one number you did not expect. Nothing formal and nothing separately
graded; it counts toward
participation. Build with that session in mind:
something that runs in front of people is a better target than something that only runs in your write-up.
Assignment 1 — Use an agent (15%)
Out Sep 10 · due Sep 29, 11:59pm
Use a coding agent — Claude Code, Cursor, Codex, whichever you like — to build an app you
actually want to exist. Vibe code it: stay in the loop, but let the agent do the writing.
The app is not really the deliverable; what you noticed is. In the write-up:
- Where did the agent fly, and where did it stall?
- Which failures were the model's, and which were the harness's — missing context,
missing tools, no feedback signal to learn from?
- What did you have to know yourself in order to unblock it? Those moments are exactly where the
abstraction leaks, and they are the best preview of the rest of this course.
- Roughly what did it cost — tokens, wall-clock, dollars?
Keep your transcripts. They are the evidence for this write-up, and you will want them again later.
Assignment 2 — Design an agent (15%)
Out Sep 15 · due Oct 8, 11:59pm
Now write the loop yourself, against a commercial API — OpenAI, Anthropic, or Gemini, your choice.
No agent framework this time: you implement tool calling, context management, retries and error recovery,
and the stopping condition. A small agent you fully understand is worth far more here than a capable one
you assembled from parts.
Instrument it from the first commit: per-step token counts, latency, tool time, retries.
Assignments 3 and 4 take this exact system apart, and you will be glad the measurement is already in place
rather than bolted on afterwards.
Report three numbers. These are the assignment's headline result, and every later
assignment is measured against them — so define them now, precisely, and do not move them
afterwards:
- Success rate — the fraction of tasks your agent completes correctly, scored
against a criterion you write down before you run it. Fix a task set first: aim for twenty or
more, varied enough that one lucky prompt cannot carry the score. Keep the scoring script in the repo, so
the same tasks can be re-scored the same way in October.
- End-to-end duration — wall-clock per task, from the request arriving to the
final answer. Report the median and the p95, not the mean alone. Agent latency is heavy-tailed
— one task that stalls on a retry loop moves the mean and hides behind it — and the tail is
what a user actually feels.
- Cost — dollars per task, with the input and output token counts reported
alongside. Give both: provider prices change, token counts do not, and the two together are what let
anyone re-price your agent later.
Then say where the tokens and the wall-clock actually go — which step, which tool, which retry. You
are renting someone else's serving stack right now and cannot see inside it; note which questions you
cannot answer from the outside. Those are the questions Part II is about.
How the 15% breaks down: a working agent, loop and all, written without a framework (5%);
the three metrics above, measured over a fixed task set with the success criterion written down in advance
(6%); and the write-up, including the token and wall-clock accounting and what you could not see from
outside the API (4%).
These three numbers outlive this assignment. Assignment 3 re-measures all three after
taking the frontier model away, and Assignment 4 holds success rate fixed while driving duration and cost
down. A task set or a success criterion that shifts in between makes both of those comparisons meaningless
— and the drift is rarely deliberate, it is a task quietly reworded or a scorer loosened to stop a
failure being annoying. Freeze all three here and version the scoring script with the code.
Assignment 3 — Serve your own agent (15%)
Out Oct 8 · due Oct 27, 11:59pm
Your Assignment 2 agent works. Some of that is your harness and some of it is a frontier model quietly
covering for your harness — and from where you are standing you cannot tell which. This assignment
takes the frontier model away.
Serve an open-weight model yourself, on an engine you run — vLLM, SGLang, whichever you like —
and point your Assignment 2 agent at it. Plan on a 7–8B class model, which fits on a
single 24GB GPU. Larger is allowed if you have the hardware, but it is not the point: the smaller the model,
the more the harness has to do, and the harness is what is being graded.
- Stand it up. Get the model served behind an OpenAI-compatible endpoint and your agent
talking to it. Report everything that broke on contact — the chat template, the tool-call format,
JSON that will not parse, stop tokens that never fire, a context window your prompt no longer fits.
- Measure the drop. Same task set as Assignment 2, same three metrics, same method.
Report the success-rate delta. It will be large. That number is not a failure, it is the assignment.
- Recover it in the harness. Constrained or structured decoding, tighter tool schemas,
decomposing a step the model cannot do in one shot, retry-with-repair on malformed output, an explicit
verification pass. What you may not do: reach for a bigger model, or fall back to the
commercial API. The question is how much of the frontier model's advantage was doing work your harness
should have been doing.
- Report the recovery curve. Success rate after each change, not just at the
end. One aggregate number cannot tell you which intervention paid, and several of them will not.
Hold out tasks before you start. Split your task set in two and tune on one half only. A
harness tuned until the eval passes, then reported on that same eval, has measured nothing — and the
gap between your tuned half and your held-out half is itself a result worth writing down.
Cost and latency are worth recording here, but they are context rather than the deliverable: this
assignment is graded on getting quality back, and Assignment 4 is where you make it cheap.
How the 15% breaks down: the bring-up, with evidence it actually runs (3%); the baseline
comparison against Assignment 2 on the same tasks and metrics (3%); harness interventions with per-change
measurement (6%); and the write-up, including the interventions that did not help (3%).
Four ways to accidentally report a number that is not real. Falling back to the commercial
API on failure and still calling the result local. Changing the task set or the success criterion between
baseline and after. Tuning on the tasks you then evaluate on. Scoring a response for having the right shape
when it does not have the right answer. Name in the write-up which of these you checked for.
Assignment 4 — Optimize the full stack (15%)
Out Oct 27 · due Nov 12, 11:59pm
Assignment 3 got the quality back. Now hold quality still and make the thing cheap. You own the whole
stack — the model, the engine serving it, and the loop driving it — so every axis is genuinely
yours to move:
- Optimize the model. Quantization, a smaller model, distillation.
- Optimize the serving system. Batching, prefix and KV-cache reuse, scheduling,
speculative decoding.
- Optimize the harness. Fewer and cheaper calls, less re-sent context, earlier stopping,
parallel tool use.
Pick the axes your own measurements justify — but touch at least one serving-side axis and
the harness, so that you feel both halves of the problem. Three steps, in this order, and the
order is part of the grade:
- Profile the Assignment 3 system until every token and every second is accounted
for.
- Predict, then lock it in. Before changing anything: which axis, why you expect it to
be the biggest win, and roughly how much. Timestamped, not revisable.
- Optimize and re-measure on the same task set, same metrics, same method.
Quality is now a constraint, not a footnote. Assignment 3 established what your system can
do; that number is the line you are optimizing against. Aggressive quantization and a smaller model both
buy speed with accuracy. A 10× cost reduction that silently halves your success rate is not an
optimization, it is a different system. Measure success rate before and after on the same held-out tasks;
if you traded quality for cost, say so and quantify it. Honest trades score well. Unreported ones do not.
How the 15% breaks down: the profile (3%); the locked prediction, judged on reasoning
rather than on whether it turned out right (3%); measured improvement with quality held and reported
honestly (6%); and the write-up, including what you tried that did not work (3%).
This one does not end when you submit it. The
final exam is a
three-hour live sprint in which you bring this system, receive a task it has never seen, and adapt it under
a clock while the class competes on accuracy and speed. Build it so that the task is configuration rather
than an assumption baked through the code, and keep the profiler switched on. See
how to prepare.