Overview

Bring the agentic system you built in Assignment 4. At the start of the exam you are handed a task it has never seen. You have the session to make it work — accurately and fast — and everyone in the room is doing the same thing on the same hardware.

This is not a written exam. There is nothing to memorise and nothing to recite. It is a three-hour engineering sprint against a live leaderboard, and the only thing it can measure is whether you can actually drive the stack you spent a semester taking apart: profile something unfamiliar under time pressure, decide which knob matters, turn it, and know whether you helped.

Worth 20% of the course grade. In person, closed room, roughly three hours, during the registrar's December final exam period. Individual — no teams.

What happens in the room

You arrive with your Assignment 4 system, already running on the exam machine image. You are given a task suite you have not seen and a harness contract you have: same interface, same submission path, same metrics as the practice runs.

The new task

You will not see it beforehand. What is fixed, and what you can therefore prepare for, is the shape: an agentic loop over a suite of tasks, scored by an automatic checker, with a tool interface your harness has to call. What changes is everything a real deployment changes underneath you — a different tool set, a different context profile, a different mix of easy and hard cases, and a different balance between how much the model has to think and how much it has to fetch.

That is the point. A system tuned to exactly one workload is not an optimized system, it is a fitted one, and the exam is designed to tell those apart in about twenty minutes.

Accuracy and speed

Two numbers are reported for every submission, and both are on the leaderboard from the first run:

• Accuracy — the fraction of held-out tasks your system solves, as judged by the staff checker. The checker is the same one you can run against the dev split, so there are no surprises in how it grades, only in what it grades.
• Speed — mean end-to-end wall-clock time per task, measured by the harness on the exam GPU, from the first request of the agent loop to the final answer. Tool time counts. Retries count. Everything you actually spend counts.

How they combine

Standings are ranked by tasks solved per minute — accuracy divided by mean wall-clock time per task — behind an accuracy gate:

Cost in dollars is not scored here, because everyone is running on the same allocation of the same hardware; wall-clock on a fixed GPU is the cost, expressed in the unit you can actually feel during a three-hour session.

Rules

• Bring your own system. Your Assignment 4 repository, your notes, your profiling scripts, your model weights cached on disk — all of it, prepared in advance. Preparation is not cheating here, it is the exam.
• Serve your own agent. The graded agent loop must run entirely on the exam machine and the GPU allocated to you. No commercial inference APIs inside the loop — not OpenAI, Anthropic, Gemini, or a hosted open-weights endpoint. A speed contest where some contestants can buy a bigger datacenter is not a speed contest, and Assignment 3 was entirely about owning this layer.
• AI coding assistants are allowed for writing and debugging your code, under the usual disclosure requirement — state in your write-up what you used. This is a course about agents; banning them from the room would be an odd way to end it. They may not, however, appear inside the graded loop: the thing being measured is the system you serve, not the API you call.
• Individual work. No collaboration with other students during the session, in any channel. You may use any static resource: documentation, papers, your own code, public repositories, anything on disk or on the open web.
• No fitting to the held-out split. Detecting which split or which task instance is running and branching on it is disqualifying, as is hardcoding answers the checker expects. The held-out split exists to make this pointless; the staff will read the diff of anything near the top.
• Submit something that runs. A submission that crashes, hangs, or exceeds the per-task time limit scores as a failure on those tasks. Keep a working version checked in at all times and know how to roll back — the most common way to lose points here is a clever change made at 2:25.
• Measure before you claim. The write-up asks what your changes did. Numbers from the harness, not impressions.

How the 20% is earned

Most of the credit is for engineering the problem well under time pressure. Winning is worth the least of it — a careful, honestly measured mid-table run scores well, and a top-of-the-leaderboard run with a write-up that cannot explain itself does not.

How to prepare

You cannot study for the task. You can make your system easy to change, which is most of the difference on the day — and doing it well is also just how the Assignment 4 write-up gets better.

1. Make the task a parameter, not an assumption. Tools, prompts, stopping conditions, and the answer format should be configuration your harness reads, not constants scattered through it. Every hour you spend on this in November is ten minutes you do not lose in December.
2. Keep the profiler in the loop. Per-step tokens, latency, tool time, and retries, on by default, printable in one command. You will be reading it cold on a workload you have never seen.
3. Know your knobs and their costs. Batch size, quantization level, cache configuration, context budget, parallel tool calls, model size. Know which ones you can change in two minutes and which need twenty — on the day, that ordering matters more than which is theoretically best.
4. Have a rollback. A tagged commit that runs, and a one-line command to get back to it.
5. Rehearse on a task you have not used. Point your Assignment 4 system at a workload it was never tuned for and give yourself two hours. Whatever breaks in that rehearsal is what would have broken in the exam.

A practice task suite on the exam harness will be posted after Assignment 4 is due, along with the machine image, so you can check that your system starts on it well before the day.

Related

• Assignment 4 — the system you will bring, and the profile-predict-measure discipline the exam runs on a clock.
• Assignment 3 — where the self-served model underneath it came from, and the harness work that makes a small model usable.
• Cache competition — the other leaderboard in this course, run over weeks instead of hours.
• Exam policy — conflicts, accommodations, and why late days do not apply.
• AI use policy — allowed in the room, with disclosure.