CS2680 Modern AI Systems: Agents and System Optimizations
Course Project

Optional Project

This project is ungraded and entirely optional. It carries no weight, no deadline, and no submission requirement — it is here for students who want to build or measure something real beyond the four assignments.

Nothing on this page is required. There is no proposal, no checkpoint, no report deadline, and no presentation slot. Skipping it costs you nothing; the graded work is the four assignments, the cache competition, your paper presentation, participation, and the final exam. What this page offers is a set of questions worth chasing, worked examples of what a good one looks like, and access to compute if you take one on.

Work alone or in teams of two or three — whatever suits the question. If you want feedback, a compute allocation, or a reader for a draft, email the instructor; that is the only process there is.

Working with AI

AI tools are expected throughout: scoping, coding, debugging, analysis, and writing. This is a course about the systems underneath these tools, and building a system without using them would be an odd way to spend the term.

Treat AI as what it is: a capable collaborator that is confidently wrong at unpredictable moments. Your job is to be the person who can tell the difference. That skill is most of what separates a trustworthy result from a plausible one.

You own everything you produce. A benchmark an AI wrote with a methodological flaw is your flaw, not its. Nothing here is graded, which means the only thing keeping a number honest is you.

If you hit a failure that generalises — something current models get consistently wrong about serving, kernels, or measurement — write it up separately. It may also qualify for the bonus.

A good project for this course does one of the following:

  • Builds a system or a substantial component — a scheduler, a cache policy, a kernel, a serving optimization — and shows it works.
  • Measures something the community assumes but has not carefully checked, and reports what you actually find.
  • Reproduces a recent paper's central result and stress-tests where it holds and where it breaks.

A negative result, honestly measured and clearly explained, is a perfectly good project. One that reports a 3× speedup that turns out to be a measurement artifact is not. Whether your numbers are trustworthy matters far more than whether they are impressive — and with nothing riding on the outcome, there is no reason to let a flattering artifact stand.

Project Ideas

These are starting points, not an assignment list. The best projects usually come from something you are already curious about. For six of these worked out in full — question, baselines, metrics, and the pitfall that would invalidate each one — see the worked examples below.

  • Characterize KV-cache reuse in a realistic multi-turn workload; quantify what prefix caching actually buys and where it fails.
  • Implement and evaluate an admission or eviction policy for a KV cache under memory pressure.
  • Measure the real cost of disaggregating prefill and decode at small scale — where is the crossover?
  • Profile a training job end to end and account for every lost GPU-second: data loading, collectives, recomputation, stragglers, checkpointing.
  • Write a fused kernel in Triton for an operator that currently is not fused, and quantify the gain against a roofline prediction.
  • Evaluate speculative decoding across draft-model choices and workloads; find where the acceptance rate makes it a loss.
  • Build a checkpointing scheme and measure recovery time against failure rate, rather than reporting only checkpoint overhead.
  • Compare quantization methods on the axis papers usually skip: not accuracy, but throughput, latency tails, and memory under real serving load.
  • Study how batching policy interacts with latency SLOs when request lengths are heavy-tailed.
  • Anything from the reading list that made you think "I do not believe that number."

Example Projects

How to read these

Six worked examples, at roughly the scope a two-to-three person team can finish in a term.

These are not a menu. They exist to show what a well-shaped project looks like: a question narrow enough to answer, a baseline worth beating, and a metric that would embarrass you if it were wrong. The best projects in a course like this usually come from something you were already curious about — these are calibration, not assignment options. Picking one as written is allowed, and you will be held to the same standard as anyone who invented their own.

Notice what every example has in common: each one names the number that would change your mind. That is the part students most often leave out, and the first thing to check against what a good one looks like.

1. What does an agent loop actually cost?

Part I & agent serving — instrumentation over invention

The question

  • For a real agent doing a real task, where do the tokens and the wall-clock seconds go? Model forward passes, tool calls, retries, or re-sending context that has not changed?

What you would build

  • Instrument the agent from Assignment 2 (or a framework agent) to log per-step token counts, latency, cache hits, and tool time.
  • Run it over a workload of 50–100 tasks with varying difficulty and trajectory length.
  • Attribute every second and every token to a component. Nothing unaccounted for.

Baselines and metrics

  • Cost per completed task, not per request — a cheap step that fails and retries is expensive.
  • Fraction of prompt tokens that are re-sent context versus genuinely new.
  • Tail latency across trajectories, not just the mean; agents are heavy-tailed.

What a good result looks like

A breakdown that surprises someone who builds agents — for instance, showing that most tokens are re-sent context rather than reasoning, and quantifying what prefix caching would recover.
Pitfall: measuring against a hosted API where you cannot separate queueing from compute. Either run the model yourself or be explicit that your latency numbers include someone else's scheduler.

Starting points

2. What does prefix caching actually buy?

Prefix cache — a characterization project

The question

  • Reported prefix-cache hit rates come from workloads chosen by the systems' authors. On a workload you construct honestly, what is the hit rate — and where does it collapse?

What you would build

  • A workload generator with controllable sharing structure: shared system prompts, multi-turn conversations, few-shot prefixes, and adversarially unshared traffic.
  • Measurement harness against a real engine with prefix caching on and off.

Baselines and metrics

  • Hit rate, time-to-first-token, and throughput — caching helps TTFT far more than throughput, and reporting only one hides that.
  • Sensitivity to cache capacity: the interesting regime is when the cache does not fit.

What a good result looks like

A curve of benefit against sharing rate with the crossover identified, plus a clear statement of the workload assumption under which published numbers hold.
Pitfall: building a workload that shares prefixes by construction, then reporting the resulting high hit rate as a finding. Your generator's sharing structure is the result.

Starting points

3. A fused kernel against its roofline prediction

GPU programming and kernels — build and justify

The question

  • Pick an operator sequence that is not currently fused. Predict the speedup from a roofline model first, then write the kernel and see whether the prediction holds.

What you would build

  • A Triton kernel fusing the sequence, correctness-checked against the unfused reference.
  • A written arithmetic-intensity analysis committed before you benchmark.

Baselines and metrics

  • The unfused PyTorch path and, where one exists, the vendor library implementation.
  • Achieved bandwidth and FLOP/s against the hardware's roofline, not just wall-clock speedup.

What a good result looks like

Prediction and measurement side by side. If they disagree, explaining why is a better project than if they had agreed — that gap is where the real behavior lives.
Pitfall: benchmarking without warmup, without CUDA synchronization, or at a single shape. Report a shape sweep; fusion wins are extremely shape-dependent.

Starting points

4. Where does the GPU time go in a training step?

Training and data pipelines — accounting for every lost second

The question

  • On a multi-GPU training job, what fraction of GPU-seconds is spent on useful compute — and where does the rest go: data loading, collectives, recomputation, stragglers, checkpointing?

What you would build

  • A profiled training run with a per-phase breakdown, then targeted interventions (prefetch depth, checkpoint interval, recomputation policy) with before-and-after numbers.

Baselines and metrics

  • Model FLOP utilization, not "GPU utilization" — a GPU spinning on a memory-bound kernel reports high occupancy while doing very little useful work.
  • Step-time distribution across ranks, which is where stragglers hide.

What a good result looks like

A budget that sums to 100% of wall-clock, with the largest recoverable slice identified and at least one intervention measured end to end.
Pitfall: conflating occupancy with utility. Say precisely which metric you mean and how you obtained it.

Starting points

5. Where does speculative decoding stop paying?

Speculative decoding — find the crossover, not the win

The question

  • Speculative decoding is reported as a straightforward win. Under what batch size, acceptance rate, and draft-model cost does it become a loss?

What you would build

  • A harness sweeping draft-model size, speculation depth, batch size, and workload domain.
  • A simple analytic model of expected speedup, checked against measurement.

Baselines and metrics

  • Standard autoregressive decoding at the same batch size — the comparison is often made at batch size 1, where speculation looks best.
  • Acceptance rate per domain, throughput, and latency together.

What a good result looks like

A phase diagram over batch size and acceptance rate showing where the technique wins, ties, and loses — useful to anyone deciding whether to deploy it.
Pitfall: reporting speedup at batch size 1 only. Under server-side batching the arithmetic changes completely, and that is the regime production runs in.

Starting points

6. Quantization under serving load, not accuracy benchmarks

Pruning and quantization — measure the axis papers skip

The question

  • Quantization papers report accuracy retention and sometimes a microbenchmark. Under realistic serving load, what do the methods actually deliver in throughput, tail latency, and memory headroom?

What you would build

  • An end-to-end serving comparison of two or three quantization methods at matched accuracy, under a request workload with realistic length distribution.

Baselines and metrics

  • Full-precision serving at the same batch policy, plus at least one competing method — not just quantized-versus-not.
  • P50 and P99 latency, sustained throughput, and how much larger a batch the memory saving buys.

What a good result looks like

Evidence about whether the memory saving converts into serving capacity, which is the claim practitioners actually care about and the one most often left unmeasured.
Pitfall: comparing methods at different accuracy levels. Match accuracy first, then compare systems metrics, or the comparison means nothing.

Starting points

What these have in common

A question, not a topic. "KV-cache optimization" is a topic. "At what sharing rate does prefix caching stop paying for its memory?" is a question, and you can tell when you have answered it.

A baseline someone would defend. The weakest part of most project reports is a baseline chosen because it was easy to beat. Pick the one a skeptical reader would demand.

A metric tied to a decision. Every example above measures something that would change what an engineer does on Monday.

A stated failure mode. Each has a pitfall that would invalidate the result. Knowing yours in advance is most of the work — see the paper reading strategy for the same habit applied to other people's papers.

Related

What a Good One Looks Like

Nothing here is graded, so read this as a self-check rather than a rubric — the questions a reviewer would ask, which are the same ones you should ask yourself before you believe your own numbers.

Dimension Worth doing Warning sign
Problem & Motivation A sharp, well-scoped question that matters, situated against what has already been done A vague question, or one already answered in a paper you have not read
Technical Execution Real implementation or measurement effort; sound engineering; someone else could reproduce it Code that only works on your machine, on one input, once
Evaluation Quality Right baselines, right metrics, error bars, ablations; artifacts anticipated and ruled out Single-run numbers presented as fact, or a baseline chosen because it loses
Reporting Clear writing, figures that support the claims, explicit about limitations Figures that look impressive but do not show what the text says they show
On honesty: every strong systems paper has a "where this does not work" paragraph. Write yours first — if you cannot, you do not yet understand what you measured.

Pacing and Scope

There are no milestones and no due dates. That freedom is also the main hazard: an open-ended build with no deadline is the easiest thing in the world to start in the last two weeks of term and never finish. Two suggestions, offered rather than imposed.

  • Pick the question before you pick the code. Write down, in one paragraph, what you will measure and what result would change your mind. If you cannot, the project is not scoped yet.
  • Get one end-to-end number early — a bad one, on a small input, is fine. A project that produces its first measurement in the first two weeks almost always finishes; one that spends a month on infrastructure first usually does not.

Scope to the hardware you can actually get. Many of the best questions on this page fit on a single GPU, and queue time on a shared cluster is real time.

If You Write It Up

No write-up is required or collected. If you want one anyway — as a portfolio piece, a workshop submission, or a draft for the instructor to read — the conventions below are the ones a systems audience expects.

  • The ACM sigconf template (LaTeX or Word), two-column, 10pt, six to eight pages excluding references.
  • Figures legible in print and in grayscale. Label your axes, including units.
  • Report the hardware, software versions, and workload for every number you present.
  • A repository with a README that explains how to reproduce your main result, plus the plotting scripts and raw measurements behind your figures.
  • An AI-use note, per the AI use policy: what you delegated and what you kept, one place AI was clearly better than you, and one place it misled you, how you noticed, and what it cost. Anyone claiming AI never led them astray over a whole term has either not looked or not been honest.

Compute Resources

These are the compute options for the whole course — the assignments, the cache competition, and this optional project alike. Access instructions will be posted here once allocations are confirmed. If you need hardware beyond what is listed, talk to the instructor early — sometimes it can be arranged, but not in the last two weeks.

Harvard FASRC (Cannon)

The FAS Research Computing Cannon cluster has GPU partitions available to Harvard courses. Plan for queue time: a job that takes an hour to run may wait considerably longer than that to start, which matters a great deal in the last week of the term.

CloudLab

CloudLab provides bare-metal machines for research and teaching, free of charge, with full control over the software stack. This is the right choice when your project needs to touch the kernel, the network, or the storage layer, or when you need reproducible isolation that a shared cluster cannot give you.

Other options

  • Your research group's cluster, if you have one and your advisor agrees.
  • Cloud credits — check with the instructor before spending your own money on this course.
  • A single consumer GPU is enough for many good projects. Scope to what you have.

AI Resources

Useful starting points for project work:

  • vLLM and SGLang — production-quality serving engines, and readable enough to modify.
  • Triton — write GPU kernels without writing CUDA.
  • Nsight Systems and the PyTorch profiler — find out where the time actually goes before you optimize.
  • The Ultra-Scale Playbook — a practical guide to distributed training that pairs well with the lectures.
  • MLPerf — for how the field defines and reports performance, and for baselines worth arguing with.