Last class we used agents and judged them. Today we write one, which means making explicit the four decisions every framework has already made for you: what goes into the context, when to call a tool, when to stop, and what to do when something fails. The centerpiece is the third of those, because context management is not prompt engineering — it is admission and eviction against a fixed-capacity cache whose misses are silent. By the end you should be able to write the loop from scratch, price each context policy in tokens, and say from a tool call's terminal state whether retrying it is safe.

Where this sits

Lecture 2 gave the cost model for one model invocation: prefill is compute-heavy, decode is bandwidth-bound, and the KV cache is the capacity you run out of. Lecture 3 treated the agent as something a user experiences and evaluates. Today you own the code: an agent is a program issuing a sequence of invocations whose shape your assemble and invoke decide. Thursday (Lecture 5) turns around and looks at the same loop from underneath — what the request stream you just designed does to the serving system, and why a system built for independent chat requests exploits none of it.

Instructor notes — Timing plan

75-minute class (Tue/Thu 11:15am–12:30pm, SEC LL2.221).

Time	Segment	Notes
0–5	Framing	One line: "you have used agents, now you own the loop." Say Assignment 2 goes out at the end so nobody drifts.
5–17	§4.1 The loop as code	Write the pseudocode on the board line by line. Do not project it.
17–27	§4.2 Tool interfaces	Before/after schema. Keep it moving; the payoff is §4.3. Derive the 320,000 on the board.
27–47	§4.3 Context as a cache policy	The centerpiece. Five policies live, then walk the three-policy table column by column. Protect all 20 minutes.
47–56	§4.4 Control flow beyond one loop	End on "dynamically generated dependency graph" — that phrase is where Thursday opens.
56–70	§4.5 Failures, budgets, stopping, durability	The split freed time here: board the state machine as five boxes rather than assigning it.
70–75	§4.6 Assignment 2	Deadlines, the Sep 29 collision, project seed. One sentence on Assignment 3.

Reading-only, not scheduled: the three assemble bodies in §4.3 and the spiky-workload caveat after its table. Each gets one sentence in class; the detail is assigned reading.

If running long: cut §4.4 to five minutes (drop parallel tool calls, keep sub-agents as context isolation) and compress §4.5's state machine back to the single line "timeout means you learned nothing." Never cut §4.3 — it is the lecture.

Learning objectives

By the end of this class you should be able to:

1. Write an agent loop from scratch and point to the line that makes each of the four design decisions: context, action, termination, failure.
2. Rewrite a weak tool schema, and price what a single uncapped tool output costs over the remainder of a task.
3. State the five context-management policies, compute the cumulative prompt tokens each submits over a given session, and name the workload on which each one fails.
4. Decide from a tool call's terminal state whether a retry is safe, and say what a timeout does and does not tell you.
5. Describe the control-flow structure an agent generates, and say what spawning a sub-agent buys and what it costs.
6. Specify an agent's stopping condition as a safety property with two independent bounds.

4.1 The loop, as code you own

An agent is a loop. Not a metaphor for one — an actual while statement that fits on a board:

def run(task, tools, budget):
    transcript = [system_prompt(tools), user_msg(task)]
    while budget.remaining():
        context = assemble(transcript, budget)        # decision 1: what goes in
        reply   = model(context, schemas(tools))      # one model invocation
        transcript.append(reply)
        if reply.is_final():                          # decision 3: when to stop
            return reply.text
        for call in reply.tool_calls:                 # decision 2: when to act
            result = invoke(tools, call)              # decision 4 lives in here
            transcript.append(result)
    return give_up(transcript)

That is the whole architecture, and every framework you might import is a set of answers to the four annotated places. assemble decides what the model is allowed to know. The branch on reply.tool_calls decides whether the system acts on the world or keeps thinking. is_final decides termination. invoke decides what happens when the world says no.

The reason to write it yourself is not purity. The four decisions are workload-specific, so a framework must pick defaults that are wrong for somebody — and the wrongness is quiet. A framework that truncates the middle of your transcript once it stops fitting has made decision 1 for you, producing a confidently wrong answer rather than an error, and the README will not tell you. That is the argument behind Assignment 2: a small agent you fully understand beats a capable one assembled from parts, because the deliverable is not the artifact but a mental model precise enough to predict what your agent does to a GPU.

Instructor notes

Minutes: 12. Board: Write the loop one line at a time, pausing at each of the four comments to ask what could go wrong there. Do not show it as a slide — watching it get written is what makes it feel small enough to own. Leave it on a side board all class; §4.3 and §4.5 both point back at it, and Thursday opens by pointing at it again. Ask the room: "Which line of this loop does a framework most often get wrong for you, and how would you find out?" Expect confusion: Students who used a framework in Assignment 1 believe agents are architecturally complicated. The fix: "the loop is fifteen lines; all the complexity is policy inside assemble and invoke, and that is where your work goes." If short on time: Skip the framework-defaults digression; keep the four annotated decisions.

4.2 Tool interfaces are API design, with an unusual client

You have designed APIs before. The unusual part is the client: it reads your documentation at call time, every time, and no compiler checks it. The schema is the documentation, so naming and description quality change behaviour. Here is a tool as a hurried engineer specifies it:

{"name": "search", "description": "Search.",
 "parameters": {"q": {"type": "string"}}}

Search what, returning how much, and what on no match? A model given this improvises. The same tool, specified as though a stranger had to use it correctly on the first try:

{"name": "search_docs",
 "description": "Full-text search over this project's Markdown documentation. Returns at most 5
   matches, each with file path, line number, and a 200-character excerpt. Use to locate where a
   topic is documented, then call read_file for full contents. Does not search source code.",
 "parameters": {"query":       {"type": "string",  "description": "Words to match, not a question."},
                "max_results": {"type": "integer", "default": 5, "maximum": 20}}}

The second version says what the tool covers, what it does not cover, the shape of the result, and which tool to reach for next. Each sentence removes a class of failed call.

Four properties matter beyond the prose. Errors are data, not exceptions: an exception propagating out of invoke kills the loop, while a returned {"error": ..., "hint": ...} gives the model something to act on, and models recover well from a legible error message. Idempotency, because retries are routine — a tool that appends a row must tolerate the same arguments twice, or take a request key. Timeouts. And output size limits: tool output enters the context, the context is re-sent every later step, and the context is what you pay for. The multiplier is the part intuition misses: a tool that dumps 20,000 tokens early in a 20-step task — say at step 4, with 16 model calls still to come — does not cost 20,000 tokens, because under keep-until-full it rides in every one of those 16 prompts, 16 · 20,000 = 320,000 prompt tokens, more than three times the 97,000 tokens the whole task submits under §4.3's session shape. That is why the limit belongs in the tool, where the output is produced, not in assemble, where it has already been paid for at least once.

Instructor notes

Minutes: 10. Board: Put the bad schema up, take three fixes from the room, then reveal the good one and check which suggestions it contains. Faster and stickier than presenting both. Then 16 · 20,000 = 320,000 in the corner; it is the number they will remember. Ask the room: "Your tool hits a rate limit. Do you raise, or return a string? Why?" Expect confusion: The description field is treated as a comment. The fix: "it is the only documentation your caller will ever read, and it is read fresh on every call." Common wrong answer: "Just retry on error." Push back — retrying a non-idempotent write is how you get two charges on a card. Forward-reference §4.5.

4.3 Context management is a cache policy

Here is the reframe worth carrying for the rest of the term. You have a fixed-capacity store, a cost model in which occupancy is charged on every access, and a workload whose future you cannot see. Deciding what stays in the context window is an admission and eviction problem. It is a cache.

Say that out loud and the design space becomes a familiar list, each entry giving something up. Keep everything until it does not fit is not a policy: it fails abruptly on the first long task, and until then every retained token is re-sent every step. Drop oldest is FIFO, and it evicts the task description — which, being first, has the highest reuse distance and the highest value. Summarize and compact is a lossy write-back: replace k tokens with m < k, pay a model call to do it, and hope the discarded detail was not load-bearing. Retrieve on demand — transcript outside the window, fragments pulled in as needed, as RAG established — trades capacity for retrieval quality. Pin the essential — system prompt, tool schemas, task statement, current sub-goal — is what every serious agent ends up with, and it is a pinned working set.

That word is the useful import. The working set of an agent task is the subset of the transcript the next few decisions depend on, and it moves: while the agent reads a file the file matters and the earlier search results do not. A good assemble tracks the working set; a bad one tracks recency and hopes the two coincide.

The analogy breaks in one place, and the break is why to be conservative. In a CPU cache a miss is slow. Here a miss is silent: evict the constraint the user stated in turn two and nothing raises, no counter increments, and the model does not report lower confidence — it produces a fluent answer violating a requirement it can no longer see. A context bug looks exactly like a correct run, so keep what you are unsure about and log every eviction, because that log is your only evidence when the answer comes back wrong.

MemGPT takes the analogy the whole way, building a paging hierarchy around the context window; it is optional reading for Thursday and returns on Nov 10, in the agent-serving lecture on session state and memory. Note also that this is the design-side cache: a second one sits underneath, the engine's KV cache, deciding whether a re-sent prefix costs a recomputation or a lookup. Conflating the two is the classic confusion, and Thursday is where they meet.

Three assemble bodies, and what each one costs

Policies named in prose are easy to nod along to. Here are three of them as drop-in implementations of the assemble call in §4.1's loop — same signature, same transcript and budget — each short enough that the eviction decision sits on one visible line:

def assemble_keep_until_full(transcript, budget):
    context = list(transcript)
    if tokens(context) > budget.context_limit:
        raise ContextOverflow()       # refusing to decide is also a decision
    return context

def assemble_sliding_window_pinned(transcript, budget, W=3_000):
    pinned  = transcript[:2]          # system prompt + task statement, never evicted
    history = transcript[2:]
    while tokens(history) > W:
        history.pop(0)                # the silent line — nothing records what left
    return pinned + history

def assemble_summarize_and_compact(transcript, budget, trigger=3_000, target=500):
    pinned  = transcript[:2]
    history = transcript[2:]
    if tokens(history) > trigger:
        summary = model(summarize_prompt(history, target))   # a paid model call
        history[:] = [summary]        # k tokens in, ~target out, the detail is gone
    return pinned + history

To compare them fairly, fix the session shape and state it as an assumption: a 2,000-token pinned preamble, exactly 300 tokens appended per turn — the model's tool call plus the tool's result — and 20 turns. Keep-until-full submits, at turn k, the preamble plus everything appended so far, so 2,000 + 300·(k−1) tokens, and over 20 turns that totals 20 · 2,000 + 300 · (0 + 1 + … + 19) = 40,000 + 300 · 190 = 97,000 cumulative prompt tokens. Hold that figure: Thursday takes the same ledger and asks what the machine underneath does with it. The sliding window with W = 3,000 — ten turns of history — matches it until the history outgrows the window: history before turn k is 300·(k−1) tokens, which fits within 3,000 through turn 11, so turns 1–11 submit 2,000 + 300·(k−1) each, summing to 11 · 2,000 + 300 · 55 = 22,000 + 16,500 = 38,500, and turns 12–20 submit a flat 2,000 + 3,000 = 5,000, adding 9 · 5,000 = 45,000 — 83,500 in total, 13,500 below keep-until-full, about a 14% saving. Summarize-and-compact with a 3,000-token trigger and a 500-token summary fires once on this run: the check runs at the top of each turn against the history accumulated so far, and 300·(k−1) first exceeds 3,000 at turn 12, where the history stands at 3,300 tokens. Turns 1–11 therefore match keep-until-full (38,500); turns 12–20 submit 2,500 + 300·(k−12) each, summing to 9 · 2,500 + 300 · 36 = 33,300 — 71,800 for the loop. But the compaction is itself a model call whose input is the 3,300 tokens being compacted, so add 3,300 tokens of compaction traffic: 75,100 all in, about 23% below keep-until-full. A 24-turn run would trip the trigger a second time and pay again; the traffic is not bookkeeping, it is token traffic like any other. For the table's middle column, the context held at turn 20 is 7,700 under keep-until-full (2,000 + 300 · 19), 5,000 under the window, and 4,900 under compaction (2,000 + 500 + 300 · 8).

Policy	Held in context at turn 20	Cumulative prompt tokens, 20 turns	Silently lost by turn 20
Keep until full	7,700	97,000 (derived above)	Nothing — until the window fills, and then the task dies abruptly
Sliding window, pinned, W = 3,000	5,000	83,500	Everything appended before turn 10 — including any constraint stated in turns 1–9
Summarize and compact, trigger 3,000, summary 500	4,900	75,100 incl. compaction traffic	The detail inside the summary — turns 1–11 survive as 500 tokens

Read the table by columns and the point makes itself. At turn 20 the three policies differ by at most about 23% in cumulative tokens, but they differ absolutely in failure mode: abrupt overflow, silent loss of the oldest turns, lossy loss of detail inside the summary. Choose by which failure your task survives, not by the token column.

The uniform-turn assumption is doing real work in that table, and it is worth knowing which way it pushes. Actual sessions are spiky: a single tool returning 20,000 tokens at step 4 — the case §4.2 prices — does not perturb this ledger, it dominates it, and only keep-until-full re-sends the spike for the remainder of the run while both other policies bound it. So a fat-tailed distribution of turn sizes would separate the token column too, by much more than 23%. What it would not do is bring the failure column any closer together. The token column is sensitive to the workload; the failure column is a property of the policy, which is why it is the one to choose on.

Instructor notes

Minutes: 20. The centerpiece — protect this budget. The lecture split bought eight extra minutes here; spend five of them on the table and three on the working set. Board: Three columns — Policy | What it evicts first | What it gives up. Fill it live with the five policies; do not pre-print it. Then below it write "miss = wrong answer, not slow answer" and box it. Derive 97,000 and 83,500 in front of them — the 14% is the setup for the punchline that 14% is the wrong reason to choose. The assemble bodies and the spiky-workload caveat are notes-only. Ask the room: "A CPU cache miss costs you 200 cycles. What does a context miss cost you?" Wait for "a wrong answer." If nobody says it, say it and let it land. Expect confusion: This cache gets conflated with the engine's KV cache. The fix: "one decides what the model is allowed to know; the other decides whether recomputing what it knows is free." Common wrong answer: "Just summarize when it gets full." Ask what a summarizer does with a numeric constraint stated in turn two, and what it costs to run — the 3,300 tokens of compaction traffic answer the second half. If short on time: Cut MemGPT and the RAG aside, and state 83,500 and 75,100 without deriving them; keep the five policies, the working set, and the silent-miss point.

4.4 Control flow beyond one loop

The single loop decides one step at a time. The alternative is to plan first — emit the intended steps, then execute them. A plan is auditable, cheaper per step, and cacheable, but it is written before any observation arrives, so it needs replanning and you now own that policy too. Deciding step by step adapts for free but re-derives the strategy every iteration, which costs tokens.

Sub-agents are the other structural move, usually sold for the wrong reason: parallelism is a side effect. The real reason to spawn one is context isolation — the sub-agent searches forty files and returns three lines, and the thirty-seven irrelevant results never enter the parent's transcript. That buys the parent a smaller working set, at the cost of a fresh preamble per sub-agent and a return interface you must specify.

Parallel tool calls are the cheap version of the same idea: if a step's calls do not depend on each other, issue them together and pay one round of model latency instead of k. The prerequisite is knowing the dependency structure, which is the point to end on. Once you allow planning, fan-out, and conditional retries, what your agent produces is a dependency graph of model invocations, generated dynamically as it executes — the shape Ray was built for, and the reason the right vocabulary here is scheduling vocabulary: dependencies, critical path, placement. That sentence is where Thursday's class begins.

Instructor notes

Minutes: 9. Board: Draw a four-node fan-out with a join. Circle the join, ask what the critical path is, then annotate each branch "fresh preamble" to make the isolation cost visible. Leave the graph up — Thursday starts by labelling its edges with token counts. Ask the room: "You spawn five sub-agents to read five files. What did that buy, and what did it cost?" Steer from "speed" toward "the parent never sees the noise." Expect confusion: Sub-agents are believed to be a performance feature. The fix: "the win is that the parent's context stays small; the parallelism is a bonus you often cannot use, because the next call depends on this one." If short on time: Drop parallel tool calls; the dynamic-dependency-graph line must survive, since it is the handoff into Lecture 5.

4.5 Failures, budgets, stopping, and durability

Tools fail and the loop must survive it. Retries with backoff are table stakes, but the ordering matters: decide idempotency before retry policy, because retrying a non-idempotent tool is strictly worse than failing. A failure the model can see and route around is recoverable; a duplicated side effect is not.

Budgets belong in the signature, not a comment — a token budget and a wall-clock budget, both checked in the loop condition, which is what budget.remaining() does above. They are the only thing standing between a plausible bug and an unbounded bill. Which gives the framing worth keeping: the stopping condition is a safety property, not a convenience. An agent without a hard bound is a program without a termination proof, so use two independent bounds — the model asserting it is done, and an external cap that fires whether or not it does. Put loop detection between them, because the common non-termination is not a runaway but a two-step cycle: the same failing call retried with cosmetically different arguments, which a hash of (tool name, normalized arguments) over a short window catches.

The life of a tool call, as a state machine

The retry question is easier to get right if you name the states. A tool call is issued when the loop serializes the arguments and appends the call to the transcript; it is running while the tool executes; and it lands in one of four terminal states. Three of those four are routinely handled as though they were the fourth, which is where duplicated side effects come from.

issued → running. Nothing has happened in the world that you know of. The wall-clock budget is ticking, and — the serving-side fact Thursday makes quantitative — the session is holding its entire KV footprint while generating nothing. A long running state is not free just because your process is idle.

running → ok. The only decision left is size: apply §4.2's output cap here, before the result enters the transcript, because after that it is re-sent on every remaining step.

running → error-returned-as-data. The tool failed and said so legibly. Do not retry in the loop. Append the error and let the model decide — it can reformulate the arguments, reach for a different tool, or give up, and it does all three reasonably well when the message says what was wrong. The loop's job here is only to count, because the same (tool name, normalized arguments) pair failing twice is exactly the two-step cycle the loop detector above is watching for.

running → crashed-mid-write. The tool process died with its side effect partly applied: half the rows written, the file truncated, the transaction neither committed nor rolled back. Retrying is not the question; reconciliation is. Read the world back, establish what state it is actually in, and only then choose. A tool whose effect cannot be read back is a state you designed yourself and cannot recover from.

running → timeout. The deadline fired and the tool said nothing. This one gets its own paragraph.

One property decides which of these you are allowed to retry. A tool is idempotent if executing it twice with the same arguments leaves the world in the same state as executing it once. Reads are idempotent for free; writes are not, unless you make them so, and the standard construction is a request key — a client-generated identifier the tool deduplicates on, which converts the unanswerable question "did my call happen?" into the answerable one "did key k commit?". Generate the key when you issue the call and put it in the transcript, not in a local variable, so it survives the replay described below. Without idempotency the rule is blunt: retry only from states that prove no side effect occurred — connection refused, a schema rejection, a 4xx — and never from a state that merely fails to prove one.

Which is why timeout is the hard state. A timeout is a silence, and a silence has no content. Three worlds produce it identically: the request never arrived; it arrived, committed, and the response was lost on the way back; it arrived, is still running, and will commit one second after you gave up. The agent cannot tell them apart from the transcript, because the transcript records what came back and nothing came back. There is no repair at the loop level, only at the interface. Either the tool takes a request key, or you pair every write with a read that establishes ground truth and pay for the extra call, or — the minimum — you surface the ambiguity to the model as data ("send_email timed out after 30 s; it may or may not have sent") so that whatever decides next at least knows the state is unknown. Silently retrying is the one response that is always wrong, and it is the default in most retry libraries.

Durability has a clean answer: the transcript is the state. Append each entry as it is produced and recovery is replay from the last one — with the caveat that a tool may commit its side effect after you append the call and before the result, the write-ahead problem, which again wants idempotency plus a request key. Checkpointing buys crash recovery, branching, replay for debugging, and the ability to answer "what did it actually do" a week later.

Instructor notes

Minutes: 14. With the serving half moved to Thursday, this section can be taught rather than assigned — spend the extra time on the state machine. Board: Two lines first: "stopping condition = safety property" and "transcript = state". Then the state machine as five boxes — issued, running, and the four exits — with "retry ⇐ idempotent" under it, and walk the four exits asking for each whether a retry is allowed. Draw the append-then-commit window when you reach durability. Ask the room: "Your agent has been running nine minutes. What in your code was supposed to have stopped it?" Expect confusion: Budgets are believed to be a production concern. The fix: "the first time you need one is the first time you leave a loop running while you get coffee." Common wrong answer: "Timeout means it failed." It means you learned nothing — the tool may have committed. If everything else in this section is cut, that sentence stays. If short on time: Drop the write-ahead caveat and compress the state machine to the timeout box; keep "transcript is the state" and the two bounds.

4.6 Assignment 2

Assignment 2 — build an agent — goes out today, due Oct 8, 11:59pm, worth 10%, individual. The spec is on the assignments page.

Write the loop yourself rather than importing one. Four things must be yours: tool calling, context management, retries and error recovery, and a stopping condition — decisions 1 through 4 of §4.1, which is why the lecture spent seventy minutes on them. Pick a task you actually want done, because you will run it many times. The credit is in the report, which wants measurements rather than claims: where the tokens go, broken out by context component, and where the wall-clock goes, split into model time, tool time, and orchestration overhead. Thursday's class specifies those counters precisely; build them in while you write the loop rather than afterwards, because a component you never attributed is a component you cannot optimize. Say what surprised you. That instinct — account for every token and every second before optimizing anything — is what Part II builds on.

Three scheduling notes. Assignment 1 is due Sep 29, nine days before this one, so start in the week of Sep 22, not the week of Oct 1. The student sharing session on Oct 15 is where you show this agent to the room, so build something you would be willing to run in front of people. And Assignment 3 arrives the day this one is due, operating on this same agent and asking you to serve it yourself, which is the other reason to build the instrumentation in now; Thursday gives it a full briefing. If your measurements turn up something you did not expect, the first worked example on the optional project page — "instrument an agent loop and account for every second" — is where that thread goes if you want to pull it further.

Instructor notes

Minutes: 5. Board: Four dates: Sep 29 (A1 due), Oct 6 (teams), Oct 8 (A2 due and A3 out), Oct 20 (A3 due). Ask the room: Nothing. Use the time to say plainly that a small working agent with good measurements outscores an ambitious broken one, and that the measurements are graded again in a later assignment, so they are not optional discipline. If short on time: Post the deadlines and the "measure where tokens and seconds go" line as an announcement; the rest is on the assignments page. The Oct 8 double date must be said out loud.

Key takeaways

Numbers worth memorizing

Quantity	Value	Where it came from
Cumulative prompt tokens, 20 turns, keep-until-full (2,000 preamble, +300/turn)	97,000	20 · 2,000 + 300 · 190, §4.3
Same session, pinned sliding window, W = 3,000	83,500	38,500 + 45,000, §4.3
Same session, summarize-and-compact (trigger 3,000, summary 500)	75,100	71,800 + 3,300 of compaction traffic
Spread between the cheapest and dearest of the three	≈23%	97,000 → 75,100
Turn at which a W = 3,000 window first binds on that session	12	300 · 11 = 3,300 > 3,000
Context held at turn 20, the three policies	7,700 / 5,000 / 4,900	§4.3 table
Cost of one uncapped 20,000-token tool output at step 4 of 20	320,000 prompt tokens	16 · 20,000, §4.2

Self-check

Exercises

1. Two policies, one session. A session has a 3,000-token pinned preamble, appends 400 tokens per turn, and runs 24 turns. Compute cumulative prompt tokens under (a) keep-until-full and (b) a sliding window over history with W = 4,000 and the preamble pinned. Then (c) state what each policy holds at turn 24, (d) say at which turn keep-until-full would die against a 16,384-token context limit, and (e) argue which policy fails worse on a task whose success depends on a numeric constraint the user stated in turn 3. Solution sketch: (a) 24 · 3,000 + 400 · (0+…+23) = 72,000 + 400 · 276 = 182,400. (b) History 400·(k−1) fits within 4,000 through turn 11, so turns 1–11 give 11 · 3,000 + 400 · 55 = 33,000 + 22,000 = 55,000 and turns 12–24 are flat at 7,000 each, 13 · 7,000 = 91,000 — 146,000, about 20% below. (c) 3,000 + 400 · 23 = 12,200 versus a flat 7,000. (d) 3,000 + 400·(k−1) > 16,384 first at k = 35 (turn 34 submits 16,200, turn 35 would submit 16,600). (e) The window holds ten turns of history at 400 tokens each, so turn 3's append is retained through turn 13 and evicted at turn 14. The window fails worse: that loss is silent, producing a confident answer that violates the constraint, whereas overflow raises and can be handled. A 20% token saving bought a wrong answer.
2. A retry policy for a tool that must not run twice. Your agent calls post_invoice(customer_id, amount_cents), which creates a billing record. It has no request key. Median latency is 400 ms, p99 is 8 s, and your client timeout is 5 s. Design the retry policy, defend it state by state against §4.5's state machine, and name the one interface change that makes the question easy — and what that change costs. Solution sketch: A 5-second timeout sits below the p99, so strictly more than 1% of calls time out, and a large share of those have already committed — blind retry duplicates on the order of one invoice in a hundred. Policy: retry on error-returned-as-data only when the error proves no side effect (validation rejection, connection refused); never retry on timeout or crashed-mid-write; on timeout, append the ambiguity to the transcript as data and call a read — list_invoices(customer_id, since) — to establish ground truth before any further action. The interface change: accept a client-generated request_key and deduplicate on it server-side, which makes retry unconditionally safe and the reconciling read unnecessary. Cost: the tool must persist keys, and the key must be written into the transcript at issue time so a replayed run reuses it rather than minting a new one. Raising the timeout above the p99 shrinks the ambiguous window but never closes it.
3. Where the output cap belongs. Take §4.3's session — 2,000-token pinned preamble, 300 tokens appended per turn, 20 turns, keep-until-full, 97,000 cumulative prompt tokens — and change one thing: at step 4 a tool returns 20,000 tokens on top of that step's usual 300. Compute (a) the new cumulative prompt tokens and the fraction of them that is the spike; (b) the total if the tool itself caps its output at 2,000 tokens, and the ratio to (a); (c) the total if instead the cap is applied in assemble, so the spike rides at full size in the first prompt that carries it and is capped thereafter, and say in one phrase what the gap between (b) and (c) is; then (d) say what the pinned sliding window with W = 3,000 would do with this transcript, and why that is not a happy ending either. Solution sketch: (a) The spike sits in the prompts of steps 5 through 20, sixteen of them, so it adds 16 · 20,000 = 320,000 to the 97,000 baseline: 417,000 tokens, of which 320,000 / 417,000 = 77% is one tool result. (b) Capped in the tool, the extra is 16 · 2,000 = 32,000, for 129,000 — a 3.2× reduction (417,000 / 129,000) bought by one line in the tool. (c) Capped in assemble, the first carrying prompt pays 20,000 and the remaining fifteen pay 2,000: 20,000 + 15 · 2,000 = 50,000, for 147,000. The 18,000-token gap is the one prompt that had already paid — the cap arrived after the bill. (d) The window pops entries from the front of history until what remains fits, and nothing containing a 20,000-token result fits under W = 3,000, so that entry and everything before it are evicted at the next assemble: the token cost is bounded to a single prompt, and the result the agent went and fetched is gone with nothing recording that it left. Bounded cost, silent miss — §4.3's two failure modes trading places.

Reading guide

Parrot — required. Read §1–§3 carefully, but read them today for a different question than the paper is asking. Parrot's argument is a serving argument and its payoff lands on Thursday; what makes it worth reading before you write your own loop is its description of what an LLM application looks like from the outside — Table 1 characterizes real applications by how many model calls a task takes and how much of their prompt text repeats, and §4's Semantic Variable API is an attempt to submit a program's structure rather than a rendered string. Then read §6, which admits that dynamic control flow and native code cannot be offloaded: that is exactly §4.1's loop, and the paper is telling you which part of your design a serving system will never be able to see. Hold this question while reading: what does your assemble know that a request API cannot recover? Save the "so what should the scheduler do about it" question for Thursday — the whole class is about it.

SGLang — optional. The abstract and the RadixAttention section, no more. The contrast with Parrot is the point: keeping the request API means no application rewrite, but also no way to recover what the rewrite would have told the system. If you controlled the engine but not the applications, which would you build?

Looking ahead

Thursday, Sep 17, is the other half of this material: having designed the loop, we look at what the workload it generates does to the machine underneath — enormous shared prefixes across the turns of a session and the branches of a fan-out, short dependent decodes, and sessions holding KV-cache capacity while a tool runs and nothing is generated. The 97,000-token ledger from §4.3 becomes a serving-side argument there, and Parrot's proposal gets its proper reading. Part II then spends seventeen classes on the fixes: Oct 6 and Oct 8 on batching and scheduling, Oct 13 on KV-cache optimization, Oct 15 on the prefix cache, Oct 20 and Oct 22 on quantization, Oct 27 on speculative decoding, Oct 29 on routing and load balancing, and then five straight classes on agent serving — Nov 3, 5, 10, 12, and 17 — where you will meet the argument of today's required reading again, having built the loop it is trying to serve.