How to run the session
Week 11 puts prices on everything the course has built. Week 10 ended with an agent that retries, checkpoints, keeps receipts and survives its own death; every one of those habits costs tokens and time, and none of them has been priced yet. Chapter 19 (Cost, Latency, and Performance) reads the two meters every run carries, the invoice and the clock, and ends on the trade between them and quality. Chapter 20 (Deploying and Scaling) takes the agent behind an endpoint, where strangers send it work at their own rate: how the answer finds its way back, where a run lives when its process dies, how traffic fits through the provider’s limits, and how a behavior change reaches users without betting the product on it. Both chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow.
The deck has 34 slides in four parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#31 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (the napkin: what a ten-step run costs against one chat call), slide 11 (three users who all say “it’s slow”), slide 18 (a four-minute task behind a load balancer) and slide 23 (launch week, and the provider starts refusing calls). Give each a minute.
Slide 4 plays the explainer Why agent cost compounds (about four minutes). Check before class that it plays in the room, with sound.
The exercise uses four browser tools: the agent cost-per-task estimator, prompt caching savings, the queue and backpressure simulator and the rollout plan generator. Every link in the exercise opens its tool with the inputs already filled in. Check before class that the links open with their numbers: the estimator link should show 157,080 tokens per run. All four tools run in the browser, need no account and no API key, and send nothing anywhere. None of them knows any provider’s prices or limits: every price and limit in this week’s material is an illustration, labeled as one, and the tools ask for your own.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the napkin, objectives | 1–3 | 4 |
| The invoice: the explainer, the staircase, what the napkin leaves out, yield, five levers, prompt caching | 4–9 | 15 |
| The clock and the triangle: three complaints, prefill and decode, the felt wait, parallelism and the other levers, pick two, on purpose | 10–16 | 10 |
| Serving: the wall, take a number, a row and not a process, rooms by the thousand, launch week, two axes and the door | 17–25 | 13 |
| The rollout ladder: Tuesday’s edit, the rungs, the canary’s disciplines, the flag and the artifact | 26–30 | 8 |
| In-class exercise: price it, cache it, admit it, ship it | 31 | 26 |
| Homework, recap | 32–34 | 4 |
The plan fits the same 80-minute slot as weeks 1 to 10. The exercise is teams of two or three: about 8 minutes for Part A, 5 for Part B, 6 for Part C and 5 for Part D, with 2 for the debrief. Every link opens prefilled, so the minutes go to reading results and arguing about them, not typing.
For a 75-minute slot, assign the explainer as viewing before class and give slide 4 one sentence (4 minutes), and cut slide 22 to its title (1 minute); the debrief stays. For a 90-minute slot, give Part A’s question 4 its own 8 minutes in the room (each team opens one of its own Lab 6 traces, writes down the input tokens of every model call, and fits F and g by hand before entering them in the estimator), with a 2-minute debrief. In the 80-minute plan, question 4 goes into the note after class. It is the first hour of the homework either way.
Common misconceptions and how to address them
“The goal is to spend fewer tokens.” The goal is to waste fewer. Chapter 19 quotes a production write-up in which token spend by itself explained most of the variation in how well a research agent did. Tokens are what capability is bought with; the chapter’s aim is “yield, never abstinence”, and its question is what share of the tokens you spend do work.
“Prompt caching means we can stop trimming context.” A cached token is cheaper on the invoice and exactly as present on the desk. The model still reads all of it, and the attention problems of a crowded context do not care what the token cost. The chapter’s line: “Cheap clutter is still clutter.”
“The cache works or it doesn’t; the order of the prompt is a detail.” The cache reuses the front of the prompt up to the first token that differs from the previous call. One small changing value near the front ends the match there, and everything after it is read at full price. The order is the whole mechanism.
“Running the work in parallel makes the agent cheaper.” Parallel workers shorten the wall clock and multiply the bill. The chapter’s lesson is that parallelism buys speed and breadth, never efficiency; use it when someone is waiting or a deadline is real, and know which meter you are paying with.
“Streaming makes the agent faster.” It makes no step faster. It moves the first evidence of progress to the start of the answer, which changes how the wait feels, and it can hide a slow backend from the people who should be fixing it.
“The triangle means every improvement costs something.” Only after the waste is gone. While a system still re-reads context nobody uses, retries nobody investigated, or runs a strong model on clerical steps, removing that waste improves cost and latency without touching accuracy.
“Raise the timeouts and long tasks will work.” The limit sits in machinery you do not own (load balancers, serverless platforms, phones that put idle connections to sleep), and raising every timeout you can find only moves it. The fix is a different shape of conversation, not a longer one.
“Webhooks are the grown-up version of polling.” They are the faster one. They get dropped by firewalls, fail silently when the receiver is down, arrive out of order on retry, and need a reachable server on the client’s side. Polling is the source of truth; a webhook only saves an interval.
“A queue protects the service in a rush.” A queue absorbs a burst. During a rush that outlasts it, a queue that accepts everything fills with tasks that wait past their callers’ patience and are never started. Protection is an admission policy: refuse at the door, in words, past a chosen depth.
“A green suite means the change is safe to ship.” The suite tests the failures someone already imagined. A model swap or a prompt restructuring changes behavior on inputs nobody wrote down, and the damage arrives days later as support tickets, not as an error at deploy time.
“Rolling back means changing the prompt back.” Editing in place leaves no exact previous version to return to. Each change ships as a new immutable version over code, prompt, model settings and tool schemas; rollback points the flag at the previous version, which still exists unchanged.
Materials and notes for instructors
All eight figures the syllabus names for this week are on the slides and available individually on the diagrams page: the cost staircase (slide 5), the prompt-caching prefix (slide 9), the streamed and buffered wait (slide 13), the triangle (slide 15), the thirty-second wall (slide 19), the two serving shapes (slide 20), the backpressure queue (slide 25) and the rollout ladder (slide 28). Slide 21 adds the Chapter 20 figure of a run surviving its own process, which the syllabus does not list. Their inner labels are small when projected; each opens full size from the diagrams page, worth a second browser tab for slides 20 and 25.
Numbers on the slides are the book’s or a tool’s, and are labeled. From Chapter 19: the napkin run (a standing prefix of about 3,000 tokens and a 200-token task, steps that each add about 1,000 tokens, ten steps), its 32,000 and 77,000 input tokens and its roughly half a million at thirty steps; the retry range of ten to twenty percent, a cost-governance write-up’s illustrative range. From Chapter 20: the duration heuristic (under ten seconds, up to an hour, beyond), the canary slices of a percent and a tenth of a percent, all of which the chapter calls illustrations. From the tools and the formula: slide 5’s 531,000 is the estimator’s arithmetic on the napkin, and its two crossover points are the formula’s (the 8-step one checks on the estimator’s slider); slide 9’s cached share and billed figure are the caching calculator’s at its pre-filled factor of 10 percent; slide 24’s ceiling is the queue simulator’s made-up round limits. No slide shows a price or a provider’s limit.
Chapters 19 and 20 cite research papers, cost-governance write-ups and engineering blogs by name, and Chapter 20 names a microVM project and durable-execution engines as examples. The deck describes the categories and the findings without naming any product, model, provider or framework.
The syllabus lists a second explainer, on the rollout ladder, which the site does not have yet. Slides 26 to 30 and Part D of the exercise cover the same ground, and the blog post Shadow Deploy an AI Agent, Then Canary: The Rollout Ladder is optional reading in its place; its sections “How big should the canary be, and how long should it run?” and “What if your traffic is too low for a canary?” are the ones Part D draws on, about ten minutes of reading. For the cost side: How Much Does an AI Agent Cost to Run? and How to Reduce AI Agent Costs: Six Levers, Ranked by Return. Week 10’s reading on durable execution is the background for slide 21.
Key terms in the free glossary: token, prompt caching, budget, durable execution, idempotency, shadow deploy, canary and feature flag. Prefill, decode, time to first token, backpressure, admission policy and the immutable artifact are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 11, 18 and 23) with what to listen for, the expected results of the exercise, the homework key and its reference script.
Exercises
In class: price it, cache it, admit it, ship it (teams of two or three, 26 minutes)
Four tools, one agent. The order desk answers customers’ questions about their orders: it looks up the order, checks the carrier’s tracking, and sends the customer a reply. Its traces show a first model call that reads 5,400 tokens (a 600-token system prompt, 4,200 tokens of tool definitions and 600 of instructions and task), then steps that each add about 700 tokens to the transcript (250 written by the model, 450 returned by a tool). A typical run takes 14 steps. Retries add about 10 percent. Every price and limit below is an illustration, not any provider’s.
Each link opens its tool with the inputs filled in. Write down what the tool shows, then answer the questions in a sentence or two each. Where a question asks for something the tool does not print, it says so, and gives you what you need to work it out.
Part A, the bill (8 minutes). Open the cost estimator with the order desk’s numbers (illustrative prices: 1 per million input tokens, 4 per million output tokens). The estimator labels its money figure with a dollar sign, and its export says “USD”: that is only how the tool displays a figure, whatever rates you enter. At the illustrative rates it is a number of illustrative units, not money. In your note, report cost in tokens and in illustrative units (for example “0.05 units per run at 1 and 4 per million”); a figure copied from the tool with its sign is accepted if you label it illustrative.
- Write down the tokens per run (in and out), the money per run at the illustrative prices, the naive estimate, the share of the input that is history being re-read, and the input-to-output ratio.
- The tool does not print the step at which the history overtakes the prefix. Work it out from F and g: at step k the model re-reads F tokens of prefix and g·(k−1) of history. Which is the first step whose history is larger than its prefix? On the staircase chart, check that the history part of the bars grows past the prefix part around there. Then move the step slider: at what run length does history become more than half of the lead agent’s input? What does that say about which lever matters most for this agent at its typical length?
- Compare two levers at 14 steps and again at 40 steps, one at a time: tick Cache the stable prefix (cached tokens at 10 percent of the normal rate, an illustration), and, separately, trim each tool result from 450 to 150 tokens. Write down the money per run for each. Which lever wins at 14 steps, which at 40, and why does the answer move?
- Your project: from one of your own Lab 6 traces, take F as the input tokens of the first model call, g as the average growth per step (the last call’s input tokens minus the first’s, divided by the number of steps minus one) and n as the run’s steps. Enter them, with the same illustrative prices, and give the money per run, the history share, and the step at which history overtakes your prefix. (In the 80-minute class this question goes into the note, after class.)
Part B, the cache (5 minutes). The order desk’s system prompt starts with the current date and time, so the agent can tell customers how long ago an order shipped. Open prompt caching savings with the order desk’s prefix and the timestamp at the front.
- Write down the cached share and the billed token-equivalents with the timestamp at the front, and after moving it to the tail. What does the move save per run?
- In Part A, ticking the cache gave a different saving on the same agent. Compare the two tools’ figures and explain the difference from what each one assumes is cached (each tool says what it models on its page). Which figure would you put in a budget, and what would you check first?
- A teammate proposes to cut the 4,200 tokens of tool definitions by sending only the tools relevant to each step, about 1,200 tokens, a different subset on each call. Model it in the tool (choose the position and the sizes that describe it) and compare the billed input with the full, stable tool block at the tail. Would you accept the proposal? Give one reason on the invoice and one about the context.
Part C, the door (6 minutes). The order desk goes live behind the async job pattern. Open the queue and backpressure simulator with its traffic: a run spends Part A’s 157,080 tokens in 15 model calls and takes 90 seconds; the illustrative provider limits are 2,000,000 tokens and 1,000 requests a minute; 8 tasks a minute arrive, tripled for an hour.
- Write down the ceiling in runs per minute, which axis binds, what the concurrency cap allows, and the cap that would match the ceiling.
- Compare the two policies in the rush: tasks started, refused at the door, accepted and abandoned, the deepest queue and the longest wait. Accepting everything starts more tasks than refusing at the door. Is it the better policy? Answer from the point of view of a caller in each column.
- Apply Part A’s trim (tool results of 150 tokens) by entering that run’s tokens per run. Write down the new ceiling and the abandoned count when accepting everything. Why did the abandoned count barely move, and what else must change to collect the gain? Name one effect of the trim on a real run that this model leaves out.
- Tick Lanes. Who waits now, behind the door and without it? Name one piece of the order desk’s work that belongs in the background lane, and one that could leave the queue entirely for a discounted batch tier.
Part D, the ladder (5 minutes). The team wants to swap the order desk’s model. It has a regression suite, no feature flag and no recorded traffic, and serves about 1,200 sessions a day. Open the rollout plan generator for that change.
- Write down how many rungs are required and the warnings the tool gives.
- The shadow runs every session twice. Using Part A’s money per run, what does one week of shadow add to the bill at 1,200 sessions a day? (The tool does not compute this.) What would make the real figure differ?
- At the canary’s first slice, how many days does it take to collect 300 canary sessions (an illustrative target)? Say what you would do instead, and give one reason a wider first slice is acceptable for this agent and one reason it is not.
- Write three rollback thresholds for the canary as “canary within ___ of the incumbent”, each for a different metric family the tool lists, using numbers you have from Parts A and C where they apply. Who or what acts when one is breached?
- Part B’s fix (moving the timestamp to the tail) is itself a change. Is it a small prompt edit or a prompt restructuring? Pick the tool’s option, write down the rungs it requires, and say which of the four parts of the artifact changes and what happens to the version id.
The last 2 minutes are the debrief.
Submit: the Markdown exports of the four tools (the estimator after Part A question 4 with your project’s numbers, the caching calculator after question 3, the simulator from its link, and the rollout plan after question 5), and a short note answering every numbered question above. The note can be finished after class, within 24 hours; the 26 minutes are for reading the tools.
Acceptance criteria:
- Part A. The note gives the tool’s figures for the order desk, works out the first step at which history exceeds the prefix from F and g (with the arithmetic shown), and gives the run length at which the history share passes half as the slider shows it. It compares the two levers at both lengths with the money figures and explains the change from the formula’s two terms. The team’s own F, g and n come from a named trace, with the step at which history overtakes the prefix.
- Part B. The timestamp and tail figures are as the tool shows them. The comparison with Part A names what each tool caches and one more assumption that differs, and says what to check before budgeting. The dynamic-tools proposal is modeled with a position that matches it, compared in token-equivalents, and decided with one reason on each side.
- Part C. The ceiling, the binding axis, the cap and the two policies’ figures are as the tool shows them. The policy answer weighs a refused caller against an abandoned one. The trim answer names the limit that took over and what to change, and one effect the model leaves out. The lanes answer says who waits and names background and batch work of the order desk.
- Part D. The rungs and warnings are as the tool shows them. The shadow’s week is computed from Part A’s figure, with one reason the real figure differs. The canary days are computed, and the answer chooses a course with a reason each way. Three thresholds cover three metric families and say that the rollback is automatic. The timestamp move is classified with a reason, its rungs recorded, and the changed part of the artifact named with a new version id.
- The tools’ verdict lines apply rules that are the tools’ own (each tool says which of its rules are the book’s). Where the team’s reading differs from a verdict line, the note says why in one sentence; a justified overrule earns full credit.
Homework: the meters (teams, about 6 hours, in two sessions)
There is no lab this week. The homework turns your project’s traces into the three meters Chapter 19 assumes you already have (cost per run, latency percentiles, step counts), and uses them to set two things the earlier labs left as guesses: the step budget and the trade your agent makes.
Run your project’s agent on its benign task set (Lab 7’s tasks, or at least 20 runs) with Lab 6’s tracer on, against the scripted model client or a local open-weight model; no paid API is needed. The scripted client answers in a fraction of a millisecond, so its runs carry cost and step data but no latency signal. For item 3, do one of these and say which: run at least ten of the tasks against a local model and take the latency figures from those runs; or add a stated per-call latency to the scripted runs (for example a fixed time plus a time per input and per output token) and label every latency figure as modeled; or, if neither is possible, report that your runs carry no latency signal and state the latency corner in item 6 as an explicit assumption. Write a small script, standard library only, that reads the trace files and computes the following.
- Cost per run. Input and output tokens per run, and money at illustrative rates you state (use the exercise’s 1 and 4 per million unless your instructor sets others). Report the mean, the median, the 95th percentile and the maximum, and say which percentile method you used.
- The staircase, measured. For each run, F (the input tokens of the first model call that succeeded), g (the average growth per step) and n. Compare the formula’s input total with the measured total and explain the gap. Give the step at which the re-read history overtakes the prefix for your median run, and put the median run’s F, output per step, tool result per step and n into the cost estimator: does its total match your measurement?
- Latency. The wall-clock time of each run, with its 50th, 90th and 95th percentiles and the maximum. Attribute each run’s wall clock to model calls, tool calls and the rest, without counting overlapping calls twice. Say which lever the split points to.
- Steps. The distribution of step counts (a histogram is enough), separating runs that succeeded from runs that stopped at your step cap, runs that failed (Lab 6’s
crashedorreplay_miss) and runs whose trace is incomplete. A model call that errored and was retried is a retry, not a step: count retries apart. - The step budget. Set it from the distribution of successful runs plus a margin, and say why that margin. Say how many of your recorded runs the budget would have stopped, and what the runs that hit your old cap tell you (and do not tell you) about how long they needed.
- The triangle statement. One paragraph per user-facing feature of your agent (at least one): which corner it sacrifices, how far it may slide, and which reading on which meter triggers a rethink, with the numbers from steps 1 to 5.
Submit: the script and its tests; a short report (Markdown) with the tables and the histogram, the estimator export for your median run, and the triangle statement.
Acceptance criteria:
- Reading traces. The script reads your project’s trace files as the tracer writes them. A test shows that an incomplete trace (a process killed mid-run, with a half-written last line) is reported as incomplete and left out of the distributions, not counted as a short run, and that a damaged line in the middle of a file is an error. Tests also show that a crashed run stays out of the budget and the median run, and that a model call that errored and was retried counts as a retry, not a step, with F taken from the first call that succeeded.
- Arithmetic. Tests check the staircase formula and the crossover step against hand-worked cases (Chapter 19’s napkin among them), the percentile method on a small known list, and that two tool calls running at the same time are not counted twice in the time attribution.
- Money. Nothing has a default price: with no rates given, the report shows tokens only. The rates used are stated as illustrative.
- Latency. The report says where its latency figures come from (a real model, or a stated model of latency, labeled), or says that the runs carry no latency signal.
- The budget. It comes from successful runs; failed runs are left out, and runs that stopped at the old cap are reported separately as runs whose length is unknown, not as runs that needed exactly the cap.
- The report. Every number in it is produced by the script on the submitted traces. The estimator comparison gives both totals and explains the difference. The triangle statement names a corner, a limit and a meter with a threshold.
- No network calls and no paid API anywhere in the tests.
Reading quiz (before week 12)
A short quiz on Chapters 19 and 20 covering the four objectives: an input bill and its crossover step from F, g and n; what a described prompt layout does to the cache; three latency complaints and their fixes; placing features on the triangle; the async job pattern’s states and its idempotent submission; and a ceiling, a queue policy and a rollout plan for a described change.
Reading for week 12: Chapter 21, Coding Agents, Chapter 22, The Coding Workflow in Practice and Chapter 26, Agents as Classifiers and Scorers (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.