How to run the session
Week 13 closes the course and hands it over to the teams. The lecture block leaves the fields that grade their own work. Chapter 23 (Research and Business Agents) asks what replaces the compiler for an agent that writes briefs and for one that issues refunds, and what each is worth once the checking is paid for. Chapter 24 (Agent UX and Human Trust) designs the room in which a person supervises an agent. Chapter 25 (Transversal Recipes) catalogs the shapes of work that recur in every industry and places them on the autonomy slider. Chapter 27 (The Frontier and How to Keep Learning) takes verification to its strongest form and gives a map of the ecosystem that outlasts its products. All four chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow. Appendix C is the further reading.
The rest of the week is the capstone demonstrations. The deck carries both: the lecture block (slides 1 to 30), a workshop on each team’s own project (slide 31), a demo-day briefing (slides 32 to 34) and the close of the course (slides 35 to 38).
The deck has 38 slides in five parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#31 is the workshop, deck.html#32 opens the demo-day briefing). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (an hour to check a research brief), slide 10 (which hire’s worst error can be priced in advance), slide 13 (two colleagues, one agent) and slide 25 (a lock and a passing test suite). Give each a minute; on slide 2, ask each student to write the plan down, because slide 7 sorts the plans.
The workshop uses three browser tools: the ROI versus risk map, the agent verifiability scorecard and the agent recipe picker. Parts A and B open the first two with their inputs filled in; Part C runs all three on each team’s own project. Check before class that the first link in Part A shows a net value of 80 units per period and the Part B link a score of 23 out of 24. All three tools run in the browser, need no account and no API key, and send nothing anywhere. No model is called in class.
Two ways to schedule the week
The syllabus puts the last lecture and the capstone demonstrations in the same week. How the two share the time depends on the number of teams. A demonstration slot is twelve minutes (six to show, five for questions including one live change, one to change over) or, for larger classes, ten (five to show, four for questions, one to change over). An 80-minute demo session opens with slides 32 to 34 (5 minutes) and then holds six teams at twelve minutes or seven at ten, per panel.
Two sessions (recommended). Run the 80-minute plan below as the lecture session and hold the demonstrations in their own session or sessions, closing each with slide 38. For more than seven teams, run two panels in parallel (the instructor and a teaching assistant, or two instructors, in two rooms, with the same protocol and the same recording sheet) or add a demo session; the instructor pack’s week 13 page has a capacity table by number of teams. If a demo session comes first in the week, run slides 32 to 34 at its start anyway and move the workshop’s Part C into the homework before it, because the demo ends on Part C’s scorecard.
One session. With five teams or fewer, everything fits in one 80-minute slot: slides 1 and 3 (2 minutes); a short lecture path through slides 6, 7, 11, 14, 15, 20, 26 and 28, one or two per objective (16 minutes); slides 32 to 34 (3 minutes); five demonstrations at eleven minutes each (55 minutes, with questions held to four); slide 35 and slide 38 (4 minutes). The workshop’s Parts A and B are dropped; Part C becomes homework due before the session.
Whichever plan you run, the reading quiz goes online within the week rather than into a demo session, and students should know the plan, their slot and their panel a week ahead.
Timing (80 minutes, lecture session)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the brief, objectives | 1–3 | 4 |
| Words without a compiler: two hires, the verification gap, four substitutes, grounding, the clerk, the budget meeting | 4–11 | 12 |
| The control room: two colleagues, appropriate reliance, the board, steering and alarms, directing | 12–17 | 9 |
| The pattern book: point-task or recipe, the slider, eight recipes, three placements | 18–22 | 7 |
| The map and the weather: the map, the lock, generator and gate, property-based tests, layers, the portable core, directions and habits | 23–30 | 11 |
| Workshop: price it, score it, place it | 31 | 24 |
| Demo-day briefing | 32–34 | 5 |
| Looking back, what is left, recap, close | 35–38 | 8 |
The plan fits the same 80-minute slot as weeks 1 to 12, with a shorter lecture block than usual so that the workshop and the briefing fit. The workshop is teams of two or three, in the teams of the capstone: about 8 minutes for Part A, 8 for Part B and 6 for Part C, with 2 for the debrief. Part B is the one to protect, because it is the panel’s view of a scorecard and every team will be on the other side of it at the demo.
For a 75-minute slot, give slide 22 its title and the composition line (saving 2 minutes) and slide 36 its left column only (saving 1); start Part C in the room and finish it as homework (saving 2). For a 90-minute slot, add Part D (10 minutes): each team writes a premortem of its own demonstration (“it is demo day and our demo went badly: what happened?”) and its mirror (“we skipped the rehearsal and it was fine: why?”), then trades papers with another team, which names the one failure it thinks most likely. In the 80-minute plan Part D is the first item of the homework.
Demo day
The capstone brief in the syllabus sets three rules: no cherry-picked runs; the demonstration opens on the eval dashboard, shows a trace and ends with the team’s agent verifiability scorecard; grading follows passᵏ (a task counts as reliably solved only when it passes on every one of its runs). The course’s capstone rubric treats them as grading conditions. Slides 33 and 34 put them to the teams in this form:
- The eval dashboard first (about 2 minutes): per-task pass rates over at least three runs, a pooled rate with its interval, pass@k beside passᵏ, and the capability suite reported apart from the regression suite.
- A trace the panel picks (about 3 minutes): the team brings its run list sorted the way its trace review sorted it (by cost or by step count) and the panel names the run to open, the worst or one drawn at random. The team walks it to the first span where the run went wrong, or says why none did.
- The scorecard last (about 1 minute): every “yes” points at an artifact the panel can open.
- Questions (about 5 minutes): the panel asks each member about one part of the project, and one member, named by the panel, makes one live change prepared to run in under two minutes, such as adding a negative case to one task and running it once. Every other member explains and changes one thing in a short follow-up conversation within the week, as the capstone rubric allows.
A team may run its agent live. A live run is read against the rate the dashboard reports and is never graded on its own: a lucky run earns nothing, and an unlucky one costs nothing that the numbers did not already show. What a live run cannot do is stand in for the numbers.
Bring a second laptop with every team’s run list open, in case a team’s machine fails mid-demonstration. The instructor pack’s week 13 page has the demo-day grading notes; the rubric itself is in the pack.
Common misconceptions and how to address them
“A research report with forty citations has been checked.” It has been referenced. Chapter 23 reports an audit in which many sentences were not supported by the very sources listed beside them, and the longer reports were no better on that count. A citation is useful only when someone opens it, and the reader’s time is the cost.
“Grounding solves hallucination.” It moves the question from the claim to the source. An agent kept strictly to its sources is exactly as right as they are, and the open web serves pages built to rank before pages that are right.
“The research agent is the safe one to start with; the business agent is the dangerous one.” Ask which one’s worst error can be bounded before the pilot. Slides 10 and 11 make the room find the answer.
“If the pilot looks good, the return is there.” The cost of checking the output comes off the value before anything else, and for research output it can eat most of it. A pilot that never measured the checking has measured half the business case.
“Good agent UX makes people trust the agent.” The target is reliance that matches reliability. An interface that raises trust across the board manufactures the colleague who approves everything; one that only shows warnings manufactures the one who checks everything twice.
“Showing the model’s reasoning is transparency.” Reasoning text is an account of the work, not the work, and a wall of it trains readers to skim. What a person signing needs is short: what the agent read, what it did, what changed, why it changed course and what proves the result.
“A stop button is enough control over a long run.” Stopping throws away the correct work along with the wrong turn, so people stop intervening. Steering needs state that survives a correction, which is an architecture decision, not a screen.
“Full autonomy is for when the model is good enough.” In the catalog’s one fully autonomous recipe, the autonomy is earned by the sandbox: nothing it can touch matters if it is lost. A better model changes how much the arithmetic lets you delegate, not the arithmetic.
“A formally verified design is correct.” It satisfies the properties someone wrote down. Whether those were the right properties is the part no checker sees, and it is where model-written specifications are weakest.
“The ecosystem moves too fast to learn; learn the tools as they ship.” The products change weekly; the six layers they fall into and the questions those layers answer do not. Learn the layers, file each product into one, and spend the time saved building small things against the ones that matter to you.
“The demo is where we show the agent working.” It is where the team shows how far the agent works, and how it knows. A team that shows its numbers and an honest trace of a bad run has given a better demonstration than a team with one flawless live run.
Materials and notes for instructors
All seven figures the syllabus names for this week are on the slides and available individually on the diagrams page: there is no compiler for facts (slide 6), the four partial substitutes (slide 7), return against risk (slide 11), the calibration line (slide 14), the eight recipes on the slider (slide 21), generator and gate (slide 26) and the durable layers (slide 28). One more figure, from Chapter 27, shows the portable core (slide 29). Chapter 23’s drawing of the clerk as a privileged employee is an illustration in the book with no diagram file; slide 9 states the rule in words.
Numbers on the slides are the book’s or the exercise’s, and are labeled. Slide 2’s twelve pages and forty-one citations are Chapter 23’s, which marks them as invented. Slide 16’s fifteen-minute run is Chapter 24’s scenario. The workshop’s two hires and the other team’s scorecard are invented for the exercise, and their amounts are in hours of staff time, not money; nothing on the slides or in the exercise is a price.
Chapters 23, 24, 25 and 27 cite named practitioners, vendors’ engineering accounts, surveys and research papers, and Chapter 27 names specification languages and proof assistants as labeled examples of their categories. The deck names no practitioner, product, vendor, model or specification language. It quotes sentences the chapters quote from others with the chapter as the source and a description of the original (“quoting a published essay on supervision surfaces”, “quoting a widely watched talk”, “adopting a survey’s phrase”), and it describes formal methods, property-based testing and the ecosystem’s layers as categories. The research audit on slide 6 is reported for its durable finding, as the chapter does, without its figures.
The glossary’s entry for grounding is the Chapter 3 sense (reasoning kept tied to facts checked against the world); slide 8 uses Chapter 23’s narrower sense (answering from sources placed in front of the model). Say so if a student brings the glossary entry to the discussion.
For reading around the topic: AI Agent ROI: Frame the Return Against the Risk, The Autonomy Slider: Four Positions Between You and the Agent, Why an AI Agent Works in Demo and Fails in Production, How Do You Know If an AI Agent Is Working? and Build vs Buy AI Agents: Let the Durable Layers Decide. Week 8 (pass@k, passᵏ, the reliability envelope, calibrated judges) is the background for slides 7, 20 and 33; week 6 (the autonomy dial, consequence tiers) for slides 11, 16 and 20; week 9 (the lethal trifecta, approval fatigue) for slides 9 and 16; week 7 (traces, the first wrong span) for slide 33.
Key terms in the free glossary: verification gap, oracle, lethal trifecta, appropriate reliance, supervision surface, review theater, autonomy dial (the autonomy slider is the same instrument), human in the loop / on the loop, pass@k and passᵏ and reliability envelope. The privileged-employee rule, the transversal recipe, property-based testing, the durable layers and the portable core are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 10, 13 and 25) with what to listen for, the expected results of the workshop, and the demo-day grading notes.
Exercises
In class: price it, score it, place it (teams of two or three, 24 minutes)
Work in your capstone team. Parts A and B use the same fictional company, a regional freight firm, and the links open the tools with the inputs filled in. Part C is your own project.
Write down what each tool shows, then answer the questions in a sentence or two each. Where a question asks for something the tool does not print, it says so, and gives you what you need to work it out. In the room, do question 1, the figures of question 3, and Part B’s questions 1 and 3: the debrief needs them. The rest (question 2’s reason, the end of question 3, and Part B’s questions 2 and 4) can be finished before the hand-in.
Part A, two hires (8 minutes). Every amount is in hours of staff time per month. The firm’s value line (the smallest value after checking that would justify a project) is 100 hours a month; its error line (the largest single-error cost the team absorbs as a correction rather than an incident) is 8 hours.
The analyst. The sales team wants a research agent to write competitor briefs. Each brief saves 6 hours of an analyst’s work, and the team wants 40 a month. Today the plan is that a rep skims each brief, about 1.5 hours. The team estimates that 5 percent of briefs carry a wrong fact that reaches a customer proposal, and that each costs about 50 hours to repair. The briefs go to the reps unreviewed. Open the analyst on the map.
- Write down the headline, the net value, the quadrant and the break-even error rate. The tool says a guaranteed review changes its verdict: set the review answer to yes. Which numbers on the page move, which do not, and why is that a problem for the business case? The review slide 8 asks for here (a person opening the load-bearing citations before a brief reaches a rep) takes about 3 hours a brief. Enter it, and write down the net value, the quadrant and both break-evens.
- With the review in place, the 50 hours per error is still the team’s guess. The tool does not print the error cost at which the net value reaches zero: find it, by trying values or from the lines of the table. Then say why no permission, consequence tier or approval gate can turn this guess into a bound.
The clerk. Operations wants an agent that matches carrier invoices and issues the routine refund when a customer was charged twice. Each case saves a quarter of an hour, and there are 3,000 a month. An audit of a sample costs 0.02 hours a case. About 1 percent of cases end in a wrong refund that lands, each costing 2 hours to reverse. Open the clerk on the map.
- Write down the headline, the net value, the quadrant and the two warnings about deeds. The team now records last quarter’s baseline, and adds an approval gate: a refund above a set amount waits for a signature, so that no landed error can cost more than 8 hours to put right. Set both answers to yes and enter 8 as the cost of one landed error. Write down the new net value and quadrant. What does the net value mean now that the cost you entered is a bound rather than a typical error? Name one cost of the gate that the page does not count, and say which line of the table it belongs in. Which hire would you pilot first, and what would have to be true of the other before it pilots? Before the clerk pilots, write down its role, its access (what it may touch, and for how long) and the questions its audit trail must let you answer afterwards, one line each.
Part B, a scorecard that flatters (8 minutes). Another team built a ticket router for the same firm: it reads incoming support tickets, picks a queue, sets a priority and can issue small goodwill credits. It scored itself on the scorecard: open the team’s scorecard. This is what its submission actually contains, one line per question of the scorecard, in the tool’s order:
| # | Evidence in the submission |
|---|---|
| 1 | A run counts as done when the router’s reply validates against the label schema. Whether the ticket went to the right queue is checked only on the eval set’s reference labels. |
| 2 | The priority is computed in code from the customer’s tier and the ticket’s age; the model only picks the queue. |
| 3 | A cap of 12 steps; no token, time or money budget. When the cap fires the run raises an exception and the transcript is lost. |
| 4 | Every tool error comes back to the model as the string failed. |
| 5 | Each step appends the full ticket thread and the customer’s whole history to the context. Nobody has looked at what was in context on a failed run. |
| 6 | The system prompt says “never issue a credit above 50 without asking”. The credit tool itself has no limit. |
| 7 | The router runs on the incoming stream around the clock; nobody presses go. Each ticket’s run ends when its reply validates or the step cap fires. There is no kill switch. |
| 8 | The design note says: “tickets are complex, so we need an agent.” |
| 9 | Every model call is a span with tokens and latency; tool arguments and results are not recorded. The ten worst runs were reviewed once, two weeks ago. |
| 10 | Thirty tasks grown from observed failures, each run once; a judge grades the free-text replies and has never been compared with human labels; a CI gate blocks merges on the pass rate. |
| 11 | The lethal-trifecta audit is written up and one leg is gated; the credit tool takes an idempotency key and writes a receipt. The three blast-radius dials are not written down. |
| 12 | Cost per run is measured on the eval set; latency and step count are not. The rollout plan has a canary, rolled back by hand. Nobody has priced the time spent checking the free-text replies. |
- Write down the team’s score and band. Re-mark each question from the evidence (yes, partly or no) and write down the new score, the band, the score by Part and the first three gaps the tool lists.
- Name the two re-marks you found hardest to decide, and say what evidence would settle each.
- The team says question 7 does not apply: “nothing runs unattended, it is just a router.” Do you agree? Answer from the evidence, and say whether a person is in this router’s loop or on it.
- How far is your re-marked score from the nearest band boundary? Name a single re-mark that would carry it across, and say what that would change on the page and what it would not change about the router. Pick the one fix you would make before demo day, and name the artifact that would show the panel it is done.
Part C, your capstone (6 minutes in class; finish before demo day).
- Enter your capstone in the ROI versus risk map in one unit you can defend (minutes or hours of staff time, say), with your own two lines. Write down the net value, the quadrant, both break-evens and every warning, and one sentence saying which of your inputs is measured and which is a guess.
- In the agent recipe picker, answer the two screening questions and tick every statement that describes your project. Answer the checklist on every card that opens. If the swap-the-industry test fails for your project, say which part of it would survive the swap.
- Fill in the agent verifiability scorecard for your own agent. Next to every “yes”, write the file, test or eval result that shows it. This is the last screen of your demonstration.
The last 2 minutes are the debrief.
Submit: the Markdown exports of the map for the analyst (after question 1) and the clerk (after question 3), of the re-marked scorecard from Part B, and of Part C’s three tools, with a short note answering every numbered question above. Parts A and B are due within 24 hours; Part C is due before the demonstrations.
Acceptance criteria:
- Part A. The figures are as the tool shows them at the inputs the questions give. Question 1 says which lines stay still when the review answer changes and why the checking cost has to move with it. Question 2 gives the error cost at which the net value reaches zero, with how it was found, and a reason drawn from the difference between words and deeds. Question 3 says what a bound does to the meaning of the net value, names a cost of the gate with the line it belongs in, and chooses a hire with a condition for the other. The clerk’s role, access and audit trail are specific to this agent: a role a job advert could use, or access with no limit in time, does not count.
- Part B. Each of the twelve re-marks is backed by its line of evidence, and the score, band, Part scores and gaps are the tool’s for the marks the team chose. Question 3 is answered from the evidence, not from the team’s description, and places the person in or on the loop. Question 4 names a re-mark that crosses a boundary from the team’s own score, separates the band from the router’s behavior, and names a fix and an artifact.
- Part C. The map’s inputs are stated with their unit, and the note says which are measured. The recipe picker’s answers are consistent with the project as the design note describes it. Every “yes” on the scorecard names an artifact.
- The tools’ verdict lines apply rules that are the tools’ own (each tool says which of its rules are the book’s). Where the team’s reading differs from a verdict line, the note says why in one sentence; a justified overrule earns full credit.
Homework: ready for the demonstration
There is no lab this week. Before the demonstrations, finish Part C, sort your run list the way your trace review sorted it, and check that every trace on it opens. Write a premortem of your own demonstration (“it went badly: what happened?”) and its mirror, and fix what the first one names. The capstone deliverables are due as the syllabus sets them.
Reading quiz (online, within the week)
A short quiz on Chapters 23, 24, 25 and 27 covering the five objectives: what each partial substitute for a compiler of facts leaves unchecked; the net value of a use case once checking and errors are subtracted, and why the error cost of words cannot be capped; telling over-trust from under-trust in a described interface and what would move each toward the diagonal; placing a project on the autonomy slider, saying where a person is in the loop or on it, and naming the safeguard its recipe cannot do without; and what a passing specification does and does not prove.
After the course: Appendix C, Annotated Further Reading and Source Map (full book) maps every chapter to the sources behind it. Start a private benchmark: five tasks from your own work that current models fail, each written down with what “solved” would mean.
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.