How to run the session
Week 6 covers four chapters and closes Part IV of the book, the half that decides what gets built. Chapter 11 (Multi-Agent Systems) takes the subtask list away from your code and gives it to a model: the orchestrator–worker pattern, the brief a worker lives or dies by, the ways coordination fails, and the merge wall. Chapter 12 (Oversight and Autonomy) asks which actions wait for a person and where that wait lives, then moves review from lines to intent and outcomes and sets the autonomy dial per task. Chapter 13 (Writing the Outer Loop) replaces you as the one who prompts the agent, with a loop that finds work, assigns it, checks it, records it and decides; most of the chapter is about the check and the brakes. Chapter 14 (Choosing Your Approach) reads the whole bill and decides whether to pay it. All four chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow the lecture.
The deck has 30 slides in four parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#27 is the exercise). The PDF is the same deck, one slide per page, without notes.
Five slides are built to be answered by the room before you advance: slide 2 (write the subtasks for a landscape survey), slide 6 (three workers, one vague brief: what comes back?), slide 12 (which of five tools wait for a signature, and what you consulted), slide 18 (a one-line loop left running overnight) and slide 24 (the cheapest thing for four requests). Slide 20 opens with a one-sentence exercise of its own: the stopping condition for “make the app better”. Give each a minute.
The exercise uses two browser tools. Check before class that both links on slide 27 open with their inputs loaded: the consequence tier classifier with seven tool names, today’s approvals and a written rule, every question blank; and the multi-agent failure explorer with a one-line brief in its “Check a brief” box. Both tools run in the browser, need no account and no API key, and carry their inputs in the page address.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the subtasks you could not write, objectives | 1–3 | 4 |
| Multi-agent: the pattern and its bill, the one-line brief, the four fields, four seams, the merge wall, the verdict | 4–10 | 15 |
| Oversight: the sorting check, consequence tiers, the gate in code and the handoff, review theater, the autonomy dial | 11–16 | 14 |
| The outer loop: the one-line loop, its three lessons, the goal function, the five verbs, debts and brakes | 17–22 | 13 |
| When not to: four requests, the ladder priced, signs both ways and agent-washing | 23–26 | 8 |
| In-class exercise: a gate policy, then a brief | 27 | 22 |
| Lab 5, the proposal, recap | 28–30 | 4 |
The plan fits the same 80-minute slot as weeks 1 to 5. Four chapters in 54 minutes of lecture is dense, so the notes hold the detail and each slide holds one idea. The exercise is two short tool runs in teams of three: about 10 minutes for the gate policy, 8 for the brief, and 4 for the debrief.
For a 75-minute slot, cut slide 19 (the three lessons of the loop) to its three cards, slide 15 (review theater) to its four artifacts, and the exercise debrief to 3 minutes. For a 90-minute slot, add a third part to the exercise: each team runs the classifier on the draft tool list of its own project (8 minutes), with a 2-minute debrief; the export can be attached to the project proposal, and it is not graded.
Common misconceptions and how to address them
“Multi-agent is what a mature system graduates into.” The chapter reads the public debate between the two camps as a disagreement about workloads: research is read-heavy and its pieces compose by juxtaposition; coding mutates one shared artifact and its pieces compose by merge. “It is an architecture with a habitat.” Start with one agent, and ask the one question: “does this work decompose into branches that can run without seeing each other?”
“Parallel agents make the work cheaper.” They make it faster and broader. One team measured its multi-agent system at roughly fifteen times the tokens of a chat interaction, a point-in-time figure for its own workload. “Parallelism buys wall-clock speed and breadth of coverage; it never buys efficiency.” Running the workers on a cheaper model tier lowers the bill without lowering the token count.
“The workers will figure out what I meant.” A worker is a stateless engine at an empty desk, and the memo is everything it knows. The chapter’s three workers given one four-word brief did competent work on the tasks they each invented. “The worker is exactly as good as the memo that summons it.” Write every brief as a ticket for a contractor who cannot ask questions.
“A good enough brief prevents coordination failures.” It prevents the first seam. The second survives good briefs: every action encodes silent decisions (a style, an interface, a way of handling the empty case), and parallel workers cannot see each other’s. That is why the field converged on keeping writes single-threaded: “one agent owns the artifact; everyone else advises.”
“A subagent’s report is our own work, so it can be trusted.” It arrives composed, in the first person and the house style, and that is the danger: if the worker read an injected instruction, its calm summary delivers it. A return is tool output, untrusted until checked. Week 9 takes up the attack.
“Gate the actions the agent is unsure about.” Stated confidence is miscalibrated in the worst direction; the chapter cites one production analysis in which a claimed ninety percent behaved like about seventy-five. The tiers key on what an action costs when it is wrong, “a fact about the world rather than a feeling about the model.”
“The safest agent asks before everything.” Approvals that become a rhythm stop being read, and the one request that deserved a hard look gets the same reflexive click as the forty before it. “Over-gating does more than annoy. It defeats the gate.” Read-only actions run without ceremony.
“Tell the agent in its prompt to ask before risky actions.” Anything that can persuade the model can persuade it to skip the asking. The gate belongs in the code that executes tool calls, after the model has chosen and before anything runs: “the gate must live in your code, never in the agent’s judgment.” Lab 5 asks for a test in which a model that never asks still meets a person.
“Tests are green, so the review is done.” Green checks certify internal consistency, not that this was the right thing to build or that it stayed in scope. Move the review to intent and outcomes: criteria stated before the run, a declared scope, a walkthrough, evidence. Then decide which changes you must understand, not just verify.
“Autonomy is a property of the agent.” It is a setting per task, moved on evidence, in both directions, and re-earned after every model upgrade, because the evidence expires with the model version it measured. Irreversible actions keep their signature at every position of the dial.
“Let the loop run until the agent says it’s done.” The model’s “done” is testimony. Unattended, a premature victory repeats all night, and a run that never declares victory bills you until something stops it. The stopping condition must be one a machine can evaluate, from outside the work, and a hard iteration cap sits under it in case done never comes true.
“A loop that runs without me saves me the review.” The reading moves; it does not disappear. A green gate is a fact about the checks you thought to write (verification debt), and work you did verify can still outrun your understanding of the system it lands in (cognitive debt). “Volume is not progress.”
“Calling it an agent makes it more capable.” The label decides nothing about the system; the control flow does. A workflow sold as an agent overcharges you in oversight ceremony; an agent shipped as an “automation” undercharges, because the budgets, gates and eval set were never bought.
“The cost of an agent is its token bill.” That is the run line of a four-line quote. Building includes the harness; being wrong costs deeds rather than words; and knowing whether it worked is the most expensive knowledge on the ladder, the line engineers leave off.
Materials and notes for instructors
All seven figures the syllabus names for this week that the site publishes are on the slides (Chapter 11: the orchestrator–worker pattern; the worker’s desk with its memo; the merge wall. Chapter 12: the consequence tiers; the autonomy dial. Chapter 13: the outer loop’s five verbs. Chapter 14: the ladder revisited) and are available individually on the diagrams page. The eighth, Chapter 13’s oracle gauge, is a book illustration with no web version; slide 20 describes it in words. The chapters have eight more figures, among them the signing ledger, review theater, the night shift, the compounding loops’ shared store, verification debt and agent-washing, which the notes describe and the reading shows.
The figures’ inner labels are small when projected; each opens full size from the diagrams page, which is worth a second browser tab for slides 9, 16, 21 and 25.
Numbers on the slides are the book’s and are labeled illustrative or sourced: about four times the tokens of a chat for one agent and about fifteen for a multi-agent system, and research time cut by up to 90 percent (one team’s report on its own system, point-in-time); over 1,600 traces, fourteen failure modes in three clusters (one research group’s trace study); a claimed confidence of ninety percent behaving like about seventy-five percent (one production analysis cited by a practitioner guide, illustrative); 847 duplicate records (one vendor’s illustration, quoted for its shape). The exercise’s 0.8 confidence threshold is part of an invented scenario, not a recommendation.
Products appear only the way the chapters present them, as labeled examples of a category. The deck names no orchestration framework, coding agent or model. The one-line loop on slide 18 is the chapter’s own generalization, with the product name stripped.
Key terms in the free glossary: orchestrator–worker, subagent, approval gate, review theater, autonomy dial, outer loop (with the inner loop), goal function, oracle, kill switch and agent-washing.
Each in-lecture check is resolved by the slides that follow it, and slide 20 resolves its own exercise in its notes. The instructor pack has the full keys to the checks (slides 2, 6, 12, 18, 20 and 24) with what to listen for, the expected results of the exercise, the Lab 5 key and the proposal rubric.
Exercises
In class: a gate policy, then a brief (teams of three, 22 minutes)
Part A, the gate policy (10 minutes). A campus IT helpdesk agent has seven tools:
search_kbsearches the help-center articles and returns matching passages. It changes nothing.read_ticketreturns a ticket’s text and history. It changes nothing.tag_ticketsets the category tag on a ticket. Only staff see tags; every change is kept in the ticket’s history and can be put back.move_attachmentmoves a file attached to a ticket into another of the helpdesk’s internal folders, which only staff see. It can be moved back, but nothing records where it came from.reply_to_requesteremails the person who opened the ticket. Once sent, it cannot be recalled.wipe_laptopremotely erases every file on a laptop.order_replacementorders a replacement laptop, charged to the department’s budget.
Today the agent asks a person before search_kb and wipe_laptop, and before nothing else. Its written rule says: “Ask a person before any action when the agent’s confidence is below 0.8.”
- Open the consequence tier classifier. The link loads the seven tool names, today’s approvals and the rule, with every other question blank. Answer each tool’s questions from its description above.
- Switch the dial position to “Every consequential action signed” and note which row’s gate policy changes. Switch back to “Standing permissions”.
- Copy the result as Markdown.
Part B, the brief (8 minutes). An orchestrator received the task “Find out why signups dropped last week.” It decided on three workers: one analyzes the signup funnel in the product analytics, one lists the product changes shipped in that week, one reviews the marketing campaigns and traffic sources.
- Open the multi-agent failure explorer. The link loads the task’s sentence as the brief, the way a careless orchestrator would forward it. Record the tool’s reading.
- Write the brief for the second worker, the one who lists product changes, with the four fields of Chapter 11. Replace the loaded text with your brief and read the result.
- Copy the result as Markdown.
The last 4 minutes are the debrief.
Submit: the two Markdown exports and a short note with: the row that changed at the tightest dial position; for each warning in the Part A export, one sentence on what you would change in the agent or its policy; the tool’s reading of the one-line brief (fields found and word count); and, if you overruled the tool on any field of your brief, the sentence of the brief that carries that field. The note can be finished after class, within 24 hours; the 22 minutes are for the two tool runs.
Acceptance criteria:
- The Part A export lists all seven tools at the “Standing permissions” position, none of them marked unanswered, and each tool’s tier follows from its description. The note names the row whose policy changes at the tightest position.
- For each warning in the Part A export, the note gives one concrete change (a gate removed or added, a log added, a rule rewritten), not a restatement of the warning.
- The note records the one-line brief’s reading as the tool reports it.
- The Part B export shows the team’s brief with all four fields present and no short-brief flag. The brief check is a word-pattern heuristic and misses some natural phrasings, so a field the tool missed may be overruled by hand, provided the note quotes the sentence of the brief that carries that field. Its objective names one deliverable and its time window; its output format gives a shape and a size limit; its tool and source guidance names at least one specific place to look; its boundaries name what belongs to the other two workers.
Lab 5: an approval gate in code and a minimal outer loop (individual, about 8 hours, in two sessions)
Extend any of your earlier agents (Lab 2, 3 or 4), still with a scripted model client. No paid API is needed; optionally, also run it against a local open-weight model. The agent works in a sandbox folder, on a small project of your choice (a set of documents, a toy repository, a folder of records).
- Tools in four tiers. Give the agent at least five tools that span all four consequence tiers of Chapter 12, including at least one externally visible tool (one whose effect a third party would see; in the sandbox, it writes to an outbox folder) and at least one irreversible or high-stakes tool (it deletes or moves money or touches the public). Write the tier of every tool in a table in code. A reversible tool must log enough to undo its change.
- The gate. In the code path between tool selection and tool invocation, after argument validation and before the tool runs: externally visible actions go to a review queue on disk and do not run until a person approves them, then run exactly once; irreversible actions block until a person decides. The decision comes from an approver function (scripted in your tests). Nothing the model writes (a reason, a confidence, a claim that someone approved) changes what the gate does.
- The handoff. The person receives a package, not a transcript: the action in plain language, why the agent wants it (labeled as the agent’s claim), what it touches and whether it can be undone, the before and after of anything it changes, and three choices: approve, reject, reject with edits. Edited arguments are validated and checked against the tool’s scope before anything runs.
- Scope. Every file tool resolves its path and refuses anything outside its own folder, including through
.., absolute paths and symbolic links. The loop’s own files (checklist, state, queue, inbox, kill switch) are outside every tool’s scope. - The outer loop. Write
find,assign,check,recordanddecideover a checklist of at least four items on disk, each item with a check a machine can run. Each pass hands one open item to a fresh run of the inner agent, with a brief that has the four fields of Chapter 11. The loop checks the item with its own check, never the agent’s last message; records every pass, failures included, to a state file; and decides: complete when every item’s check passes, rerun from disk; escalate to an inbox file when an item waits for review or fails a stated number of passes in a row; stop at a stated pass cap. - The kill switch. One fast way to stop everything (for example, a file whose presence means stop), checked by the loop before every pass and by the inner agent before every model call and every tool call.
- Keep the Lab 1 and 2 guarantees: structured errors,
MAX_STEPSwith a loudstopped, “done” decided by a check.
Acceptance criteria:
- Tiers. Every tool has a tier in code; a test shows that building the gate fails for a tool without one.
- A rejection is final. A test in which the model proposes an irreversible action the approver rejected again, in a later turn, three times in one reply, and in a later pass: the approver is asked once, and every repeat gets a structured error.
- The gate holds. A scripted run in which the model calls an irreversible tool without asking, with a reason that claims approval, still reaches the approver; a rejection leaves the sandbox unchanged and returns a structured error. A second test runs the same script with the gate written only as an instruction in the prompt and shows that it fails the first test.
- Parallel calls. A test with at least two gated calls in one model reply shows each reaching the approver as its own handoff, and a rejected one not running while an approved one does.
- The handoff. A test checks every gated tool’s handoff for all five parts. A reject-with-edits test runs the edited action, and another shows that edits pointing outside the tool’s scope are refused with nothing run.
- The review queue. A test that an externally visible action is queued and not executed; that the same action submitted twice (the loop coming back to the item) is queued once; and that an approved action runs exactly once, even if the approval is processed twice. A test shows that a queue without deduplication sends twice.
- Scope. A test of every file tool against
.., an absolute path and a symbolic link pointing at the checklist: each returns a structured error, the approver is never asked, and no file changes. - The stop is the oracle. A test in which the agent claims the item is done while its check fails: the loop does not report complete. A second test shows that a loop that stops on the agent’s claim reports complete in the same situation. Any “done” mark written into the checklist file is ignored, and the loop does not report complete while an action still waits in the review queue.
- Fresh windows and briefs. A test on each pass’s first request: it holds the instructions and a brief with the four fields, and nothing of an earlier pass (check with a marker string).
- Brakes. A test with an item that can never pass: with stuck detection, it is escalated after your stated number of failures; with stuck detection off, the loop stops at the pass cap with a loud status. A test that the state file records failed passes, and that the count and the blocked item survive a restart of the loop: run it again and the blocked item is neither reassigned nor escalated a second time.
- The kill switch. Two tests: tripped before the run, no model is called; tripped while an approval is pending, the approved action and any other call in the same reply do not run, the model is not called again, and the loop ends with a status that says it was killed. Processing approved queue entries is refused while the kill switch is on.
- The Lab 1 and 2 guarantees still hold. No network calls and no paid API anywhere in the test suite.
Project proposal (teams of two or three, due with Lab 5)
One page for the verification-centered project.
Submit: the problem in two or three sentences; its rung on the ladder (plain code, a single call, a workflow, an agent, or a workflow with one or two agentic steps) with the four-line quote; one sentence on why the rung below would not do; and the verification signal: what will tell you, by machine where possible, that a run worked.
Acceptance criteria:
- The problem names its users and one concrete input and output.
- The four-line quote has one line each for build, run, wrong and know, and each line names a concrete cost or artifact (for example, an eval set of a stated size, calls per task times the expected volume, what a wrong run breaks and who finds it, how a run is checked).
- The rung is justified with a sign from Chapter 14, and the sentence on the rung below names what that rung could not do.
- The verification signal is specific enough that two people would agree on whether a given run passed.
Reading quiz (before week 7)
A short quiz on Chapters 11 to 14 covering the four objectives: the decomposition question applied to several tasks, and the missing field in several briefs; tiers and a gate policy for a described agent; stopping conditions for several loop goals; and a request priced on the ladder, with a small calculation.
Reading for week 7: Chapter 15, Observability and Debugging (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.