How to run the session
Week 3 has two halves that meet in one idea. Chapter 4 (Planning, Reasoning, and Self-Correction) asks how a goal becomes steps, how a model thinks harder about one step, and whether it can catch its own mistakes; its answer, “a real verifier beats the model grading itself,” points the course out of the model and into the environment. Chapter 5 (Tools and the Action Space) is the craft of that environment: the tools that are the agent’s entire reach, and the surfaces (descriptions, schemas, results, errors) through which the model holds them. Both chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow the lecture.
The deck has 29 slides in two halves, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#24 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (what would you do first with the migration goal?), slide 8 (which row does a given task belong in?), slide 11 (does “review your answer” improve it on average? take a show of hands and make everyone commit) and slide 18 (would wrapping the three calendar endpoints work?). Slide 9 opens with the chapter’s 38 × 27 mental-arithmetic exercise; give it the full fifteen seconds, because the felt strain is the explanation of chain-of-thought.
Have the in-class handout ready (the seven flawed definitions below, printed or posted), and check before class that the linter link on slide 24 opens with the set loaded; the link carries the JSON in the page address, so it works offline once the page is cached.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the migration goal, objectives | 1–3 | 5 |
| Planning: decomposition, interleaved vs plan-then-execute, replanning, choosing | 4–8 | 13 |
| Reasoning: chain-of-thought, test-time compute, self-consistency | 9–10 | 7 |
| Self-correction: the prediction, two columns, find vs fix, the verifier rule and ladder | 11–15 | 12 |
| Tools: the action space, the calendar worked example, too many tools | 16–19 | 9 |
| The interface: anatomy, errors, retrofitting for agents (two slides) | 20–23 | 8 |
| In-class exercise: lint, fix, re-lint, then debrief | 24 | 20 |
| Code execution and computer use | 25–26 | 4 |
| Recap, Lab 2, reading | 27–29 | 2 |
The plan fits the same 80-minute slot as weeks 1 and 2. The exercise is the tightest segment: 20 minutes leaves about 13 for lint-fix-re-lint and 7 for the debrief, so tell teams to fix the three “Fix first” findings and the delete tool before anything else. It sits before code execution and computer use on purpose: it applies slides 18–23 while they are fresh, and slides 25–26 can be compressed if the debrief runs long. If your slot is 85 minutes or more, give the exercise the extra five minutes first; that is the variant the exercise was first timed at. For a 75-minute slot, cut the exercise to 17 minutes and cover slides 25–26 in one pass of two minutes, pointing students to the chapter. For a 90-minute slot, give the exercise 30 minutes and have one team present its whole redesigned set.
Common misconceptions and how to address them
“Plan-then-execute means the agent follows a fixed script.” That is the blind executor Chapter 3 rejected. The chapter’s plan is a working document inside a loop that still observes, and the replan path is what makes the design serious: “A plan with no replanning step is a guess with a schedule.” Slide 6’s figure draws the replan path as its widest arc, from the executor back to the planner; point at it, then let slide 7 carry the quotation.
“More planning is always safer.” Planning is insurance bought in proportion to the cost of the accident, and decomposition has a bottom: every extra step is another call and another chance to lose the thread. If the steps are known, the right answer is not a better planner but a workflow (week 1’s litmus test).
“The reasoning trace explains why the model answered as it did.” A trace is prose from the same machinery as the answer. It is a map of places to dig, never a proof. A fluent chain to a wrong answer reads exactly as confident as one to a right answer.
“Asking the model to double-check its work catches errors.” This is the misconception slide 11 is built to surface, and many students will vote yes. The evidence splits: revision helps with style, tone and missing requirements; intrinsic self-correction of reasoning errors often makes things worse, and early positive results had smuggled in an oracle through the stopping rule. Do not overcorrect into “reflection is useless”: reflection downstream of a real signal (a failing test fed back whole) is the workhorse of practical reliability. The rule is about where the verdict comes from.
“Self-consistency is a verifier.” It is evidence gathered without an oracle, and it improves the attempt, but agreement is not correctness: paths can converge on the same wrong answer, and the technique needs answers comparable for equality. It sits with the reasoning techniques, not on the verifier rung.
“A good tool set mirrors the API.” The calendar example on slide 18 is the antidote; one-for-one wrapping works, which is why it is a trap. Ask “what task is the model trying to do?” and move the deterministic work inside the tool.
“If the model misuses a tool, we need a smarter model.” The chapter’s debugging order is the reverse: suspect the contract before you blame the intelligence. Its two vendor anecdotes (a benchmark gain from description refinements alone; a search tool that appended the current year until its parameter description was fixed) are on the notes for slide 20.
“A clean lint means a good tool.” The linter checks the shape of the contract with heuristics; it cannot tell that two tools should be one, or that a description is untrue. Step 3 of the exercise (find two flaws the linter missed) exists to make this concrete.
Materials and notes for instructors
All seven figures named in the syllabus for this week are on the slides (Chapter 4: plan vs interleaved, self-consistency vote, verifier vs self-judgment; Chapter 5: task-shaped tools, tool interface anatomy, code sandbox, UI reliability ladder) and are available individually on the diagrams page. Numbers on the slides are the book’s and are labeled as dated or illustrative (the Self-Refine average, the 150,000-to-2,000-token example). No slide depends on a particular model, vendor or framework; where the chapter names a vendor’s experience, the slides present it as one reported example.
The structured error shape required in Lab 2 (code, message, retryable, hint) comes from Chapter 18, “Reliability, State, and the Harness”; slide 21 says so. Students meet it fully in week 10.
The instructor key for this exercise is in the instructor pack.
Exercises
In class: lint seven flawed tools, fix them, re-lint (teams of three, 20 minutes)
Each team opens the seven deliberately flawed tool definitions below in the browser tool-contract linter (the link loads the set; teams can also paste the JSON). The linter runs in the browser, needs no account and no API key, and quotes the chapter rule behind every finding.
[
{
"name": "read_logs",
"description": "Reads the service logs.",
"input_schema": {
"type": "object",
"properties": { "service": { "type": "string" } }
}
},
{
"name": "search",
"description": "Searches for things in the system and returns stuff.",
"input_schema": {
"type": "object",
"properties": { "q": { "type": "string", "description": "What to search for." } },
"required": ["q"]
}
},
{
"name": "find_records",
"description": "Finds records. Returns a list of matching records with their UUIDs.",
"input_schema": {
"type": "object",
"properties": { "query": { "type": "string" } },
"required": ["query"]
}
},
{
"name": "create_ticket",
"description": "Creates a support ticket for a user. Returns ok.",
"input_schema": {
"type": "object",
"properties": {
"user": { "type": "string", "description": "The user." },
"severity": { "type": "string", "description": "How bad it is." },
"due": { "type": "string" },
"title": { "type": "string" }
}
}
},
{
"name": "deleteUser",
"description": "Deletes a user account from the database. Use when an admin asks to remove someone.",
"input_schema": {
"type": "object",
"properties": { "id": { "type": "string" } },
"required": ["id"]
}
},
{
"name": "get_customer",
"description": "Gets a customer profile by email and then sends them a satisfaction survey. Returns the profile.",
"input_schema": {
"type": "object",
"properties": { "customer": { "type": "string" } },
"required": ["customer"]
}
},
{
"name": "run",
"description": "Runs a command.",
"input_schema": {
"type": "object",
"properties": { "options": { "type": "object" } }
}
}
]
Teams fix the “Fix first” findings first, then the “Should fix” ones, renaming, merging or splitting tools where the design calls for it. Then they look for flaws the linter did not report, and re-lint.
Submit: the fixed JSON, the linter’s “Copy result as Markdown” export of the final run, and a short note.
Acceptance criteria:
- The final lint reports zero “Fix first” findings.
- Every remaining “Should fix” finding is either fixed or answered in one sentence (for example, a set-level warning the team judges acceptable, with the reason).
- The redesign removes at least one overlap or one-for-one wrapper, with one sentence naming what the model no longer has to do in its context.
- Every mutating tool has either an idempotency mechanism or a dry run, and the destructive one states whether it can be undone.
- The note names two flaws the linter missed and, for each, the Chapter 5 rule it breaks.
Lab 2: a verifier, a task-shaped tool and structured errors (individual, about 4 hours)
Extend your Lab 1 agent (Appendix A’s minimal agent with a scripted model client and the three file tools). No paid API is needed: the model remains a scripted client that returns a fixed sequence of responses. Optionally, also run it against a local open-weight model.
- Add
run_tests, a real verifier: it runs the project’s test command in a subprocess with a timeout and returns pass/fail, the number of failures and the first failure’s message, never the full log. - Add a task-shaped
search_logs(pattern, limit)(or an equivalent search over files) that returns matching lines with a little surrounding context, capped bylimit, and says when results were truncated, in place of reading whole files. - Make every tool error structured:
code(a stable identifier),message(what happened and what a correct call looks like),retryable(boolean) andhint(what to try next). No tool raises out ofexecute. - Give the mutating tool (
edit_file)dry_runsemantics: withdry_run: trueit validates the arguments and returns the diff it would apply, changing nothing. - Write the tool descriptions and schemas for all five tools and lint them with the tool-contract linter.
Acceptance criteria:
- A scripted-client test in which the model’s first call to a tool has a malformed argument (a missing required field or a value outside an enum); the test asserts that the tool returns a structured error with
retryableandhint, that the error is appended to the message history as a tool result, that the next scripted call is corrected, and that the run reaches its final answer. - A test that
run_testsreports failure on a deliberately broken file and success after the scripted edit, and that the loop’s “done” depends onrun_testspassing, not on the model saying so (keep the checker separate from the maker). - A test that
edit_filewithdry_run: trueleaves the file unchanged and returns the diff. - A test that
search_logsrespectslimitand reports truncation. - The Lab 1 guarantees still hold:
MAX_STEPSwith a loudstoppedoutcome, and every failure turned into a result. - The linter’s Markdown export for the five tools, with no “Fix first” findings, is committed alongside the code.
- No network calls and no paid API anywhere in the test suite.
Reading quiz (before week 4)
A short quiz on Chapters 4 and 5 covering the four objectives: choose a planning strategy for four described tasks; explain the find-versus-fix asymmetry and name a verifier for a given task; redesign a three-endpoint tool set into one task-shaped tool; rewrite a bad error message into a structured one.
Reading for week 4: Chapter 6, Skills, Protocols, and Interoperability and Chapter 7, Managing the Context Window (both in the full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.