How to run the session
Week 12 visits the two places where the machinery of the course already earns its keep. Chapter 21 (Coding Agents) asks why software was the field where agents first shipped work, and turns the answer into a template for any other field. Chapter 22 (The Coding Workflow in Practice) follows a working day with a coding agent, from the first interview to the receipts at the end. Chapter 26 (Agents as Classifiers and Scorers) builds the small instrument that sits inside most of what the course has built (the router, the judge, the guardrail): a model that reads something and picks a label from a short menu, written as a document and measured without flattering its author. All three chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow.
The deck has 35 slides in four parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#32 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (two tasks left running over a weekend), slide 7 (two engineers, one tool), slide 20 (a ticket that is both billing and a bug) and slide 27 (twenty out of twenty). Give each a minute; on slide 2, ask each student to write down the reason for their choice, because slide 5 sorts the reasons.
The exercise uses three browser tools: the three-set splitter, the judge agreement calculator and the cascade cost estimator. It runs on one invented data set, published with this lecture: tickets.csv, 200 support tickets for a fictional product, each with a text, one of four labels (billing, bug, feature-request, other) and an id. Students paste the file into the splitter; the calculator and estimator links in the exercise open with their inputs filled in. Check before class that the file downloads from the lecture page and that the first calculator link shows a kappa of 0.70. A second file, worksheets.csv, holds frozen rubric answers for the tickets the split puts in the guide and the development set (the sealed set’s answers stay with the instructor); the in-class exercise prints the few rows it needs, and only the optional lab uses the file. All three tools run in the browser, need no account and no API key, and send nothing anywhere. No model is called in class.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the weekend, objectives | 1–3 | 4 |
| Why coding went first: the domain grades its own work, the loop, two engineers, the spectrum, agent-legible code, the ground-truth loop | 4–10 | 10 |
| Briefs down, receipts up: brainstorm to execute, spec-driven development, tests first, TCR, savepoints and recorded proof, cognitive debt | 11–17 | 14 |
| A classifier you can read: the examiner, the two-headed ticket, a definition’s anatomy, examples, the worksheet, where a label went wrong, scores | 18–25 | 13 |
| Measurement you cannot fool: twenty for twenty, three sets, counting, the folder and the fresh desk, the cascade | 26–31 | 9 |
| In-class exercise: split it, score it, price it | 32 | 26 |
| Optional lab and reading, recap, close | 33–35 | 4 |
The plan fits the same 80-minute slot as weeks 1 to 11. The exercise is teams of two or three: about 7 minutes for Part A, 10 for Part B and 6 for Part C, with 3 for the debrief. Part B is the heart of the exercise and the part to protect if time runs short.
For a 75-minute slot, give slides 16 and 17 one sentence each (saving 3 minutes) and cut slide 25 to its title and the severity scale (saving 2); the debrief stays. For a 90-minute slot, add Part D (10 minutes): each team turns the severity request of slide 25 into a four-level ordinal scale for the support desk’s tickets, writes one boundary sentence between each pair of adjacent levels, and scores ten tickets from their development set independently, two people each; they enter the two columns in the judge agreement calculator in paste mode and read its level check for a scale. In the 80-minute plan Part D is not set.
Common misconceptions and how to address them
“Models are better at code than at other writing.” The model writes a market memo as fluently as a fix. What differs is what the field says back: a failing test returns a verdict the loop can consult as often as it likes, and a wrong memo returns nothing. Ask what the reason given on slide 2 would predict for a field with a strong checker and a model that writes it badly.
“Green tests mean the code is correct.” Green means the assertions somebody wrote are satisfied. Architecture, conventions, security and the assumption that breaks next year sit outside the suite. Chapter 21’s image for it: “the river only pushes on what touches the wheel.”
“Vibe coding is what beginners do; augmented coding is what professionals do.” They are two activities, chosen per artifact by how long it must live. A senior engineer prototyping a throwaway demo is right not to read the diff; the failure is a prototype that quietly becomes load-bearing while still unread.
“Agent-legible code is a new style invented for machines.” Almost every item on slide 9 is what good engineers always asked for: local reasoning, greppable names, checks where the action is. What is new is that the cost of ignoring it can now be measured in tool calls and wrong-file edits.
“A spec is waterfall again.” Waterfall’s trouble was the months between a decision and its test. Here a cycle lasts minutes, and the spec is wired to tests so that drift turns red. The real risks are a spec nobody updates and a handoff treated as final.
“If the agent wrote the tests, TDD is covered.” Only if someone saw them fail first and nobody let the implementer edit them. A test that passed on its first run may be passing for reasons unrelated to the code; a test the implementer can rewrite is a checker the work can corrupt.
“An LLM classifier needs a training set, or at least many examples in the prompt.” Its training is the guide: definitions, boundary rules and a handful of examples chosen for a role. Fifty examples cost tokens on every call and bury the definitions; the second fifty teach almost nothing.
“The rationale tells you how the model decided.” It is an account written by the same machinery as the label. Written before the label it can inform it; written after it can only defend it. Use it to choose where to look; only counting says whether the classifier works.
“Ask for a score from 0 to 1 and you get a ranking.” You get numbers nobody defined, which bunch into a favorite band. A small ordinal scale with a sentence at each boundary, scored several times and averaged, gives a fine-grained number that means something.
“Our classifier scores 20 out of 20.” On which tickets? If the guide was written with them open, the score measures the guide on its own worked examples. Only a set nobody looked at while writing can say how the classifier does on tickets it has not met.
“A higher development score means the revision worked.” The development set is read after every revision, so the guide slowly fits it. A revision also moves tickets the author was not thinking about: “no edit is local”. Compare the whole matrix before and after, not one number.
“A high flag rate is safe.” Flagging every hard ticket makes the kept tickets look excellent and sends the work to a person. The flag rate is reported beside the score, always.
Materials and notes for instructors
Seven of the eight figures the syllabus names for this week are on the slides and available individually on the diagrams page: the augmented-to-vibe spectrum (slide 8), the ground-truth loop (slide 10), brainstorm, spec, plan and execute (slide 12), the TDD and TCR guardrail (slide 15), the worksheet before the verdict (slide 23), the three sets and the sealed envelope (slide 28) and the screen-then-detail cascade (slide 31). The eighth, the examiner and the marking guide, is an illustration in the book; slide 19 describes it in words.
The data set is invented for the exercise and belongs to no real product or company. Its labels are the ones the exercise treats as correct, and some tickets are hard on purpose. The worksheet answers in worksheets.csv stand in for a model’s answers to the three rubric questions of slide 23 (does money move wrongly, is the product behaving contrary to its design, what remedy is asked), plus a fourth, whether the ticket is about money at all; they were written once and frozen, so every team gets the same results and no model is needed. The guide’s boundary rules are written as an ordered decision table that turns those answers into a label: the stand-in for the reading a model would do. This makes the exercise reproducible, and it has one honest limit students should hear: a scripted client follows the table exactly, so the examples in a guide change nothing in its output, whereas with a model they would. Part B’s second question asks students to reason about that difference.
Use the splitter’s default settings (2 demonstrations per class, development share 50 percent, seed 26) so that every team’s development set is the one the calculator links in Part B were computed on.
Numbers on the slides are the book’s or the exercise’s, and are labeled. From Chapter 26: the billing definition (slide 21) and the four-level severity scale (slide 25) are condensed from the chapter’s sketches; the repeated draws of 3, 3, 3 and 2 averaging 2.75 are worked on the slide; the near-even split of a few hundred labels is the chapter’s, marked by the chapter as illustrative. From Chapter 21: the seventy-percent problem is reported in the notes as a proverb, which is how the chapter treats it.
Chapters 21 and 22 cite named practitioners, published experience reports, research papers and named coding tools as examples; Chapter 26 cites research on in-context learning and on judge scales. The deck names one practitioner, Kent Beck, as the chapter does on slide 8, credits the other practitioners it quotes as “a practitioner” with the chapter as the source, and describes the tools and practices as categories without naming any product, model, vendor or harness. Commands appear only as the general shell idiom test && commit || revert, which is the name of the practice.
For reading around the topic: Build an LLM Classifier Without a Training Set, LLM Classifier vs Fine Tuned Classifier: A Decision Table, Spec Driven Development With AI Agents, An Agentic Coding Workflow for Teams and A Coding Agent in a Legacy Codebase. Week 8 (judges, kappa, the lazy judge, the level check) is the background for Part 4 and for Part B of the exercise; week 4 (skills, in-context learning) for slides 22 and 30.
Key terms in the free glossary: vibe coding, augmented coding, agent-legible code, compound step, standing project instructions file, oracle, classifier, ordinal scale, three-set discipline and in-context learning. Spec-driven development, TCR, the fresh desk and the confusion pair are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 7, 20 and 27) with what to listen for, the expected results of the exercise, the key to the optional lab and its reference implementation.
Exercises
In class: split it, score it, price it (teams of two or three, 26 minutes)
One data set runs through all three parts. A fictional company’s support desk receives tickets in four classes: billing, bug, feature-request and other. Someone labeled 200 of them; they are in tickets.csv (columns text, label, id). The team is building a classifier for the desk in the shape of slides 21 to 24: a guide with class definitions and boundary rules, three rubric questions answered before the label, and a flag for tickets the rules cannot settle.
Write down what each tool shows, then answer the questions in a sentence or two each. Where a question asks for something the tool does not print, it says so, and gives you what you need to work it out.
Part A, the split (7 minutes). Open the three-set splitter, paste the whole of tickets.csv into Labeled rows (the header row included) and keep the default settings: 2 demonstrations per class, development share 50, seed 26.
- Write down the rows read and placed, how many rows each set gets, the per-class counts, what was left out and why, the lazy baseline, and the three fingerprints.
- One text appears under two labels. Read it. Which label would you keep? Slide 21’s boundary rule decides two-headed tickets by the remedy asked for: does it decide this one? Write the boundary sentence your guide would need, or say why the ticket should be flagged instead.
- Set the development share to 80. Write down the sealed test count for
other. The tool raises no warning for it (it warns below five): how many points ofother’s recall does one ticket move in the sealed set at 80, and how many at 50? Set the share back to 50. - A week from now, after reading the development score, a teammate proposes changing the seed “to get a less awkward test set”. What do the fingerprints let you check, and what would the reseeded test set no longer be able to tell you?
Part B, the revision (10 minutes; question 4 can be finished after class). The team’s first guide, v1, ends in this decision table. The rules are tried in order and the first that matches decides; the four fields are a ticket’s frozen worksheet answers (about_money, money_wrong and defect are yes or no; asks_for is refund, fix, new, help or none).
money_wrong=yes -> billing
about_money=yes -> billing
defect=yes -> bug
asks_for=new -> feature-request
otherwise -> other
Open the calculator with v1’s results on the development set. The rows are the labels in tickets.csv (the tool’s “human”); the columns are the classifier’s (the tool’s “judge”).
Write down kappa with its interval, the band, the lazy baseline and how far v1 beats it, the level check, and the top three confusion pairs.
Here are six of v1’s failures on the development set, with their worksheet answers:
id Ticket Label about_money money_wrong defect asks_for T126 Changing the seat count makes the payment form crash before I can pay. bug yes no yes fix T106 Your app double-charged me after freezing. My bank already reversed it, but please fix the freeze. bug yes yes yes fix T184 Please add the option to pay by direct debit. feature-request yes no no new T066 Export to slides is broken: there is no such option in the menu. feature-request no no yes fix T038 I can’t find where to export a document. other no no yes fix T081 Since the update the app takes almost a minute to open. bug no no no help For each, say which row of v1 fired and which label it gave, and whether it is an evidence failure or a definitions failure (slide 24). For the definitions failures, write your revision of the table: rows added, removed or moved. Which of the six can no row fix without breaking more correct tickets than it fixes, and why? Last, T081: its worksheet reads the slowdown as no defect. With a live model, could an example in
examples.mdmove that answer? Write the example and the comment you would put on it, say which role it plays, and say why the scripted client cannot show its effect.The team’s own revision, v2, added and reordered rows. Open the calculator with v2’s results. Compare the two matrices cell by cell. Which cells moved? From the cells alone, which of the six failures above did v2 fix, and does that match the revision you wrote? One cell got worse. Name it. Its two tickets are T085 (“The team plan features stopped working the day our invoice became overdue.”) and T101 (“Since my payment failed I can’t open the shared calendar any more.”); both have the worksheet answers yes, no, yes, fix. v1 labeled them correctly. Was v1 right for the right reason?
v2 flags one ticket of 95. What does the tool say about that rate, and what does it not tell you about how the flag will behave in production? Kappa went from 0.70 to 0.75: write the CHANGELOG row for v2 (version, what changed, the failure that motivated it, the development score before and after). Is this the moment to open the sealed set?
Part C, the volume (6 minutes). The desk wants a cheap screen in front of the full classifier: the screen labels every ticket and sends onward the ones it is not sure of. The illustrative costs are in cost units per ticket (300 for the screen, 2,000 for the full classifier, no separate gate); the screen escalates 20 percent; the full classifier flags 5 percent of what reaches it; the screen’s error on what it clears is guessed at 4 percent; a person spends 3 minutes per flagged ticket. Open the estimator at 50 tickets a day and at 50,000 tickets a day.
- At each volume, write down the cascade’s cost against the full classifier alone, the saving, the break-even escalation rate, the tickets cleared and wrong per day, and the person-hours. The percentage is the same at both volumes. What changes, and at which volume would you build the screen at all?
- Part B’s v2 flagged 1 percent of the development set. Why is that the wrong number to enter as the estimator’s flag rate, and how would you measure the right one?
- The 4 percent error is a guess. Say how you would measure it, on which set, and what you would have to show, on which set, before claiming the cascade is about as accurate as the full classifier alone.
- At 50,000 a day the screen turns out to escalate 50 percent. Write down the new cost share, flagged tickets and person-hours. Which number would worry you first, and who has to agree before the screen ships?
The last 3 minutes are the debrief.
Submit: the splitter’s MANIFEST.md and CHANGELOG.md (with your v2 row filled in), the Markdown exports of the calculator for v1 and v2 and of the estimator at 50,000 a day, and a short note answering every numbered question above. The note can be finished after class, within 24 hours; the 26 minutes are for reading the tools.
Acceptance criteria:
- Part A. The counts, the left-out rows and the fingerprints are as the tool shows them at the default settings. The two-label ticket gets a decision with a reason, and the note says whether the remedy rule settles it. The sealed count for
otherat 80 percent is the tool’s, and the swing per ticket is computed at both shares. The reseeding answer names what the fingerprints detect and why choosing a test set after seeing results spoils it. - Part B. The v1 figures are as the tool shows them. Each of the six failures names the row that fired and is classified as evidence or definitions with a reason taken from the worksheet. The revision is written as table rows in order, and the note names the failures no row can fix. The comparison of v1 and v2 is made cell by cell and names the fixed failures, the worsened cell and its cause. The flag answer says what a development-set rate cannot predict. The changelog row has all five columns with both scores, and the envelope answer gives a reason.
- Part C. The figures at both volumes are as the tool shows them, with what changes between them. The flag-rate answer says which tickets the full classifier sees behind a screen. The measurement answer names a set for the error estimate and a set and a comparison for the accuracy claim. The 50 percent figures are the tool’s, with an owner for the decision.
- The tools’ verdict lines apply rules that are the tools’ own (each tool says which of its rules are the book’s). Where the team’s reading differs from a verdict line, the note says why in one sentence; a justified overrule earns full credit.
Optional Week 12 lab: the triage classifier (individual or team, about 8 hours)
The optional Week 12 lab replaces your lowest lab grade. It builds the support desk’s classifier from the exercise as a skill folder, develops it on the development set, and grades it on the sealed set exactly once.
The classifier runs against a scripted client: worksheets.csv holds a frozen worksheet (the four rubric answers) for every ticket in the guide and the development set of the default split, and the instructor holds the sealed set’s, and your guide states its boundary rules as an ordered decision table in a fenced block marked decision-table. Each row reads field=value and field=value -> label, with the fields about_money, money_wrong, defect and asks_for; a row may end in -> abstain to flag; the last row is otherwise -> label; the first row that matches decides. A ticket no row matches is flagged as one the guide cannot classify. Teams with a local model may run it on the same guide instead and say so; the scripted client is the reference. No paid API is needed.
Start from v1 (the table in Part B) and the split from Part A at the default settings. So that the instructor’s checks can read your folder, keep to these formats: in examples.md, put each example’s input on its own line as Ticket: "…" (in a list or a blockquote is fine); in CHANGELOG.md, start each revision row with its version name, v and a number (v1, v2, v3.1), keep each version of the guide as versions/GUIDE-<version>.md, write each score with the word kappa before the number (the interval in brackets, accuracy, on <n> and flagged <k> may follow, in any order), and log openings under a heading ## Envelope log. Work in the shape of Chapter 22: a short SPEC.md for the scorer and the checks before any code, a PLAN.md cut into steps, and tests written and seen failing before each piece. Deliver:
- The skill folder
ticket-triage/:GUIDE.md(each class with an essence, inclusions, exclusions and boundary rules; the rubric questions; the decision table; a version number in its front matter),examples.md(a few examples, each labeled with its role: prototype, edge or contrastive pair, with a comment saying why it is there),schema.json(the output record: the four rubric answers, the label from the closed set or none, a flag, a short rationale, with the label written after the answers) andCHANGELOG.md. - A runner and a scorer, standard library only. The runner classifies each ticket in its own call, with the guide and that ticket and nothing else, and validates every record against the schema. The scorer reports kappa with its interval, accuracy, the lazy baseline, the confusion pairs and the flag rate, and prints a calculator link for the matrix.
- At least two revisions after v1, each motivated by failures you read on the development set and logged in
CHANGELOG.mdwith the development score before and after. Keep every version of the guide (for exampleversions/GUIDE-v2.md), so that every number in the changelog can be recomputed. - One opening of the sealed set, at the milestone you choose. You cannot score the sealed set yourself: ask the instructor, who runs your guide on it once and returns the score. Log it in the changelog’s envelope section with the date, the milestone, the guide version and the score.
- A short report (one page): the final development and sealed scores with their intervals and flag rates, the remaining confusion pairs and which of them are evidence failures no rule can fix, and one paragraph on what you would change if the worksheets came from a live model instead of a frozen file.
Acceptance criteria:
- The folder. It holds the four files. Every class in the guide has an essence, inclusions, exclusions and at least one boundary rule. The schema’s label field is a closed set of the four labels plus none, and the flag and the label cannot contradict each other.
- No leaks. No example in
examples.mdis a development or test ticket (a test checks the example texts against both sets, ignoring case and extra spaces), and the three sets are disjoint. - The fresh desk. A test shows that each ticket is classified in one call, and that the result for a ticket does not depend on the tickets before it (for example by running the set in two orders).
- Failures handled. A row with an unknown field or value, a table without an
otherwiserow, a ticket with no worksheet and a record outside the schema each produce a clear error or a flag, not a silent label. - The numbers. Every development score in the changelog is reproduced by the scorer from the archived guide version. Kappa is checked in a test against a small matrix worked by hand. The sealed set was opened once, and the logged score is the one the instructor returned.
- The report. It gives the sealed score once, with its interval and flag rate, and keeps the development and sealed numbers apart.
- No network calls and no paid API anywhere in the tests.
Homework: the capstone
There is no required lab this week. The time goes to the capstone, whose demonstrations are next week: the agent, its eval set and dashboard, a trace review, the security audit and the rollout plan. One question from today belongs in the demo: which slice of correctness do your checks cover, and what lies outside it?
Reading quiz (before week 13)
A short quiz on Chapters 21, 22 and 26 covering the four objectives: filling the four slots of the ground-truth loop for a field outside software; placing two artifacts on the vibe-to-augmented spectrum; what red-before-green and test-commit-or-revert each protect against; writing an exclusion and a boundary rule for a two-headed ticket; telling an evidence failure from a definitions failure from a worksheet, and what an example could change with a live model; and what a score on a set the guide was written from measures.
Reading for week 13: Chapter 23, Research and Business Agents, Chapter 24, Agent UX and Human Trust, Chapter 25, Transversal Recipes and Chapter 27, The Frontier and How to Keep Learning (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.