Home / Tools

Tools

Each tool computes one thing the book teaches, with the chapter beside it. They run entirely in your browser: nothing you type leaves the page, and every result can be copied as Markdown.

Chapter 15

Agent bug bestiary

A field guide to AI agent bugs: start from the symptom, answer the trace questions, get ranked suspects, the book's fix, and an eval case for each failure.

Chapter 19

Agent cost-per-task estimator

Estimate what one agent run costs in tokens: the fixed prompt, the history it re-reads every step, retries and subagents. Free AI agent cost estimator.

Chapter 25

Agent recipe picker

Match your work to one of eight recurring agent recipes, see where it sits on the autonomy slider, how it fails, and which safeguards must be in place.

Chapter 1

Agent verifiability scorecard

Score an AI agent on twelve questions, one per Part of the book: how would you know it worked? Get a total out of 24, a radar by Part, and the chapters to open.

Chapter 26

Cascade cost estimator

Estimate what a cheap screen in front of an LLM classifier saves: cost per day, the break-even escalation rate, and the wrong labels it lets through unchecked.

Chapter 18

Circuit breaker and retry simulator

A retry storm simulator for agent tools: compare immediate retries, backoff and jitter, count nested retries, and see what a circuit breaker saves.

Chapter 2

Compounding error calculator

A free compounding error calculator for AI agents: whole-run success from per-step reliability, and the reliability a long task needs.

Chapter 12

Consequence tier classifier

Build a human in the loop approval policy for your AI agent: sort each action into four consequence tiers and get the gate each one needs. Try it free.

Chapter 7

Context window budget planner

A free context window budget planner: see how full your agent's window is, when it crosses the 40% rule of thumb, and which of four fixes to apply.

Chapter 16

Eval sample-size calculator

A free eval sample size calculator for AI agents: confidence intervals for a pass rate and the number of tasks you need, computed in your browser.

Chapter 18

Harness grid builder

Agent harness engineering as a worksheet: sort your controls into guides and sensors, see which cells are empty, and gauge how much harness you need.

Chapter 16

LLM Judge Agreement Calculator

A free LLM judge agreement calculator: Cohen's kappa with an interval, precision and recall, the level check and bias tests. Paste your labels to start.

Chapter 17

Lethal trifecta audit

A lethal trifecta checklist for AI agents: see which legs your design holds, whether a sharp tool adds risk, and the cheapest leg to cut. Print the audit.

Chapter 11

Multi-agent failure explorer

Pick the symptom your multi-agent run showed, read the seam behind it and the fix, check a worker brief for four fields, and decide whether to split.

Chapter 6

M×N protocol calculator

Count the M×N integration problem for your agents: adapters wired by hand versus one shared protocol, what an addition costs, and when a direct call wins.

Chapter 16

pass@k and pass^k calculator

Compute pass@k and pass^k for an AI agent from its per-attempt success rate or your own runs, and see the reliability envelope. Free pass@k calculator.

Chapter 19

Prompt caching savings

A prompt caching calculator for agent runs: see which input tokens hit the cache, what a misplaced timestamp costs, and what moving it to the tail saves.

Chapter 20

Queue and backpressure simulator

Model AI agent rate limiting: your ceiling in runs per minute, what a queue does in a rush, and why refusing at the door beats accepting and abandoning.

Chapter 23

ROI versus risk map

Work out the return on an AI agent use case after the cost of checking its output, place it on a value-versus-risk map, and see when to refuse.

Chapter 20

Rollout plan generator

Build an AI agent rollout plan: which rungs your change must climb (eval gate, shadow deploy, canary, ramp), what to watch at each, and how to roll back.

Chapter 3

Run the loop

An agent loop simulator: play the harness for a scripted run, choose append, retry, stop or escalate on each pass, and see which failures you cause.

Chapter 14

Should this be an agent?

When to use AI agents, and when plain code, one model call or a workflow does the job better. Answer ten questions and see where your task sits on the ladder.

Chapter 6

Skill token budget

Budget the context cost of agent skills: the standing menu, what one task loads under progressive disclosure, and a check of whether a description fires.

Chapter 17

Spot the injection

A prompt injection game: read a poisoned inbox, grant an agent its tools and settings, run it, and see what leaks. Keep the most capability with no leak.

Chapter 26

Three-set splitter

Split labeled examples for an LLM classifier into prompt demonstrations, a development set and a sealed test set, with a duplicate check. Runs in your browser.

Chapter 5

Tool schema linter

A free tool schema linter for LLM agents: paste a tool definition and get findings on its name, description, schema, side effects and hidden injections.

Chapter 15

Trace waterfall viewer

Load one agent run as JSON and read it as a span waterfall: missing attributes per span, flags for loops, cut results and errors, and a diff of two runs.