How to choose an AI agent framework: decide which jobs you need it to take over before you look at a single name. Score each candidate, and a bare loop beside it, on those jobs, using evidence from a short spike that crashes, pauses, steers and exports it on purpose. Keep your prompts, tools, test cases and traces outside whatever you pick.
I wrote this for the tech lead who has to pick what the team’s next year of agents will be built on and then defend the pick in a design review. You will not find a ranking here. The book this site belongs to names no winners, because the products change faster than a book can, and the pages that do rank them tend to be written by someone with a product on the list. What you will find is a rubric you can copy, a three-day test protocol, and the red flags that should end an evaluation early.
Why can’t a framework ranking tell you which one to use?
Because a ranking answers a question about products, and yours is about one agent, one team and one way of running it. The top results for this query are lists written by vendors, organized by product name, and they date as soon as the products ship their next release.
The bias is visible on the page. A no-code platform’s list (February 2026) ranks its own product first and another no-code vendor second. A framework vendor’s guide (June 2026) has a sensible list of criteria, then a comparison table that marks tool-protocol support “Native” for its own product and one other and “Community” for the remaining five.
Neither page tells you how to check a single claim. An independent author’s guide (September 2025) explains each product’s design idea well, and is organized by product, so it ages with them.
Chapter 27 of the book (in the full book) gives the reason to change the question: “I find this inventory more useful than any framework comparison, because it converts a shopping question into a custody question, and custody has a discipline: never store what you own inside what you rent.” The framework is rented. The rest of this post is about choosing what to rent and testing it before you sign.
Has anyone measured frameworks against each other?
Once, as far as I could find, and the result is a warning about orchestration code. A demo paper by Milev and Kanagala at ACM CAIS in May 2026 held one model and one tool server fixed across six frameworks and SDKs and scored three scenarios. Its abstract reports that “as complexity grows, traditional frameworks require 2–4× more scenario-specific orchestration code yet gain no correctness advantage.”
Read it as one team’s result on one model with three tasks, from a demo track, and framed around one provider SDK’s generic loop as the comparison point. It does not compare any framework with a hand-written loop. What it supports is narrow and useful: more orchestration code is a cost you should be able to justify, not evidence of a better agent.
Do you need an agent framework at all?
Only if it passes a test for something you would otherwise write yourself. Chapter 3, The Agent Loop (in the full book) lists what a framework adds to a loop: state and persistence, retries, tracing, memory management and multi-agent orchestration. All five are plumbing. None changes what the model does on a single pass.
That is also why a team that has never written the bare loop should write it before choosing. The tutorial on building an AI agent from scratch does it in one file, and the post on what an agent harness is covers which named need justifies each piece of plumbing. Chapter 3 states the payoff: “When a framework’s agent misbehaves, there are only four places to look—the parts you have just met, whatever names the framework has painted over the doors.” A team that knows the four parts can read a framework’s documentation as a list of claims to test.
The question “framework or not” is covered in those two posts. This one assumes you have at least one named need and are now choosing among candidates, with the bare loop kept in the comparison as a baseline.
What does an AI agent framework actually own?
An AI agent framework can take over eight jobs, and each one is a separate selection criterion with its own test. A candidate that is excellent at orchestrating several agents and silent on resuming after a crash is two different answers, and only one of them may matter to you.
Pick the situation that describes your agent to narrow the table to the jobs it makes important. Every agent needs the first three rows.
| Job the framework can own | What it does | Test that checks it | Matters when |
|---|---|---|---|
| The loop and its stop rules | Calls the model, dispatches tools, decides whether to go around again | Can you cap steps, tokens and time, and plug in a programmatic success check? | every agent |
| The tool contract and errors | Turns your functions into tool definitions and returns their results | Do tool failures reach the model as structured errors, or vanish as empty results? | every agent |
| Visibility of the prompt | Assembles what is sent to the model on each call | Can you see the exact request, unedited, for every call? | every agent |
| State and recovery | Saves the run so another process can resume it | Kill the process mid-run; does it resume without repeating a side effect? | runs outlive a process, runs go unwatched |
| Approvals and steering | Pauses for a person and resumes; accepts a correction mid-run | Pause for an approval, restart the service, approve; then inject a note mid-run | people approve steps |
| Tracing hooks | Records each model call, tool call, timing and cost | Can you export a step-level trace to a store you choose, in a documented format? | runs go unwatched, people approve steps |
| Eval hooks | Lets you run test cases against the agent without a person | Can you run your cases headless, with the model stubbed, and get per-case results? | runs go unwatched |
| Multi-agent orchestration | Hands work between several agents and merges results | Can you see and bound every handoff, and run each agent alone? | several agents |
| The model adapter | Hides the differences between model providers | Swap the provider; how many files change, and does any code read a provider’s raw response? | the provider may change |
A checkpoint is a saved snapshot of a run’s working state from which a fresh process can continue. An approval gate is a point where the run stops until a person says yes. A trace is the step-by-step record of what the agent did. Those three rows are where the vendor guides are thinnest, and where the framework exits I read about began.
Which of those jobs should stay yours whatever you pick?
The plumbing in each row can be rented; the content flowing through it should stay in your files. The post on build vs buy for AI agents walks the layers one by one. In short: prompts and instructions, tool definitions, the eval set (your test cases with their correct outcomes), the approval policy and the traces are yours. A framework that can only hold them in its own formats is storing what you own inside what you rent.
Practitioners who have built many of these arrive at the same split. Dex Horthy’s 12-Factor Agents (2025) names three of its factors “Own your prompts”, “Own your context window” and “Own your control flow”. It also reports, of the founders he talked to, “Most of them are rolling the stack themselves.” Treat that as one well-connected observer’s impression, not a survey.
What kinds of agent framework are there?
There are roughly six kinds, and they differ mostly in how much of the run they take over. The names below are examples of each category as of this writing, not recommendations, and any one product can sit across two rows or move between them.
| Category | What it tends to own | Where lock-in tends to sit | Examples |
|---|---|---|---|
| Model-provider agent SDK | The loop, tool calling, tracing, often sessions | One provider’s models, response shapes and commercial terms | OpenAI Agents SDK, Claude Agent SDK, Google’s Agent Development Kit |
| Graph or state-machine orchestration library | Explicit states and transitions, checkpointing, human pauses | Agent logic written as the library’s graph | LangGraph, Mastra’s workflows |
| Role- or conversation-based multi-agent framework | Agents with roles that hand work to each other | Control flow implied by the roles, hard to bound | CrewAI, AutoGen |
| Thin typed library | Typed tool definitions, structured output, a small loop | Little beyond its tool wrapper | Pydantic AI, smolagents, Vercel’s AI SDK |
| Durable-execution engine | Journaling and replay of every step; long waits | Code written to the engine’s determinism rules | Temporal, Restate, Inngest |
| Visual or low-code builder | The whole flow, drawn on a canvas | Logic that exists only in the builder’s format | n8n, Dify, Flowise |
Two rows deserve care. A model-provider SDK can change its terms as well as its code: one provider’s billing for usage through its agent SDK was announced in May 2026, paused in June and replaced in October, and a commenter who was in the middle of moving a workflow onto it asked: “can these people just make up their minds?” A durable-execution engine is not an agent framework, but it is often what the state and approvals rows need, paired with a thin library. The post on durable execution for AI agents covers how it works.
How to choose an AI agent framework with a scoring rubric
Weight each criterion by your agent’s needs, score every candidate from evidence, and let any zero on a must-have disqualify it. The rubric below is this post’s own, built from the eight jobs plus exit cost and operational fit. Copy it into the design document before the spike, so the weights are set before anyone has a favorite.
AGENT FRAMEWORK RUBRIC: <agent name> Date: <date> Owner: <name>
Weights (set BEFORE the spike):
3 = named need, a must 1 = useful 0 = not needed for this agent
Scores (from spike evidence only):
2 = passes the test as shipped
1 = passes only in part, or only with glue we wrote in under a day
0 = fails, or untested, or "the docs say so"
Gate: a weight-3 criterion scored 0 disqualifies the candidate.
Result: sum(weight x score) / sum(weight x 2), as a percentage.
Pick: the highest result among columns with no failed gate; on a tie,
the column with fewer dependencies, the baseline first.
If every column fails a gate: add glue and rescore, or test a combination
as one more column. Never change the weights after the spike.
Weight Baseline Cand. A Cand. B
C1 Visibility: exact request per call seen [ ] [ ] [ ] [ ]
C2 Loop control: caps, success check, flow [ ] [ ] [ ] [ ]
C3 Tools: plain functions, structured errors,
permissions scoped per tool [ ] [ ] [ ] [ ]
C4 State: survives a kill, no repeated write [ ] [ ] [ ] [ ]
C5 Approvals and steering across a restart [ ] [ ] [ ] [ ]
C6 Tracing: step-level export, our store [ ] [ ] [ ] [ ]
C7 Evals: our cases headless, model stubbed [ ] [ ] [ ] [ ]
C8 Model adapter: swap provider in one place [ ] [ ] [ ] [ ]
C9 Multi-agent: bounded, visible handoffs [ ] [ ] [ ] [ ]
C10 Exit: files importing it, glue to rewrite [ ] [ ] [ ] [ ]
C11 Fit: deployment shape, language, license,
upgrade and deprecation policy [ ] [ ] [ ] [ ]
Gates failed:
Glue we wrote (what, where, how long):
Decision and the named need each rented part serves:
Reopen when: <a must-have fails in production, a breaking upgrade,
a change of terms, a new named need>
The baseline column is the bare loop your team wrote. It usually passes the first three criteria, fails state and approvals, and is partial on tracing, and that pattern is the point: the criteria a baseline fails are the plumbing a candidate has to earn its place by supplying. If a candidate fails the same ones, it adds a dependency and saves you nothing.
The rule that untested claims score zero is the one teams resist. It is also the one that matters, because the documentation describes the happy path and the spike exists to try the unhappy one. Schluntz and Zhang’s essay on building effective agents (2024) puts the stake plainly: “Incorrect assumptions about what’s under the hood are a common source of customer error.”
How do you run a framework spike in three days?
Build the same thin task in each candidate and in the bare loop, then spend more time breaking it than building it. The task should include one read tool, one tool that writes, one approval and about ten test cases with known outcomes. Three days per candidate is a guide, not a rule; the order matters more than the length.
Day one builds the task and reads the prompt. Hamel Husain’s method from 2024 still applies to any framework: intercept the calls it makes to the model and read what it actually sends. His reason is the one to carry into a review: “I’m wary of frameworks that distance the human too far from LLMs” (Husain, 2024).
Day two breaks it on purpose, following Chapter 18 (in the full book). Its test for recovery is blunt: “a checkpoint you have never restored from is a guess.” The chapter also warns about defaults you did not write: “There is a subtlety that bites teams precisely because each layer of it is sensible alone: retries nest.” A framework’s retries inside your tool’s retries inside a workflow engine’s retries multiply into dozens of calls per failure.
The third day tests the exit. Export the traces, run the cases without a person, swap the model for a scripted stub, and count the files that import the framework. The last number is the cheapest estimate of exit cost you will get.
- Build the same thin task in the candidate and in the bare loop: one read tool, one write tool, one approval, about ten test cases.
- Capture the exact request sent to the model on every call and read it end to end.
- Make the read tool fail and confirm the model receives a structured error, not an empty result.
- Kill the process right after the write tool succeeds and before the next checkpoint; resume and check the write did not run twice.
- Pause for the approval, restart or redeploy the service, then approve; confirm the run resumes and the approval is recorded with who, when and what they saw.
- Send a correction mid-run and check it lands at the next step without a restart.
- Change the prompt or a tool’s schema between a crash and its resume, and see what the framework does.
- Count every retry layer active by default, including the framework’s own.
- Export a step-level trace of one run to a store you chose, in a documented format.
- Run all test cases headless with the model replaced by a scripted stub, and get per-case results.
- Swap to a second model provider and count the files that change.
- Count the files that import the framework, and write down every piece of glue you added and how long it took.
The prompt-change item comes from Chapter 18’s warning on replay: “if you depend on replay, record the versions each step depended on.” A framework that resumes silently onto a changed prompt is replaying a run that never happened. Whether yours does is a fact you can learn in an hour.
What should a framework do about approvals and steering?
It should let a run wait for a person for hours, survive a restart while waiting, and accept a correction without throwing the run away. Most guides list “human-in-the-loop” as a checkbox. The test is what happens on the day a deploy lands while twelve runs are waiting for a manager.
Chapter 18 sets the standard for the approval itself: “make the approval itself durable state, recorded with who granted it, when, and a fingerprint of the exact artifact they saw”. An approval held in a process’s memory, as a blocking call waiting on a callback, fails that standard the first time the process restarts.
Steering is the requirement most often missed. Chapter 24 (in the full book) builds it from three parts and then says what that means for the architecture: “Between the three of them, notice how little is novel: a checkpointer, a plan file, a message queue. Steering is a transport-and-state problem wearing a UX costume, and teams that treat it as a skin to paint on later discover they have to re-architect the loop to get it.”
That sentence is a selection criterion. If your agent’s users will ever need to redirect a run, the framework’s state model decides whether they can, because, in the chapter’s restatement of an earlier design law, “interruption is only cheap if resuming is”. The post on approval gates by consequence covers which steps deserve a gate at all.
What are the red flags in an agent framework?
The red flags are the places where a framework hides the run from you or holds what you own. Any one of them is reason to stop the spike and write down why.
- You cannot see the prompt. If the exact request sent to the model is not available on every call, every later debugging session starts blind. A framework author makes the same point about his own category: “Agent abstractions can make it easy to get started, but they can often obfuscate and make it hard to make sure the LLM has the appropriate context at each step” (Chase, 2025).
- Tool errors disappear. A failure returned as an empty result lets the model write a confident summary of data it never read. Chapter 18’s rule is short: “the model can only recover from a failure it can see.”
- Approvals live in process memory. A pause that does not survive a restart is not an approval gate.
- State or traces live only in the vendor’s service. Export is the test; a dashboard you can look at is not one.
- Tools must be written in the framework’s own format. A tool definition that passes the tool contract linter as a plain function survives a change of framework with a new wrapper.
- The deployment shape does not match yours. One team on Hacker News found a provider SDK “oriented toward single-user CLI use” when it needed a multi-tenant web backend (April 2026).
- Upgrades break without migration notes. Read the changelog for the last several releases before the first line of code.
The flags about hidden internals have a public exit story behind them. Octomind ran a popular framework in production for over twelve months before removing it in 2024, and wrote: “When our team began spending as much time understanding and debugging [the framework] as it did building features, it wasn’t a good sign” (Octomind, 2024). That framework has changed a great deal since. The pattern has not: the cost showed up as debugging time long before anyone counted it.
Worked example: scoring two candidates for an approval-heavy agent
In this example the rubric disqualifies the candidate with the friendliest demo and picks the one with the most glue to write. The agent, candidates and scores are illustrative. A procurement team wants an agent that drafts purchase orders from requests, waits for a manager’s approval (often for hours), then posts approved orders to the finance system. It runs as a shared web service for several teams.
The tech lead sets weights before the spike. Visibility, loop control, tools, state, approvals, tracing (finance audits every posted order) and deployment fit are musts, weight 3. Evals, the model adapter and exit cost are useful, weight 1. Multi-agent orchestration is not needed, weight 0.
The maximum is 2 × (7 × 3 + 3 × 1) = 48 points.
Three columns are scored: the team’s bare loop, Candidate A (a graph-style orchestration library with a built-in checkpointer) and Candidate B (a model-provider agent SDK).
| Criterion | Weight | Baseline | A | B |
|---|---|---|---|---|
| C1 Visibility | 3 | 2 | 1 | 2 |
| C2 Loop control | 3 | 2 | 2 | 2 |
| C3 Tools and errors | 3 | 2 | 2 | 2 |
| C4 State and recovery | 3 | 0 | 1 | 0 |
| C5 Approvals and steering | 3 | 0 | 1 | 0 |
| C6 Tracing | 3 | 1 | 1 | 2 |
| C7 Evals | 1 | 2 | 1 | 2 |
| C8 Model adapter | 1 | 2 | 2 | 0 |
| C9 Multi-agent | 0 | n/a | n/a | n/a |
| C10 Exit cost | 1 | 2 | 1 | 2 |
| C11 Deployment fit | 3 | 2 | 2 | 0 |
| Points (of 48) | 33 | 34 | 28 | |
| Score | 69% | 71% | 58% | |
| Gates failed | C4, C5 | none | C4, C5, C11 |
The arithmetic for A: the weight-3 scores sum to 10, times 3 is 30, plus 4 from the weight-1 rows, for 34 of 48, or 71%. The baseline’s 33 and B’s 28 come out the same way.
B failed three gates in the spike. Its sessions were stored on the local disk of one process, so a killed run did not resume on another instance, and its approval was a callback held in memory, so a restart lost it. Those failures are about fit, not quality: for a single-user tool on one machine, C11 would score 2 and the picture would change.
A passed every gate but scored 1 on four musts, and those are the glue the team now owns. Its checkpointer resumed the killed run, but the posting call in flight ran again, so the team added an idempotency key to it. That key, a trace exporter to the team’s own store, a steering inbox the run checks between steps and a debug hook to log the assembled prompt took under a day each, about two and a half days in total. The harness grid builder is a quick way to lay those controls out and see which cells are still empty.
The baseline’s two zeros are the case for adopting anything. Writing durable state and approvals by hand is the work A saves. A fourth option, B paired with a durable-execution engine, was not tested; the rubric scores a combination the same way, as one column.
Agent framework vs building from scratch: when does the baseline win?
The baseline wins when it fails no gate and no candidate outscores it, which is common for a single loop that finishes in one sitting. With no need for state, approvals or unattended tracing, every framework criterion becomes a weight of 0 or 1, and the bare loop’s visibility and zero exit cost decide it.
It also wins later than people expect. A loop that starts bare can add a checkpoint, a model client seam and structured logs one named need at a time. Chapter 18’s scaling law is the guide for when: “harness investment should scale with how long the codebase must live, how many people must trust the agent’s output, and how unattended the agent runs.” Re-run the rubric when one of those three changes.
Where does this method not apply?
It does not apply to throwaway prototypes, and it has three other limits. A weekend prototype needs no rubric; pick whatever the team knows, and write the rubric when the prototype gets a second user.
Second, the scores are judgments recorded with evidence, not measurements. Two leads can weight the same agent differently, which is why the weights are set before the spike and written down.
Third, this method chooses a framework for an agent your team builds. Buying a finished agent product is a different decision with different tests, covered in the build vs buy post linked above.
Fourth, I found no controlled study that compares frameworks with a hand-written loop on the same model and tasks. The one fixed-model comparison I found compares frameworks with each other. The rubric is a way to decide without that evidence, by producing your own on the only workload that matters to you.
The criteria outlast the names
Every product on this year’s lists will be renamed, merged or replaced, and the eight jobs will still be the jobs. That is the durable answer to how to choose an AI agent framework: name the jobs you need done, weight them before the spike, score from tests you ran, and keep what you own in your own files.
State, recovery, retries and the harness debate are in Chapter 18, “Reliability, State, and the Harness”, and steering and approvals are in Chapter 24, both in the full book. The framework question at the scale of one loop is settled in Chapter 3, The Agent Loop (in the full book). The Agents at work guide collects the related posts and tools, and you can see the formats.
Questions readers ask
- How do you choose an AI agent framework?
- List the jobs you need a framework to take over for this agent, such as surviving restarts, pausing for approvals or tracing runs nobody watched. Weight each one, then score two candidates and a bare loop on the same rubric, using evidence from a short spike that crashes, pauses, steers and exports each candidate. Drop any candidate that fails a job you marked as a must.
- What is the best AI agent framework?
- There is no best one apart from a specific agent and team. Most rankings are written by vendors and date within months. The durable answer is the candidate that passes the tests for your named needs, such as resuming after a crash or holding an approval across a deploy, while keeping your prompts, tools, eval cases and traces in your own files.
- What is the difference between an AI agent SDK and an agent framework?
- The words overlap. A model provider's agent SDK usually packages the loop, tool calling and some tracing around that provider's models. A framework usually adds orchestration and state on top and supports several providers. Judge either by the same jobs and tests, and check how tightly it ties you to one model provider's response shapes and terms.
- Should you use an agent framework or build from scratch?
- Build the small loop first if nobody on the team has, because it shows what each part is for. Adopt a framework when it passes a test for something you would otherwise write yourself, such as crash-safe resumption or durable approvals. Score the bare loop beside every candidate so the comparison shows what the framework actually saves.
- How long should evaluating an agent framework take?
- A focused spike of about three working days per candidate is enough to produce evidence: one day to build the same thin task, one day to break it on purpose, one day to test the exit. Longer evaluations tend to drift into feature tours. Shorter ones skip the crash and approval tests, which are the ones that decide most choices.
Read the first part offline.
Get the free PDF sample, the preface and all of Part I, plus a two-page checklist: does this need an agent at all, and would you know if it failed? Then a few short emails on verifying agents. One-click unsubscribe.
Sources
- Erik Schluntz and Barry Zhang, Anthropic (2024). Building Effective Agents
- Harrison Chase, LangChain (2025). How to think about agent frameworks
- Hamel Husain (2024). Fuck You, Show Me The Prompt.
- Octomind (2024). Why we no longer use LangChain for building our AI agents
- Dex Horthy, HumanLayer (2025). 12-Factor Agents
- Roberto Milev and Uday Kanagala, ACM CAIS demo (2026). Arena: Benchmarking AI Agent Frameworks Under Fixed-Model Conditions
- Aron Schuhmann, Mastra (2026). AI agent framework: a practical guide to choosing and using the right one
- Omid Ghiam, Gumloop (2026). 6 best AI agent frameworks (and how I picked one) in 2026
- Nir Diamant (2025). How to Choose Your AI Agent Framework
- Hacker News thread (2026). Ask HN: What makes an AI agent framework production-ready vs. a toy?
- Hacker News comment (2026). Comment on a change to agent SDK billing
- Hacker News thread (2026). We're pausing the Agent SDK credit change
- Hacker News thread (2026). Claude Agent SDK vs. Messages API for a Production AI Assistant?