AI · Framework
You Don't Understand AI Agents
A framework prioritizing end-user business processes and evals over agent architectures.
An essay and mental model by Wayne Hamadi reframing AI agents as systems that act autonomously and proactively by mirroring a specific human worker's reward system. It argues that agent builders waste time on complex multi-agent frameworks, vector databases, and memory architectures instead of deeply understanding the business process and designing granular evals.
The framework
Author: Wayne Hamadi (@wayne_hamadi). Link: x.com/i/article/1867032485768597505.
KEY THESIS: An AI agent is 'a system that can act autonomously and proactively.' The technology (RAG, memory, multi-agents, loops, LLMs) is the least interesting part. The most interesting part is 'Anna' — an end user following a business process. Anna already has agency; the agent is just a bot version of Anna, 'a machine with the same reward system as Anna.' Building good agents means writing evals that force you to understand Anna better than anyone else.
HISTORY:
- Pre-ChatGPT: traditional apps with explicit user interfaces; automation done by engineers or low-code workflow engines (Zapier, MuleSoft, n8n, Windmill) that remained too technical for most knowledge workers.
- Nov 2022–Apr 2023: text-davinci-003 was 'an auto-completer'; GPT-3.5 answered reliably, enabling early self-prompting experiments but not sustained loops.
- March 14, 2023: GPT-4 0314 was a step change — 'the best model until Claude Sonnet 3.5 in June 2024'; excellent at self-prompting, which is what made agentic software possible.
- Toran Bruce Richards (a game developer, used to 'while true' loops) built AutoGPT days later — the first AI agent, which the author calls 'chat with side effects.' All impressive AutoGPT demos came from GPT-4 (most people lacked API access).
WHAT NOT TO FOCUS ON (crossed-out diagram): multi-agent architectures with shared memory, digital business context (vector stores, knowledge graphs, structured/unstructured data), physical environment context.
THE ANNA EXAMPLE: Anna is a Coca-Cola recruiter whose ATS only sends junior applicants for a senior PM role. With an agentic ATS she states the high-level problem; the ATS troubleshoots and returns a diagnostic. The design questions are three evals:
- A (incident): if >40% of applicants for a job titled 'senior' are junior, report an incident.
- B (diagnostic accuracy): if the diagnostic claims a mismatch between two fields, verify the mismatch actually exists.
- C (fix): for a field-mismatch diagnostic, the agent should alert Anna and keep alerting unless she declines — this is where the agent goes in the loop.
Lesson on evals vs benchmarks: eval C ranks lower — passing A and B beats A and C, because Anna can trivially edit the job description herself. Evals also rank/prioritize what's worth automating.
DIFFERENCE BETWEEN EVAL AND BENCHMARK: evals are the individual pass/fail checks; the benchmark is the ranked combination you optimize the agent against.
What it's for
- Design practical AI agents by modeling existing human workflows instead of technical architectures
- Define and prioritize agent evaluation criteria (evals) before writing automation loops
- Identify where human-in-the-loop oversight is necessary versus automatable
Key points
- Core thesis that an agent is a software proxy of an existing knowledge worker (the 'Anna' persona) with identical reward incentives
- Argument de-emphasizing multi-agent frameworks, shared memory, and vector databases in favor of domain understanding
- Distinction between individual pass/fail evals and ranked benchmarks used for optimization
- Three-tier evaluation hierarchy: problem detection (incident), diagnostic accuracy, and autonomous resolution
Pricing
Free
Details
- Type
- Framework
- Category
- AI
- Topic
- Design Systems
- Open source
- No
- Platforms
- Web
- For
- AI engineers, product managers, and founders building autonomous agent systems
- Tags
- ai agents, evals, mental model, system design, product management
Why it was saved
- @wayne_hamadi: A mental model for designing agents: focus on the user's business process and evals, not frameworks or architectures.
Related
Curated collection of plain-text design systems extracted from popular tech websites.
Quick typography, color palette, and iconography reference for clean product interfaces.
Open-source, token-driven design system and Figma UI kit by GR8 Tech.
AI-readable DESIGN.md style references extracted from popular websites.
Architectural framework categorizing LLM workflows and autonomous agent patterns
Curated UI and UX checklists for websites, apps, and user flows
Updated