AI · Article or course
AutoHarness
Framework for LLMs to automatically synthesize their own execution harnesses
AutoHarness is a research framework by Google DeepMind that allows LLMs to automatically write and refine their own code harnesses using iterative tree search and environment feedback. Instead of relying on hand-crafted scaffolding or costly fine-tuning, the agent generates external code (such as action verifiers or complete policies) to prevent illegal actions. In benchmarks across 145 TextArena games, it eliminated illegal moves and enabled smaller models like Gemini-2.5-Flash to outperform larger models.
What it's for
- Prevent LLM agents from submitting invalid or illegal actions in constrained environments
- Synthesize executable Python control harnesses without manually writing scaffolding
- Generate standalone Python code policies to eliminate LLM inference costs at test time
Features
- Thompson sampling-guided tree search over generated program candidates
- Critic and Refiner feedback loop using environment rollout errors
- Harness-as-action-verifier mode acting as a learned rejection sampler
- Harness-as-policy mode generating pure Python code to run inference-free
- 100% legal action success rate across 145 TextArena games
Pricing
Free
Details
- Type
- Article or course
- Category
- AI
- Topic
- Agent Scaffolding
- Open source
- No
- Platforms
- Web
- For
- AI researchers and agent developers building tool-use workflows and game-playing systems
- Alternative to
- Voyager, Eureka, Reflexion
- Tags
- research, agents, code generation, deepmind, llm scaffolding
Why it was saved
- @dair_ai: Introduces 'code as harness': instead of hand-crafting the scaffolding (tools, code execution, file systems, APIs) around an LLM agent, the agent itself generates its harness via iterative code refinement with environment feedback. On TextArena games, Gemini-2.5-Flash with an auto-synthesized harness prevents all illegal moves across 145 games and outperforms larger models like Gemini-2.5-Pro; pushed further, the model generates the entire policy in code, beating Gemini-2.5-Pro and GPT-5.2-High on 16 one-player games, cheaper than a larger model.
Related
A research paper formalizing architecture and context engineering for CLI coding agents.
Fleet of autonomous AI agents that research, build, and deliver work
Autonomous AI agent that executes complex research, coding, and workflow tasks
Generative AI video model producing cinematic footage with native audio
AI research assistant grounded strictly in your uploaded notes and documents.
Open-weights fine-tune of Llama 3.1 built for agentic function calling.
Updated