AI · GitHub repo
llm-council
Multi-agent system querying multiple LLMs that peer-review answers into one synthesis.
An open-source local web app created by Andrej Karpathy that routes a single prompt to multiple frontier models simultaneously. The models anonymously review and rank each other's responses before a designated chairman model synthesizes a final, refined answer. It is designed to expose diverse perspectives, reduce single-model blind spots, and answer complex questions through cross-model critique.
What it's for
- Compare responses across top frontier models side by side for a difficult prompt
- Reduce hallucinations on complex research questions through peer critique
- Synthesize multiple model viewpoints into one comprehensive answer
- Evaluate and benchmark frontier LLM reasoning capabilities on specific domains
Features
- Three-stage workflow: initial independent answers, anonymous peer review, and chairman synthesis
- Anonymous model evaluation where LLMs rank peer answers without knowing who wrote them
- Interactive tabbed web UI built with React to inspect each model's individual response and critique
- Single OpenRouter API integration to access multiple model providers without separate accounts
- Configurable council lineup and customizable chairman selection in a simple Python config
- Local file-based conversation storage using JSON
Pricing
Open source. Free and open-source code; requires your own OpenRouter API credits
Details
- Type
- GitHub repo
- Category
- AI
- Topic
- Multi-Agent Systems
- Open source
- Yes
- Platforms
- Self-hosted, Web
- For
- Developers, AI researchers, and power users seeking higher confidence answers to complex questions.
- Alternative to
- ChatGPT, Claude, Poe, Perplexity
- Tags
- llm, multi agent, openrouter, peer review, evals, python
Why it was saved
- @100xengineers: Reduces hallucinations and improves reasoning quality by having multiple frontier AI models anonymously debate and rank answers before delivering a single verified response.
Compare
Related
Python framework for building multi-modal AI agents with memory and tools
Proof-of-concept AI multi-agent trading system with paper trading and backtesting.
Open-source platform and framework for building autonomous AI agents
Open-source Python library enabling AI agents to control web browsers.
Open-source pipeline that parses resumes, checks GitHub profiles, and scores candidates
Open-source Python framework for building realtime voice and multimodal AI assistants
Updated