# llm-council

> Multi-agent system querying multiple LLMs that peer-review answers into one synthesis.

An open-source local web app created by Andrej Karpathy that routes a single prompt to multiple frontier models simultaneously. The models anonymously review and rank each other's responses before a designated chairman model synthesizes a final, refined answer. It is designed to expose diverse perspectives, reduce single-model blind spots, and answer complex questions through cross-model critique.

Website: https://github.com/karpathy/llm-council

## What it is for

- Compare responses across top frontier models side by side for a difficult prompt
- Reduce hallucinations on complex research questions through peer critique
- Synthesize multiple model viewpoints into one comprehensive answer
- Evaluate and benchmark frontier LLM reasoning capabilities on specific domains

## Features

- Three-stage workflow: initial independent answers, anonymous peer review, and chairman synthesis
- Anonymous model evaluation where LLMs rank peer answers without knowing who wrote them
- Interactive tabbed web UI built with React to inspect each model's individual response and critique
- Single OpenRouter API integration to access multiple model providers without separate accounts
- Configurable council lineup and customizable chairman selection in a simple Python config
- Local file-based conversation storage using JSON

## Pricing

Open source. Free and open-source code; requires your own OpenRouter API credits

## Details

- Type: GitHub repo
- Category: [AI](https://xiry.app/topics/ai.md)
- Open source: yes
- Platforms: Self-hosted, Web
- For: Developers, AI researchers, and power users seeking higher confidence answers to complex questions.
- Alternative to: ChatGPT, Claude, Poe, Perplexity
- Tags: llm, multi agent, openrouter, peer review, evals, python

## Why it was saved

- [@100xengineers, 29 November 2025](https://www.instagram.com/reel/DRpR9S5iZH8/): Reduces hallucinations and improves reasoning quality by having multiple frontier AI models anonymously debate and rank answers before delivering a single verified response.

## Compare

- [Alternatives to Claude](https://xiry.app/alternatives/claude.md)
- [Alternatives to Perplexity](https://xiry.app/alternatives/perplexity.md)
- [Alternatives to ChatGPT](https://xiry.app/alternatives/chatgpt.md)

## Related

- [phidata](https://xiry.app/resources/phidata): Python framework for building multi-modal AI agents with memory and tools (GitHub repo, Open source)
- [ai-hedge-fund](https://xiry.app/resources/ai-hedge-fund): Proof-of-concept AI multi-agent trading system with paper trading and backtesting. (GitHub repo, Open source)
- [AutoGPT](https://xiry.app/resources/autogpt): Open-source platform and framework for building autonomous AI agents (GitHub repo, Freemium)
- [browser-use](https://xiry.app/resources/browser-use): Open-source Python library enabling AI agents to control web browsers. (GitHub repo, Open source)
- [Hiring Agent](https://xiry.app/resources/hiring-agent): Open-source pipeline that parses resumes, checks GitHub profiles, and scores candidates (GitHub repo, Open source)
- [LiveKit Agents](https://xiry.app/resources/livekit-agents): Open-source Python framework for building realtime voice and multimodal AI assistants (GitHub repo, Open source)

Updated 3 October 2026. Page: https://xiry.app/resources/llm-council

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.