# LLM Living Knowledge Base Pipeline

> Architecture framework for building an LLM-maintained markdown wiki that compounds knowledge

A conceptual framework based on Andrej Karpathy's method for organizing research into a persistent, self-updating markdown knowledge base. Instead of relying on RAG or infinite context windows, an LLM compiles raw sources into structured markdown files, queries them, and files its own outputs back into the wiki. The user steers and inspects the knowledge base via an IDE like Obsidian while the model writes and maintains the index.

## The framework

Main pipeline: (1) Sources — articles + papers, repos + datasets, images + diagrams; (2) raw/ — unprocessed files stored as-is, local images; (3) WIKI — .md knowledge base: summaries + backlinks, concept categories, auto-maintained index (~100 articles / ~400K words); (4) Q&A Agent — complex questions against full wiki, no RAG needed; (5) Output — markdown files, Marp slide decks, matplotlib plots. Feedback loop: outputs are filed back into the wiki — knowledge compounds. Support layer (all connected to the wiki): Obsidian (IDE frontend for wiki, view raw + compiled data, Marp plugin for slides; human reads, LLM writes), Lint + Heal (find inconsistent data, impute missing info, suggest new articles, web search to fill gaps), CLI Tools (search engine over wiki, web UI for direct use, CLI interface for LLM; vibe-coded, keeps growing). Looking ahead: finetune LLM on wiki data — knowledge in weights, not just context. Core insight: 'You never write the wiki. The LLM writes everything. You just steer — every answer compounds.' Tweeter's additions: agents that own their knowledge layer don't need infinite context windows — just good file organisation and reading their own indexes; cheaper, more scalable, more inspectable than stuffing everything into one prompt.

## What it is for

- Design a persistent memory layer for autonomous AI agents
- Organize large collections of research papers, repositories, and raw notes into a compounding knowledge base
- Structure an Obsidian vault that an LLM automatically maintains, links, and lints
- Prepare structured domain datasets to fine-tune future local models

## Key points

- Four-stage pipeline: raw ingestion, LLM-compiled markdown wiki, direct Q&A agent, and multi-format outputs
- Feedback loop where new answers, slides, and plots are indexed back into the wiki
- Obsidian frontend integration acting as the human inspection layer
- Automated 'lint and heal' maintenance to resolve inconsistencies and fill gaps using web search
- CLI and search tooling built specifically for model and user retrieval

## Pricing

Free

## Details

- Type: Framework
- Category: [AI](https://xiry.app/topics/ai.md)
- Topic: [Knowledge Management](https://xiry.app/topics/productivity/knowledge-management.md)
- Open source: no
- Platforms: other
- For: AI developers, researchers, and knowledge workers designing agentic workflows or personal knowledge systems
- Alternative to: Vector RAG pipelines, MemGPT
- Tags: knowledge base, llm agents, obsidian, markdown, architecture, second brain

## Why it was saved

- [@jumperz, 2 April 2026](https://x.com/jumperz/status/2039826228224430323): A complete reference architecture for an AI second brain: ingest, compile, query, file back — usable as a blueprint for agent memory systems

## Related

- [Notion](https://xiry.app/resources/notion): Connected workspace for notes, docs, databases, and AI automation. (Web app, Freemium)
- [Andrej Karpathy's LLM Knowledge Base Workflow](https://xiry.app/resources/andrej-karpathy-s-llm-knowledge-base-workflow): A workflow for using LLMs to compile and maintain personal wikis (Framework, Free)
- [Are.na](https://xiry.app/resources/are-na): Ad-free platform for saving, organizing, and connecting ideas into channels (Web app, Freemium)
- [Ars Umbris](https://xiry.app/resources/ars-umbris): Local-first IDE and framework for building typed, agentic knowledge graphs (Web app, Open source)
- [llm-wiki](https://xiry.app/resources/llm-wiki): A design pattern for building LLM-maintained personal knowledge bases in markdown (GitHub repo, Free)
- [YapThread](https://xiry.app/resources/yapthread): Turn Twitter bookmarks into a searchable AI knowledge base with citations (Web app)

Updated 11 October 2026. Page: https://xiry.app/resources/llm-living-knowledge-base-pipeline

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.