AI · Web app
LLMs.txt generator
Generates llms.txt and llms-full.txt markdown files from any website URL.
LLMs.txt generator extracts and cleans a website's content into structured markdown formatted for LLMs and RAG pipelines. It can be accessed directly on the web or by prepending 'llmstxt.new/' in front of any URL in the browser address bar. The tool is powered by Firecrawl and accepts an API key for complete site scraping.
What it's for
- Format website documentation into llms.txt standard files for AI agents
- Prepare clean web content for retrieval-augmented generation (RAG) pipelines
- Quickly convert any target URL to LLM-ready markdown using a browser URL prefix
Features
- Browser URL prefix shortcut (llmstxt.new/<url>) for quick generation
- Produces standard llms.txt link indexes with markdown descriptions
- Generates llms-full.txt consolidated full-text files
- Web scraping and parsing powered by Firecrawl
- Optional Firecrawl API key input for complete site generation
- Direct API access support
Pricing
Freemium. Free for partial generation; requires Firecrawl API key for full generation
Details
- Type
- Web app
- Category
- AI
- Topic
- Data Preparation
- Open source
- No
- Platforms
- Web, API
- For
- AI developers, RAG pipeline builders, and technical writers
- Alternative to
- Firecrawl, Jina Reader
- Tags
- llmstxt, rag, firecrawl, web scraping, markdown
Why it was saved
- @Sumanth_077: Prefixing any URL with llmstxt.new/ crawls, cleans and formats the site into Markdown ready for RAG pipelines, AI agents or LLMs; full generation needs a Firecrawl API key, free partial result otherwise.
Compare
Related
Autonomous web data extraction API driven by prompts without needing explicit URLs.
Open-source vision-language model converting complex documents into structured Markdown using reasoning.
Apify web crawler that extracts clean text and Markdown for LLMs
Open-source tutorials and project templates for LLMs, RAG, and AI agents.
AI research assistant grounded strictly in your uploaded notes and documents.
Free search and fetch endpoints plus persistent logins for AI agents.
Updated