# LLMs.txt generator

> Generates llms.txt and llms-full.txt markdown files from any website URL.

LLMs.txt generator extracts and cleans a website's content into structured markdown formatted for LLMs and RAG pipelines. It can be accessed directly on the web or by prepending 'llmstxt.new/' in front of any URL in the browser address bar. The tool is powered by Firecrawl and accepts an API key for complete site scraping.

Website: https://llmstxt.new/

## What it is for

- Format website documentation into llms.txt standard files for AI agents
- Prepare clean web content for retrieval-augmented generation (RAG) pipelines
- Quickly convert any target URL to LLM-ready markdown using a browser URL prefix

## Features

- Browser URL prefix shortcut (llmstxt.new/\<url\>) for quick generation
- Produces standard llms.txt link indexes with markdown descriptions
- Generates llms-full.txt consolidated full-text files
- Web scraping and parsing powered by Firecrawl
- Optional Firecrawl API key input for complete site generation
- Direct API access support

## Pricing

Freemium. Free for partial generation; requires Firecrawl API key for full generation

## Details

- Type: Web app
- Category: [AI](https://xiry.app/topics/ai.md)
- Open source: no
- Platforms: Web, API
- For: AI developers, RAG pipeline builders, and technical writers
- Alternative to: Firecrawl, Jina Reader
- Tags: llmstxt, rag, firecrawl, web scraping, markdown

## Why it was saved

- [@Sumanth_077, 13 April 2025](https://x.com/Sumanth_077/status/1911420645256110514): Prefixing any URL with llmstxt.new/ crawls, cleans and formats the site into Markdown ready for RAG pipelines, AI agents or LLMs; full generation needs a Firecrawl API key, free partial result otherwise.

## Compare

- [Alternatives to Firecrawl](https://xiry.app/alternatives/firecrawl.md)

## Related

- [Firecrawl Agent](https://xiry.app/resources/firecrawl-agent): Autonomous web data extraction API driven by prompts without needing explicit URLs. (GitHub repo, Freemium)
- [NuMarkdown-8B-Thinking](https://xiry.app/resources/numarkdown-8b-thinking): Open-source vision-language model converting complex documents into structured Markdown using reasoning. (AI model, Open source)
- [Website Content Crawler](https://xiry.app/resources/website-content-crawler): Apify web crawler that extracts clean text and Markdown for LLMs (Web app, Freemium)
- [ai-engineering-hub](https://xiry.app/resources/ai-engineering-hub): Open-source tutorials and project templates for LLMs, RAG, and AI agents. (GitHub repo, Open source)
- [Gemini Notebook](https://xiry.app/resources/gemini-notebook): AI research assistant grounded strictly in your uploaded notes and documents. (Web app, Freemium)
- [TinyFish on Monid](https://xiry.app/resources/tinyfish-on-monid): Free search and fetch endpoints plus persistent logins for AI agents. (Web app, Freemium)

Updated 10 October 2026. Page: https://xiry.app/resources/llms-txt-generator

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.