Development · Tip
Cloudflare /crawl Endpoint
Single API endpoint to crawl entire websites to Markdown, HTML, or JSON
Cloudflare's Browser Rendering /crawl endpoint extracts entire websites via a single API call without managing headless browsers or scrapers. It outputs parsed content directly into formats like Markdown, HTML, or JSON, making it ideal for LLM ingestion and content scraping pipelines.
How it works
POST https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl
Headers:
- Authorization: Bearer <apiToken>
- Content-Type: application/json
Body:
{
"url": "https://blog.cloudflare.com/"
}
One API call returns an entire site crawled, as HTML, Markdown, or JSON. No scripts. No browser management. Just the content.
What it's for
- Crawl documentation sites directly into Markdown for LLM ingestion and RAG pipelines
- Extract clean page content without writing or maintaining custom scraper code
- Convert full website structures into structured JSON feeds
Key points
- Single POST API call to crawl an entire domain or site path
- Outputs in multiple clean formats including Markdown, HTML, and JSON
- Eliminates the need to manage headless browsers or proxy networks
- Integrated into Cloudflare's Browser Rendering infrastructure
Pricing
Freemium. Included with Cloudflare Workers / Browser Rendering plans; specific /crawl pricing follows Cloudflare account tiers.
Details
- Type
- Tip
- Category
- Development
- Topic
- Web Scraping
- Open source
- No
- Platforms
- API
- For
- Developers building RAG pipelines, scrapers, and AI data extraction workflows
- Alternative to
- Firecrawl, Spider, Apify
- Tags
- web scraping, rag, cloudflare, api, crawler, markdown
Why it was saved
- @TukiFromKL: A practical shortcut for anyone building scrapers, RAG pipelines, or site-to-Markdown converters: no crawler code, no headless browser setup. The tweet frames the irony (Cloudflare built its name on blocking scrapers and now sells a crawler), but the save is the tool itself.
Compare
Related
Turn websites and documents into clean, LLM-ready markdown or structured data.
Turn any website into a structured REST API using natural language
Cloud platform and marketplace for web scraping, automation, and AI data extraction.
Open-source Python library and API for prompt-based web scraping using LLMs
AI Chrome extension for scraping web data using natural language chat prompts.
Feature and cost breakdown across cloud browser sandbox providers for scraping
Updated