# Cloudflare /crawl Endpoint

> Single API endpoint to crawl entire websites to Markdown, HTML, or JSON

Cloudflare's Browser Rendering /crawl endpoint extracts entire websites via a single API call without managing headless browsers or scrapers. It outputs parsed content directly into formats like Markdown, HTML, or JSON, making it ideal for LLM ingestion and content scraping pipelines.

## How it works

POST https://api.cloudflare.com/client/v4/accounts/{account\_id}/browser-rendering/crawl

Headers:

- Authorization: Bearer \<apiToken\>

- Content-Type: application/json

Body:

{

"url": "https://blog.cloudflare.com/"

}

One API call returns an entire site crawled, as HTML, Markdown, or JSON. No scripts. No browser management. Just the content.

## What it is for

- Crawl documentation sites directly into Markdown for LLM ingestion and RAG pipelines
- Extract clean page content without writing or maintaining custom scraper code
- Convert full website structures into structured JSON feeds

## Key points

- Single POST API call to crawl an entire domain or site path
- Outputs in multiple clean formats including Markdown, HTML, and JSON
- Eliminates the need to manage headless browsers or proxy networks
- Integrated into Cloudflare's Browser Rendering infrastructure

## Pricing

Freemium. Included with Cloudflare Workers / Browser Rendering plans; specific /crawl pricing follows Cloudflare account tiers.

## Details

- Type: Tip
- Category: [Development](https://xiry.app/topics/development.md)
- Topic: [Web Scraping](https://xiry.app/topics/development/web-scraping.md)
- Open source: no
- Platforms: API
- For: Developers building RAG pipelines, scrapers, and AI data extraction workflows
- Alternative to: Firecrawl, Spider, Apify
- Tags: web scraping, rag, cloudflare, api, crawler, markdown

## Why it was saved

- [@TukiFromKL, 11 March 2026](https://x.com/TukiFromKL/status/2031597711808410075): A practical shortcut for anyone building scrapers, RAG pipelines, or site-to-Markdown converters: no crawler code, no headless browser setup. The tweet frames the irony (Cloudflare built its name on blocking scrapers and now sells a crawler), but the save is the tool itself.

## Compare

- [Alternatives to Apify](https://xiry.app/alternatives/apify.md)
- [Alternatives to Firecrawl](https://xiry.app/alternatives/firecrawl.md)

## Related

- [Firecrawl](https://xiry.app/resources/firecrawl-1984): Turn websites and documents into clean, LLM-ready markdown or structured data. (Website, Freemium)
- [Parse](https://xiry.app/resources/parse): Turn any website into a structured REST API using natural language (Web app, Freemium)
- [Apify](https://xiry.app/resources/apify): Cloud platform and marketplace for web scraping, automation, and AI data extraction. (Web app, Freemium)
- [ScrapeGraphAI](https://xiry.app/resources/scrapegraphai): Open-source Python library and API for prompt-based web scraping using LLMs (GitHub repo, Freemium)
- [Chat4Data](https://xiry.app/resources/chat4data): AI Chrome extension for scraping web data using natural language chat prompts. (Browser extension, Freemium)
- [Cloud Scraping Browsers Comparison](https://xiry.app/resources/cloud-scraping-browsers-comparison): Feature and cost breakdown across cloud browser sandbox providers for scraping (Framework, Free)

Updated 11 October 2026. Page: https://xiry.app/resources/cloudflare-crawl-endpoint

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.