AI · Web app
Website Content Crawler
Apify web crawler that extracts clean text and Markdown for LLMs
An Apify Actor designed to perform deep crawls of websites and extract clean content formatted as plain text, Markdown, or HTML. It strips away navigation bars, headers, ads, and cookie banners to output structured data ready for AI models, RAG pipelines, and automated site audits. You can run it via Apify's web console or API and export datasets as JSON, CSV, or XML.
What it's for
- Scrape entire documentation sites to build RAG knowledge bases or custom GPTs
- Extract clean website text and metadata for automated SEO audits with LLMs
- Gather company blog posts to train or fine-tune models on brand voice
Features
- Adaptive crawling that switches between raw HTTP and headless Firefox via Playwright
- Automatic DOM cleanup removing headers, footers, navigation, modals, and cookie notices
- Markdown, plain text, and HTML export formats with metadata preservation
- Automated document downloading for PDF, DOC, DOCX, XLS, and XLSX files
- AI summarization option per page powered by OpenRouter
- Direct API access and integrations with LangChain, LlamaIndex, and vector databases
Pricing
Freemium. Free tier available via Apify; pay-per-usage compute credits for larger crawls
Details
- Type
- Web app
- Category
- AI
- Topic
- Web Scraping
- Open source
- No
- Platforms
- Web, API
- For
- Developers, AI engineers, and SEO specialists who need clean web data for LLM workflows.
- Alternative to
- Firecrawl, Diffbot
- Tags
- apify, crawler, scraping, rag, llm, markdown
Why it was saved
- @digitalsamaritan: Quickly extracts website content into downloadable files (such as HTML or JSON datasets) ready to feed into LLMs for analysis like SEO audits.
Compare
Related
Turn websites and documents into clean, LLM-ready markdown or structured data.
Turn any website into a structured REST API using natural language
Cloud platform and marketplace for web scraping, automation, and AI data extraction.
Open-source Python library and API for prompt-based web scraping using LLMs
AI Chrome extension for scraping web data using natural language chat prompts.
Feature and cost breakdown across cloud browser sandbox providers for scraping
Updated