Development · GitHub repo
Scrapling
Adaptive Python web scraping framework with anti-bot bypass and element relocation.
Scrapling is an open-source Python framework designed to handle web scraping tasks from single requests up to concurrent, large-scale crawls. It features adaptive parsing that automatically relocates page elements when website HTML structures change, along with built-in stealth fetchers to bypass bot protections like Cloudflare Turnstile. It also includes MCP server support, making it simple to plug web data directly into AI agent workflows.
What it's for
- Scrape websites protected by bot challenges and anti-bot systems like Cloudflare Turnstile
- Maintain long-term scraping jobs without breaking when page layouts change
- Provide live web browsing and data extraction tools to AI agents via MCP
- Run concurrent multi-page crawls with session reuse and proxy rotation
Features
- Adaptive element parsing that automatically relocates target elements across site updates
- StealthyFetcher and DynamicFetcher for bypassing bot detection mechanisms
- Built-in Model Context Protocol (MCP) server support for AI agent integrations
- CSS, XPath, text, and regex selector support
- Concurrent spider engine supporting multi-session scraping, browser tab pooling, and proxy rotation
- Zero-code CLI mode alongside the full Python library
Pricing
Open source
Details
- Type
- GitHub repo
- Category
- Development
- Topic
- Web Scraping
- Open source
- Yes
- Platforms
- Code library, Command line
- For
- Python developers and AI engineers building web scrapers and browser agents
- Alternative to
- Scrapy, Beautiful Soup, Playwright, Crawlee
- Tags
- scraping, crawler, python, anti bot, stealth, mcp, ai agents
Why it was saved
- @hasantoxr: Handles single requests to full-scale crawls; parser auto-relocates elements when site structure changes, fetchers bypass anti-bot systems (Cloudflare Turnstile) out of the box, and spider framework supports concurrent multi-session crawls with pause/resume and automatic proxy rotation. Claimed 774x faster than BeautifulSoup with lxml. Useful for feeding real web data into AI agents like OpenClaw.
Compare
Related
Turn websites and documents into clean, LLM-ready markdown or structured data.
Turn any website into a structured REST API using natural language
Cloud platform and marketplace for web scraping, automation, and AI data extraction.
Open-source Python library and API for prompt-based web scraping using LLMs
AI Chrome extension for scraping web data using natural language chat prompts.
Feature and cost breakdown across cloud browser sandbox providers for scraping
Updated