Development · GitHub repo
ScrapeGraphAI
Open-source Python library and API for prompt-based web scraping using LLMs
ScrapeGraphAI is a Python library and managed API that uses large language models and graph logic to extract structured data from websites and documents. Instead of writing and maintaining CSS selectors or XPath rules, users specify the data they need using natural language prompts. It handles dynamic pages, multi-page crawling, and integrates with major AI agent frameworks.
What it's for
- Extract structured JSON from websites using natural language prompts without writing selectors
- Scrape JavaScript-rendered dynamic web pages without manual browser automation setup
- Integrate web scraping capabilities into LLM agents built with CrewAI, LlamaIndex, or Agno
Features
- Prompt-based data extraction from raw HTML, XML, and document formats
- Graph-based scraping pipelines that dynamically determine extraction workflows
- Managed API option alongside the open-source Python library
- Support for local and commercial LLMs via LangChain and direct integrations
- Compatible with AI frameworks such as CrewAI, LlamaIndex, Smithery, and n8n
- No proxy or selector maintenance required for changing page structures
Pricing
Freemium. Open-source Python library is free (MIT); managed cloud API offers free tier and paid usage plans
Details
- Type
- GitHub repo
- Category
- Development
- Topic
- Web Scraping
- Open source
- Yes
- Platforms
- Code library, API, Linux, macOS, Windows
- For
- Developers and data engineers building AI workflows or scraping sites without maintaining brittle selectors
- Alternative to
- Apify, Bright Data, Octoparse
- Tags
- web scraping, python, llm, data extraction, open source, agents
Why it was saved
- @criscatalyst: Replaces paid scraping services like Apify and Bright Data by letting you scrape multi-page sites and extract specific fields just by describing what you need in plain English.
- @thevibefounder: Replaces complex scraper development and expensive services like Apify or Octoparse by letting you scrape URLs with a single text prompt without maintaining selectors or code.
Compare
Related
Turn websites and documents into clean, LLM-ready markdown or structured data.
Turn any website into a structured REST API using natural language
Cloud platform and marketplace for web scraping, automation, and AI data extraction.
AI Chrome extension for scraping web data using natural language chat prompts.
Feature and cost breakdown across cloud browser sandbox providers for scraping
Single API endpoint to crawl entire websites to Markdown, HTML, or JSON
Updated