Development · Tip
Parallel scraping fleet pattern with Claude Code and boat.dev
Pattern for orchestrating mass parallel browser scraping fleets using AI subagents
A workflow pattern that pairs Claude Code with cloud sandboxes like boat.dev to run parallel browser-based scraping jobs. It uses subagents in isolated virtual machines with human-like input emulation to scrape sites concurrently and aggregate data into structured reports. It serves as a lower-cost alternative to dedicated managed browser platforms for large research or lead generation projects.
How it works
Ask Claude Code: "use boat.dev VMs in parallel (100 of them) to spawn browsers and prompt subagents inside to go use smooth mouse movement to scrap each different subsets of <whatever you want to scrap> undetected. then generate an aggregate CSV & website report". Author's scale claim: 2k concurrent 24/7 VMs → ~29M 3-min sessions/month for ~$26k vs ~$155k on Browserbase. Also an onboarding prompt: "onboard me on boat.dev till we got a first VM & 'boat prompt' working on it with AI credentials, then spawn tens of VMs & agents to find prospective customers for my company called <company domain>".
What it's for
- Scrape large directories of company, pricing, or product data across hundreds of concurrent sessions
- Automate lead generation by having parallel subagents research target prospects
- Aggregate web data into CSV summaries and static web reports via natural language CLI instructions
Key points
- Concurrent VM orchestration across dozens or hundreds of instances
- Subagent browser control with smooth mouse movement emulation to bypass basic bot detection
- Automatic aggregation of distributed scraping results into unified CSV reports
- Prompt-driven setup and execution using Claude Code CLI
Details
- Type
- Tip
- Category
- Development
- Topic
- Web Scraping
- Open source
- No
- Platforms
- Command line, API
- For
- Developers and growth engineers building large-scale automated data scraping and research fleets
- Alternative to
- Browserbase
- Tags
- web scraping, claude code, ai agents, automation, virtual machines
Why it was saved
- @AniC_dev: Concrete recipe used to gather feature, pricing & funding data for 366+ sandbox providers.
Compare
Related
Turn websites and documents into clean, LLM-ready markdown or structured data.
Turn any website into a structured REST API using natural language
Cloud platform and marketplace for web scraping, automation, and AI data extraction.
Open-source Python library and API for prompt-based web scraping using LLMs
AI Chrome extension for scraping web data using natural language chat prompts.
Feature and cost breakdown across cloud browser sandbox providers for scraping
Updated