Marketing & Sales · Tip
Free Backlink Data via Common Crawl's Web Graph
Extract domain backlink data from Common Crawl datasets using Bash
This tip and script show how to query Common Crawl's free web graph datasets to extract backlinks for any target domain. It provides an alternative to expensive SEO subscriptions by letting you calculate top linking domains and host counts locally using simple command-line tools.
How it works
TIL you can pull the backlinks to any domain for free using Common Crawl's web graph instead of a paid service. A tiny bash script does it: https://gist.github.com/retlehs/cf0ac6c74476e766fba2f14076fff501. Example output (linking_domain / num_hosts): github.io 284009, substack.com 199844, cloudfront.net 89247, godaddy.com 60673, google.com 41269, ddns.net 7170, github.com 6038, run.app 4187, auth0.com 2397, adobe.com 2260, uservoice.com 2080, cmu.edu 1923, js.org 1403, wordpress.org 1271, wikimedia.org 1094, mozilla.org 960, heroku.com 886, hubspot.com 574, uberspace.de 573, userecho.com 548, hosteurope.de 424, pressidium.com 296, pantheon.io 278, dreamhost.com 269, seravo.com 266, ithome.com.tw 159, kinsta.com 151. Longer write-up: https://benword.com/the-free-data-sitting-in-common-crawl — backlink tools, domain authority scores, historical web archives are all sold as subscriptions but derivable from Common Crawl's free quarterly release.
What it's for
- Extract top referring domains for a competitor without paying for SEO tools
- Audit incoming backlink volume and host counts for your own site
- Process quarterly Common Crawl releases to track changes in web graph connectivity
Key points
- Bash script to query and parse public Common Crawl web graph releases
- Outputs referring domains along with associated link and host counts
- Avoids reliance on costly recurring SEO tool subscriptions
- Leverages free, openly distributed quarterly web crawl datasets
Pricing
Free
Details
- Type
- Tip
- Category
- Marketing & Sales
- Topic
- Search Engine Optimization
- Open source
- Yes
- Platforms
- Command line, Linux, macOS
- For
- SEO professionals, data engineers, and web developers
- Alternative to
- Ahrefs, Semrush, Moz
- Tags
- seo, backlinks, common crawl, bash, web scraping, data analysis
Why it was saved
- @retlehs: Replaces backlink tools that charge hundreds a month; shows how to derive backlinks (top linking domains and host counts) for any domain from Common Crawl's free quarterly release.
Compare
Related
Empirical SEO findings on how AI models and Google AI Overviews cite content
Checklist of 18 trust signals to improve website About pages for SEO
On-page SEO practice for targeting primary keywords in homepage headings
Turn Claude Code into an automated end-to-end SEO pipeline using external APIs
Warning on how Cloudflare's AI bot blocking breaks Google search indexing
Find competitor search queries your site misses to guide new content.
Updated