# Free Backlink Data via Common Crawl's Web Graph

> Extract domain backlink data from Common Crawl datasets using Bash

This tip and script show how to query Common Crawl's free web graph datasets to extract backlinks for any target domain. It provides an alternative to expensive SEO subscriptions by letting you calculate top linking domains and host counts locally using simple command-line tools.

## How it works

TIL you can pull the backlinks to any domain for free using Common Crawl's web graph instead of a paid service. A tiny bash script does it: https://gist.github.com/retlehs/cf0ac6c74476e766fba2f14076fff501. Example output (linking\_domain / num\_hosts): github.io 284009, substack.com 199844, cloudfront.net 89247, godaddy.com 60673, google.com 41269, ddns.net 7170, github.com 6038, run.app 4187, auth0.com 2397, adobe.com 2260, uservoice.com 2080, cmu.edu 1923, js.org 1403, wordpress.org 1271, wikimedia.org 1094, mozilla.org 960, heroku.com 886, hubspot.com 574, uberspace.de 573, userecho.com 548, hosteurope.de 424, pressidium.com 296, pantheon.io 278, dreamhost.com 269, seravo.com 266, ithome.com.tw 159, kinsta.com 151. Longer write-up: https://benword.com/the-free-data-sitting-in-common-crawl — backlink tools, domain authority scores, historical web archives are all sold as subscriptions but derivable from Common Crawl's free quarterly release.

## What it is for

- Extract top referring domains for a competitor without paying for SEO tools
- Audit incoming backlink volume and host counts for your own site
- Process quarterly Common Crawl releases to track changes in web graph connectivity

## Key points

- Bash script to query and parse public Common Crawl web graph releases
- Outputs referring domains along with associated link and host counts
- Avoids reliance on costly recurring SEO tool subscriptions
- Leverages free, openly distributed quarterly web crawl datasets

## Pricing

Free

## Details

- Type: Tip
- Category: [Marketing & Sales](https://xiry.app/topics/marketing-sales.md)
- Topic: [Search Engine Optimization](https://xiry.app/topics/marketing-sales/search-engine-optimization.md)
- Open source: yes
- Platforms: Command line, Linux, macOS
- For: SEO professionals, data engineers, and web developers
- Alternative to: Ahrefs, Semrush, Moz
- Tags: seo, backlinks, common crawl, bash, web scraping, data analysis

## Why it was saved

- [@retlehs, 17 April 2026](https://x.com/retlehs/status/2045169132748877992): Replaces backlink tools that charge hundreds a month; shows how to derive backlinks (top linking domains and host counts) for any domain from Common Crawl's free quarterly release.

## Compare

- [Alternatives to Semrush](https://xiry.app/alternatives/semrush.md)
- [Alternatives to Ahrefs](https://xiry.app/alternatives/ahrefs.md)

## Related

- [10 AI Search Optimization Findings from 1 Billion Data Points](https://xiry.app/resources/10-ai-search-optimization-findings-from-1-billion-data-point): Empirical SEO findings on how AI models and Google AI Overviews cite content (List, Free)
- [18 About Us Page Elements for E-E-A-T](https://xiry.app/resources/18-about-us-page-elements-for-e-e-a-t): Checklist of 18 trust signals to improve website About pages for SEO (Tip, Free)
- [Above-the-Fold H1 Heading Optimization for Homepage SEO](https://xiry.app/resources/above-the-fold-h1-heading-optimization-for-homepage-seo): On-page SEO practice for targeting primary keywords in homepage headings (Tip, Free)
- [Claude Code for SEO Playbook](https://xiry.app/resources/claude-code-for-seo-playbook): Turn Claude Code into an automated end-to-end SEO pipeline using external APIs (Tip, Free)
- [Cloudflare 'Block AI bots' SEO Troubleshooting Tip](https://xiry.app/resources/cloudflare-block-ai-bots-seo-troubleshooting-tip): Warning on how Cloudflare's AI bot blocking breaks Google search indexing (Tip, Free)
- [Competitor Keyword Gap Analysis Workflow](https://xiry.app/resources/competitor-keyword-gap-analysis-workflow): Find competitor search queries your site misses to guide new content. (Tip, Freemium)

Updated 11 October 2026. Page: https://xiry.app/resources/free-backlink-data-via-common-crawl-s-web-graph

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.