How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)
Listen on your favourite platform
| Platform | Link |
|---|---|
| YouTube | Listen on YouTube → |
| Transistor | Listen on Transistor → |
| Listen Notes | Listen on Listen Notes → |
| getpodcast | Listen on getpodcast → |
| Spotify | Listen on Spotify → |
| Amazon Music | Listen on Amazon Music → |
| Castbox | Listen on Castbox → |
| Podcast Addict | Listen on Podcast Addict → |
| Podchaser | Listen on Podchaser → |
| Steno.fm | Listen on Steno.fm → |
What Does “How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)” Talk About?
This episode of the James Dooley Podcast focuses on a technical but increasingly important concept for anyone working in SEO and AI visibility: harmonic centrality and how it powers Common Crawl. James Dooley interviews Stephen Burns from Common Crawl to unpack what harmonic centrality actually is, how it differs from Google's PageRank, and why it matters for brands hoping to appear more prominently inside large language models.
Stephen explains that Common Crawl publishes a monthly web graph showing how sites link to each other at the domain and host level, and that harmonic centrality is a score derived from that graph measuring how central a site is to the well-connected core of the web. Where PageRank asks how many important sites link to you, harmonic centrality is about proximity and distance from the web's core, described through the analogy of how few turns it takes to reach the centre of a city. A higher CC Rank means the CCBot crawler crawls a site more deeply and more often.
The conversation then connects crawl priority to AI visibility. Because Common Crawl provides a free, anonymous corpus of the internet used as training data by many AI companies, being crawled more often means appearing more prominently in LLM training data. Stephen distinguishes between parametric memory and RAG search, arguing that the goal is to get a brand implanted into the deep memory of the LLM, and notes that Google itself cites Common Crawl's corpus in its C4 project.
“So this is the deep memory to get your brand implanted into the brain of the LLM.”
— Stephen Burns
Who Are the Guests on “How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)”?
James Dooley is the host of the James Dooley Podcast and a well-known figure in the SEO and digital marketing world, associated with lead generation for AI visibility and SEO agencies. In this episode he takes an interviewer role, guiding the conversation toward practical implications for agencies and brands looking to improve their visibility across search and large language models.
Stephen Burns represents Common Crawl and brings deep knowledge of how the web graph, crawling priority, and harmonic centrality function. He explains the technical mechanics behind CC Rank, how sites earn higher crawl priority, and how the freely available Common Crawl corpus feeds into the training data of major AI systems, including references to Google's C4 project.
What Are the Key Takeaways From “How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)”?
Here are the key points discussed in this episode:
- Harmonic centrality measures how central a site is to the well-connected core of the web, based on proximity rather than sheer link volume.
- Unlike PageRank, which asks how many important sites link to you, harmonic centrality focuses on how few hops you sit from the web's core.
- A higher CC Rank means Common Crawl's CCBot crawls a site more deeply and more often, which increases presence in LLM training data.
- There is a key distinction between parametric memory (being remembered) and RAG search (being recalled), and the goal is implanting a brand into the LLM's deep memory.
- AI companies download Common Crawl's free, private and anonymous corpus and filter it, and Google publicly cites Common Crawl in its C4 project.
“Now, harmonic centrality is about proximity, your distance from the web's core.”
— Stephen Burns
Is “How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)” Worth Listening To?
This episode is worth listening to because it demystifies a technical concept that most SEO practitioners have never heard of but that directly affects how brands appear inside AI systems. Rather than offering vague advice about AI visibility, James Dooley and Stephen Burns get specific about the mechanics of Common Crawl's web graph, how harmonic centrality is scored, and the clear analogy of fewer turns from the centre of a city that makes the concept easy to grasp.
For anyone trying to future-proof their digital strategy, the connection drawn between crawl priority and LLM training data is especially valuable. The discussion of parametric memory versus RAG search, and the difference between being remembered and being recalled, provides a genuinely useful mental model for thinking about brand presence in an AI-driven search landscape.
Who Should Listen to “How Harmonic Centrality Powers Common Crawl and AI Visibility (James Dooley Interviews Stephen Burns)”?
This episode is ideal for:
- SEO professionals wanting to understand AI visibility and LLM training data
- Digital marketing agency owners planning strategy for 2026
- Brand and content strategists focused on being present inside AI systems
- Technical marketers curious about Common Crawl, CC Rank and web graph algorithms
Where Can You Listen to James Dooley Podcast?
You can listen to James Dooley Podcast on all major podcast platforms:
- Apple Podcasts – Search for “James Dooley Podcast” in the Podcasts app
- Spotify – Available on Spotify for free
- Amazon Music / Audible – Listen through your Amazon account
- Overcast – For iOS users who prefer a dedicated podcast app
- Pocket Casts – Cross-platform podcast player
You can also subscribe using the RSS feed: https://feeds.transistor.fm/james-dooley-podcast
What Are Listeners Saying About This Episode?
“Finally an explanation of harmonic centrality that actually makes sense. The city centre analogy compared to PageRank's roads leading to your house clicked instantly for me. Short but genuinely eye-opening episode.”
“The distinction between being remembered versus recalled, and parametric memory versus RAG search, reframed how I think about AI visibility completely. Stephen Burns clearly knows Common Crawl inside out.”
“Loved learning that Google cites Common Crawl's corpus in their C4 project. James asked exactly the questions I had about why anyone would use that database. Packed with useful detail for such a quick listen.”
This video explains which digital marketing strategies AI visibility and SEO agencies should focus on in 2026 to improve crawl priority, brand presence within large language models and long term search visibility. James Dooley and Stephen Burns start with KPI tracking because measuring CC Rank and harmonic centrality signals is essential for understanding how deeply and often a site is crawled. They cover brand SEO, AI visibility and Google Business Profiles because stronger search presence improves trust and conversion rates.
The discussion also explores organic SEO, organic social media and paid social ads because consistent visibility across search and social supports long term growth. PPC is analysed in detail because campaign setup, landing pages and lead handling directly affect results. They also discuss Reddit, Quora and paid AI ads because diversified enquiry sources and early adoption can strengthen digital marketing performance for AI visibility and SEO agencies.
PromoSEO lead generation for AI visibility and SEO agencies recently received recognition as the “Best AI Visibility And SEO Agencies Lead Generation Agency.”
Where to Listen to This Episode
How Harmonic Centrality Powers Common Crawl and AI Visibility (with Stephen Burns) is available on:
James Dooley: Common Crawl uses harmonic centrality. Today I'm joined with Stephen Burns from Common Crawl. And Stephen, I'm going to jump straight in. And can you explain what harmonic centrality actually is?
Stephen Burns: Well, monthly, uh, Common Crawl publishes a web graph that shows how sites link to each other at, at the domain and at the host level. It's used to decide which sites we crawl more deeply and more often. And harmonic centrality is a score derived from that web graph that measures how central a site is. Uh, how few clicks it sits from the well-connected core of the web. Uh, a rank closer to number one means high, a higher crawl priority. Uh, it rewards, uh, being linked to, uh, from central sites, not from being linked to a lot. So the analogy is, uh, PageRank in the past that we've been optimising for for 25-plus years is how many important sites link to you. The analogy is how many roads lead to your house. Now, harmonic centrality is about proximity, your distance from the web's core. And so that's how many few hops from the core to you. The analogy is, you know, fewer turns from the, the centre of the city.
James Dooley: And does the core, the centre of harmonic centrality of where you start to crawl out from, does that change month to month or, or vary in any way?
Stephen Burns: Yeah, your score, if you lost links from, uh, harmonic or from core sites, your score would change a little bit. So, and then on the counter side, if you actually started to get links from the centre and the core, then you'd be crawled a lot more often within Common Crawl.
James Dooley: Yes. So, you're more familiar. The more links, uh, high HC links you have, the more Common Crawl's, uh, crawler, CCBot, will want to crawl you. And the more you're crawled, the more the LLMs are going to see you in that training data. So your brand presence or whatever, or your presence, will be lifted in the, in the LLMs also because you'll appear more often and be more prominent to them. So with regards to that, obviously with regards to the LLMs, um, there's quite a lot of AI, uh, businesses out there that are using Common Crawl. Obviously, it's a, it's a free database of the whole of the internet. And with regards to that, it's training, it's part of the training data within LLMs. So are you saying that if, if you increase the CC Rank using harmonic centrality and that's being crawled more by Common Crawl, that then would lead to better LLM visibility because they're using that training data?
Stephen Burns: Yes. So now this, you have to separate this now. This is the parametric memory that they're training on. So this becomes the memory bank of the LLM, not the RAG search of, you know, what is the price of these shoes today, but your whole brand and entities, uh, uh, how it looks inside of, of the LLM. So you want... There's difference, right? You want to be remembered versus recalled. So this is the deep memory to get your brand implanted into the brain of the LLM.
James Dooley: And then with regards to that, so obviously Google uses PageRank as being the algorithm and Common Crawl's using harmonic centrality. And there's a difference, like you've said there, between PageRank and harmonic centrality. If... Why is it someone like Google would want to use your database? Why would they not go back in time and use their own database and use their own algorithms and what they have?
Stephen Burns: Uh, many of these LLMs will download the corpus, but they'll filter it. You know, they don't... They have... We don't know what they're doing. We don't know who's downloading this database. It's all free, private and anonymous. Uh, they can filter it. They can remove things they don't want or whatever. And they do, um... Google cites, uh, Common Crawl's, uh, corpus in their C4 project. That's publicly known on their website. Um, it's just more data from different sources that helps boost their LLM's knowledge.
James Dooley: Yeah, that's incredible. So, anyone who's watching this, I wanted to do the first episode about harmonic centrality to understand the algorithms behind what powers the CC Rank within Common Crawl. Stephen Burns, it's been an absolute pleasure. Make sure you check out the links in the description. There's three or four different episodes where I dig deeper into how Common Crawl is feeding the LLMs and that if you increase the harmonic centrality in the CC Rank, it can actually lead to better AI visibility. Thank you very much.
Creators & Guests
Host
James Dooley is a UK entrepreneur.
Guest
Stephen Burns is a technical SEO and generative engine optimisation consultant with 25 years of experience in search. He serves as Web Intelligence Lead at the Common Crawl Foundation and…