The Future of AI SEO – Futuristic Ideas of SEO & AI (James Dooley Interviews Dan Petrovic)

James Dooley: Hi. So today I’m joined with Dan Petrovic, who has always been someone I’ve admired within the industry. He’s always been two steps ahead of the game. Back in 2013, there’s a famous video now out there about how he said people would now be chatting with Google in 10 years’ time and how agentic search would be around. People couldn’t believe it back then and roll on 12 years. It’s where we are today. So the first topic I want to run through with you is, what do you see now for the future of AI SEO?

Dan Petrovic: Loaded question. The best kind of question because there’s a lot to unpack. Obviously the future of SEO has many additions. But we’ve been through this before with the introduction of various modalities, voice search, conversational search, mobile-first, and so on. We’ve been through all these adaptations. I see SEO as evolving and embracing AI as just another modality, another interface that sits between search and the end user. There’s going to be a lot more that an SEO needs to do. Some companies will start specialising and some consultants will start specialising. Some will say, “I’m an AI SEO”, just like some people say “I’m an app store optimisation guy”, “I’m mobile SEO”, “I’m local SEO”, and so on. AI SEO is an interest for me because it’s been an interest for the longest time, like you said, and we finally got to the point where the technology is accessible and widely adopted. All these technologies that were only available to very large corporations are available to us now. I remember Quill, you couldn’t get access to it unless you were a multi-million dollar corporation. Now it’s all there. So it’s like a dog off the leash. Everything is fun. Everything is useful. I came out of my retirement to get back into this and I’m enjoying it. I’m enjoying it because I see fantastic opportunities for new methods and new things to develop. One of them, and you’re probably asking for something actionable, is understanding model psychology. That sounds abstract or vague or hippie, but it isn’t. I’ve spent the last two or three years understanding the field of mechanistic interpretability. Mechanistic interpretability is understanding what models do, why they do it, and what happens in their neural network when they do that. Similar to human psychology, you get input through ears, eyes, tactile, smell. That multimodal input gets into your brain. Something happens. We don’t really know what happens. Then output comes out. We make a decision. Psychologists deal with a black box, which is the brain. In SEO, we’re going to be probing the models and understanding what they do and why they do it. Effectively we’re trying to understand the input and the output, and what happens in the middle. With open source models like Gemma, Llama, DeepSeek and others, we can get down to the circuitry of what they do and why they do it. I’m not suggesting every SEO needs to geek out to that level. My findings indicate that small models like Gemma, built on the same data and technology that Gemini has, have the same multi-dimensional geometry as Gemini. Basically semantic concepts cluster together. You create vector embeddings for one, vector embeddings for the other. They’re rotated differently and they still collapse on the same entities, the same things. Apple fruit, Apple technology. Then I realised there’s a lot of work for us to do in model probing, entity associations, and so on. On a practical level, the opportunity is understanding how different inputs to the model affect its outputs. This is RAG, retrieval augmented generation, which is basically just search. The future of AI SEO is dealing with the interpretative layer on top of search. That’s it. Models are too big and too expensive to train every day. They’re trained a couple of times a year. Models are released and then they rely on search for up to date information, for factual stuff, to prevent hallucination. There’s no way around that at this level of technology. So we have an opportunity to analyse that. There’s also something below that. The model’s primary bias against or towards brands and associations in their entities. There’s an associative element. When I say, “Dan, AI, what do you associate with this brand?”, the model lists things. That’s a smooth and efficient way of probing models. You’re mapping the associations the model has towards a brand. Then you flip it around. You say, “AI SEO, what brands do you associate with that?” and it starts listing brands. So you’re doing bidirectional probing, mapping that space of the model’s choices, which reflects pre-training, post-training, reinforcement learning, fine-tuning, and so on. This is the stuff a model would say even if it wasn’t grounded in search. You need to understand that because it influences choices once the model is furnished with grounding data. These are the most important things to understand. These top level concepts break down into specifics. We used to have clickthrough rate optimisation. You adjust titles, descriptions, schema and so on to make SERPs more clickable for humans. Now we have selection rate optimisation. There’s no clicks with machines. They select things. So we’re trying to make your brand, your search grounding snippet, more attractive for a machine to select. Understanding what works and what doesn’t is a big deal. Surveying people is difficult. People get tired or lie. Surveying models is pretty sweet.

James Dooley: I’ve never heard anyone talk about selection rate optimisation. I’ve seen a lot of people in SEO obsessed with CTR and conversion rate optimisation, but like you said, with LLMs there’s no clicks. There’s just selections they’re pulling from the data. You touched on the human brain. I’m going to try to explain it from a human standpoint. If you go to a restaurant and you get a plate of food and it’s presented nicely, you might like the taste, but you might not like the smell. It could be a specific fish. It doesn’t smell very nice but it might taste good. Or it could smell nice but not taste great. You spoke about input into the brain and then the black box decides whether you order it again or not. Sometimes the smell outweighs the taste, so you don’t have it again. What I’m trying to get to is there’s multiple signals coming along in RAG. It could be semantic content, third party corroboration, reviews, and so on. In your opinion, with the future of AI SEO, is there anything that outweighs everything else? Is it down to semantic content networks and how deep you go, entities and entity SEO, or is third party corroboration more important? Backlinks are important, but I’m not bothered about PageRank. I’m bothered about corroboration proving what you say. Is content becoming more important than ever? People seem obsessed with on-page or obsessed with links. What are your thoughts on that? Is there a Midas touch for ranking in LLMs, or is it just do everything as well as you can?

Dan Petrovic: The models are really quite simple. They get plain text input or plain multimodal input. There’s none of that sophistication we have in search pipelines. Search is way more complex and sophisticated than the current models that sit on top of search. Think of it this way. There’s no subtleties. The model will receive the plate of content and there will be no smell. It’ll perceive that plate through vector representations and have an initial perception. But the model doesn’t get the whole plate. It gets one slice. You supply a whole pie through your content, but the model only gets a slice. This happens because in a RAG pipeline they look at the query, then fan out queries that decompose the query into multiple aspects. They run multiple searches, collect search results, trim most of it, and supply the finals. All that stuff you mentioned, semantics, links, reviews, nuance, is just search. What gets brought back to the model is a trimmed set of results. For each search result, the model gets a sample. A taste. A slice. That is what it uses to generate an answer. That’s a huge problem because you might have a page that’s 2,000 words and the model will pick 400 or 500 words to represent your content, your brand, your message. It won’t do abstractive summarisation. It picks verbatim snippets. That’s extractive summarisation. It picks exact sentences word for word, chunks, stitches them together, and merges segments scored by a cross encoder in the background pipeline. That becomes a grounding snippet that represents your page relevant to the prompt. That, plus competing results, plus the prompt, gets pushed into the model as input. That’s it. No schema. No sophistication. Just plain text. Unless you attach imagery as multimodal input, which is different, but it still gets treated as vectors. The current biggest challenge is, will my content be represented well enough based on the small sample it sees? Most of the sophistication is still in traditional search.

James Dooley: That’s interesting. One last question on this. You mentioned query fan out. On the Dejan website, you can type a long query and it gives you 10 query fan out terms, improving how you’d search. If I search, “I’m looking for an accountant in Manchester that’s going to save me the most tax as I move to Dubai”, there’s multiple intents in there. Fan out might return international accountant, tax advisor, and so on. Is the aim now to score across all those fan outs? Is it an overall consensus score across all 10? Or do you only need a few? How does query fan out work with scoring across variations? Old school methodology was query string matching, BM25, put it in the title, H1, URL, and a few times in the content. Now with fan out, how does scoring work in RAG?

Dan Petrovic: It’s the top five. Top five. When you retrieve hundreds of results, the old school techniques still work because they’re cheap. Google still relies on the cheapest methods first. Semantic search kicks in at a reranking level. Let’s say you run seven fan out queries based on the prompt, addressing different aspects. That brings 700 results. Then there’s attrition filtering in the backend. Those results are collected and trimmed down to a narrow selection. Some will be from one query, some from another. There’s a layer you don’t control. As an SEO, you can’t really do anything except traditional SEO and ranking well for those queries. Tools like the query fan out tool help you think about closing the gap between the queries you rank for and the ones you could rank for. They give you ideas. I call these synthetic queries. I have two types of synthetic queries. One is, “Dejan AI, what non-branded queries would a user type into Google to find results from this brand?” I generate a variety of queries from the model’s head about what it thinks is relevant to the brand. Then I do a Venn diagram with Search Console. What’s in Search Console, what’s not, what overlaps, what’s the gap. Then I close the gap with content. Fan out queries are a form of synthetic query that result from the composition of the prompt. Some will match Search Console data. Some are opportunity queries. But if you do this a lot, you realise there’s an almost infinite amount of fan out queries. You can’t generate a million content pages. Google is good at semantic understanding. You need to bring it back to basics. Understand the primary entities you’re trying to cover. Understand what search queries stem from those. Group them. Create a graph of concepts and map them. Once you’ve mapped entities, synthetic and organic queries, PPC queries, you branch into prompts. But I draw the line at prompt tracking. People say, “I have 100 prompts and I track them every day to see how I rank.” Absolute nonsense. There’s no utility. I do it as one-off research, get insights, change prompts, and provide insights, but regular tracking is nonsense. There’s another aspect. When you do ongoing tracking, you want to probe relationships. I do it in two ways. One is associative probing. The other is relevance probing. If you saw Dan Petrovic in search, in the RAG pipeline, suggested as a result for a prompt, would you recommend Dan as a consultant for this problem, yes or no. Allow the model to output one token. Yes or no. Repeat 10 times or 100 times and come up with statistics. Sixty-eight out of 100 times it said yes. That’s how you embrace the probabilistic nature of models. Models have temperature and other parameters like top P, top K. They sample within probability buckets when producing outputs. You want to see how your brand, product, service fits within that.

James Dooley: You mentioned temperature and you mentioned six queries and then 6,000 results. Do they always look at the top 100? Or top 20? How deep do they go when they’re looking at search?

Dan Petrovic: The models don’t make those choices. Temperature and sampling depth is next word prediction. How much energy does the model have to sample. How wide and how deep. Anything to do with search results and what’s presented is Google’s software choices in backend processing. The model gets served information. To influence that, you influence traditional search. To influence the model’s choices, you probe, test, and survey models. Let me give a concrete example. I made a tool called Treewalker. You enter your brand or website and it says a sentence about you. It does two things. First, it’s not allowed to sample into search. It speaks its mind about what your brand does. Then, completely separate API call, it’s allowed to see search and your website, then answers again. In the third call, it self-evaluates. How did I do before seeing search and after? How close was I? That’s the first sentence. Then we allow the model to walk all the probability paths of all the things it could have said about you. We look for high entropy points. Moments where the model wasn’t confident. Like it says, “Dan Petrovic is an Australian”, and then it isn’t sure what to say next. That’s a high entropy token. From there, branching is more likely. When you have low entropy tokens, the model is confident about you and won’t flip to other things. That is the target for modern AI SEO. Find the low confidence spots where the model isn’t sure about you. Reinforce those with on-page content, outreach, and so on. You can’t just start digital PR or link building without understanding where the weak spots are in the model’s understanding of your brand and its associations.

James Dooley: So when you’re talking about link building and off-page, are you more bothered about confidence and clarity rather than link juice and PageRank? It’s going to be about who you are and what you do, getting the confidence score up, and using third party corroboration more than DR and traditional metrics.

Dan Petrovic: As far as model influence, the model is unaware of PageRank and authority. There are no authority signals flowing to it. It doesn’t get inputs other than text and multimodal input. The only source of authority the model has is its training. And that’s hard work. Some people think they’ll do a bit of digital PR and influence training data. No, you won’t. That’s monumental. It’s like moving a glacier. You need real heavy lifting in terms of discussions in forums, Reddit, and notability in its training data. That initial signal trumps everything else. When the model is presented with search, I really do mean top five results. From those top five, it gets bits and pieces from the page content. It has very little to go by. So I try to have my content as condensed as possible while still readable and friendly for humans. Get to the point. Give answers quickly. Avoid fluff. I trained a deep learning model I call a content substance classifier. You give it content A and B and it tells you which one is substantial and which one is fluffy.

James Dooley: Yeah. Words with no meaning. No substance. One of the biggest things we do when we’re acquiring sites is reducing the size of the website. We delete diluted categories. Delete pages that have no value. Then at heading level, paragraph level, sentence level, even word level, we remove fluff. We try to turn a 2,000-word article into 1,000 words while keeping the substance. People think word count needs to be 5,000 words, so they write 4,500 words of fluff.

Dan Petrovic: As a teenager, I learned a concept from Isaac Asimov in the Foundation series. There’s a scene where an envoy from the emperor visits a Foundation outpost and does diplomatic talk. They record the speech and a linguist analyses it with mathematics, like NLP. They cross everything out, distil everything, and conclude he said nothing. He was there for three days, he spoke a lot, and when you equalise everything, he didn’t say anything of substance. This is a problem with web content. We have both humans and machines who have attention patterns. Humans are drawn to visual cues like bullet points. Models are transformer models. They run on attention patterns. The paper that kicked off this revolution is called “Attention Is All You Need”. The page of content is seen by the model, but the model has different attention patterns. That’s worth investigating. A fascinating bit of research came out recently. Models pay attention to the beginning and the end and get lost in the middle. The same happens for humans. I wrote an old article, “Here’s why nobody reads your content”, because people don’t read, they scan. Content sells itself to the reader. If it sells, they read. Otherwise they scan, scroll, and read the bottom or the comments to get the TLDR. Now we’ve got AI between content and humans. Humans want the TLDR and the model delivers it, preventing time waste. If you have pointless content, both users and AI will trim it down. So what are you doing with that content? This is instructive for writing on the web. Journalists know how to write because they use the inverted pyramid. Main news at the top. Secondary details next. Tertiary details at the bottom. If you follow that principle, you’ll be good in the AI age.

James Dooley: Two last questions. Earlier you mentioned yes and no and you said 68 yes and 32 no. On the no responses, would you try to influence that if you wanted it to be yes? Would you do optimisation, new pages, and work out why those were no and push them to yes?

Dan Petrovic: Yes. Go after that. There are two paths. One is difficult and long, influencing training data. That takes time and effort, but you have to do it. If you don’t, you won’t catch up. It’s a slow burn. What you can do more immediately is make quick on-page changes and influence RAG. You can also do third party citations. Even if you’re not one of the cited URLs because the model didn’t include you, you can do outreach similar to link building and get into the results that are already cited. Here’s the trick. When you do citation mining, you probe the model for a variety of prompts. You should survey models, but not to track rankings over time. Things like prompt volumes are nonsense. There is no prompt volume. Let’s say you’ve done a thousand cycles of prompting and you build a map of the top domains consistently cited. Those choices are a combination of being in the search results and then being selected by the model. You have to be performing well in SEO to start with. Then you go deeper. Which URLs from those domains have the highest citation counts for your set of prompts, relevant to entities, fan out queries, organic queries, and paid queries. It’s all connected. What do you do with that information? Two options. One, create lookalike content that looks and feels like what’s constantly being cited. Two, find domains where those pages are accepted and make one on that domain. Pay money. Negotiate. Guest post. Digital PR. Partnerships. Whatever it takes. Get into the citations of top cited domains and piggyback your way into visibility while you work on the slower job of changing the model’s mind, which can take a year or two unless you have huge budgets.

James Dooley: One extra option is reaching out for a niche edit or link insert on the existing article that’s being cited. Get a snippet added into the article to include why you’re better. We do both. New posts similar to what gets cited, and updates to existing pages so they add the brand in. You mentioned Treewalker. It can show what it says without search, then with search, and show the difference. If it’s positive when grounded in search but not when ungrounded, how long does it take to get into training models? Is it a year, six months, or the next time they run training?

Dan Petrovic: I have one case study and we got a bit lucky. A German brand called AIO. They do custom sports jerseys, top quality sublimation printing. They have an online 3D configurator. You design your kit, place an order, and they ship worldwide. During probing, we noticed GPT and Gemini constantly excluded them from recommendations when the user was outside Europe and Germany. We tried lots of things and nothing worked. So we ran a campaign, like associative PR. We pushed “AIO USA”, “America”, and placed them alongside big brands like Nike, Adidas, Under Armour. We noticed low authority brands oscillate a lot in visibility tracking, but high authority brands are stable. Nike, Adidas and Under Armour sit at the top and don’t move. The bottom undulates. So we tried to attach AIO to that set of associations. It took six months. I almost gave up. In hindsight, nothing happens until Google releases a new model because you can’t influence a model already launched. It’s frozen in training data. This was Gemini 2. When Gemini 2.5 was released, and we switched off grounding with search, AIO started becoming one of the natively suggested results based on raw model memory. No search grounding. That was a big celebration moment. It took about six months, give or take. What worked fast was basic SEO. Within a week. We removed one set of signals and placed LLC, an American phone number and address because they have a local address in the US. A local SEO style change. When the model grounded with the page, Google gets full HTML, so it saw those signals. When we asked, “Would you recommend this page?”, it said yes. Grounded with search, within one week, we got them in. Ungrounded, natively selected, took around six months. To be safe, if you influence model behaviour within a year, you did well.

James Dooley: Anyone watching this, we hope you like the video. This was only meant to be a 10 to 15 minute video, but I love speaking with Dan Petrovic. He’s always two steps ahead of the game. Make certain you check the links in the description. There are a few videos. One on being a dad entrepreneur, raising kids in the AI era. One on optimising for LLMs. One on how AI is impacting link building. Dan, it’s been an absolute pleasure. Thank you very much.

Dan Petrovic: Thank you.