What Happens When You Probe LLMs? | James Dooley ft Malte Landwehr, Sanjay Singh & Jabez Reuben

James Dooley: Probing large language models. Should we try to ask the AI why it gave that particular answer? Should we investigate AI, trying to figure out the logic, sources of justification, and everything related to the recommendations received? Is he going to hallucinate? Will he give you false, incorrect information? Or will it help you begin to understand why the big language models come back with these recommendations? Jabez, you and I participated in a podcast with Dan Petrovic. It was a revelation. How many times has he done Anthropic probing, Claude. How many times did he ask the same questions to ChatGPT, test the AI, and try to get information about why they came to such answers. I don't know if you want to expand on that a little bit, Jabez, and then we can discuss with Malte and Jay how to better understand the LLM probing results.

Jabez: Yes, it was indeed a revelation, because the concept that Dan Petrovic shared with us is to ask hundreds of questions to different LLMs to get an idea of what they think about your brand or how they perceive it. It could also be hallucinations, as that is what we want to fix. This may be a completely negative attitude that doesn't actually exist, but LLMs picked up on incorrect information somewhere that was mistakenly published or simply made up. And the whole idea was, for example, if you run a brand of ergonomic chairs. So the idea is to ask it many times from different angles. What do you think about brand X, which makes ergonomic chairs? Why is this brand inferior to brand Y? What should we pay attention to before buying ergonomic chairs from brand X, etc.? Just going into as much detail as possible, because you're trying to get into the "head" of the LLM as a negotiator. You are not trying to force the model to respond in a way that agrees with you. It would be a mistake to ask, "Do you think brand X is great?" "Should I buy it?" Because to such questions, he will simply try to say: "Yes, I think it's good," and you can safely buy something in that spirit. Instead, you try to ask questions like a smart negotiator to get answers from the inside, to understand the direction of these LLMs' thoughts or what shapes their response. So, this was the main idea of testing LLM: to understand how they perceive the brand, what positive or negative sentiments exist, and what attributes are attached to it. And speaking of attributes, we'll get to that later when we discuss the attributes added to fanout queries these days. But what do you guys think about this LLM testing in general now? Have you tested this yet? I know there's this fear of hallucinations, but what are your thoughts?

Malte Landwehr: Yes, I think there are different angles here, and the process you just described makes absolute sense to me. For example, Dixon Jones from the What AI Knows About You project has been talking about this same approach for what seems to be the last two years. And at PKI, we also do this not for visibility tracking, but for brand perception. We just keep asking LLM: "name the attributes of a brand," and then we ask: "now name the best brands in the industry for each of these attributes." And then we can see which brands in the LLM industry are considered trustworthy, reliable, or perhaps have poor or good customer support, etc. So I think it works really well if you want to understand what biases, assumptions and presuppositions the LLM has about you. The problem is that for a perfect result, you need to do this twice: once without grounding (learning on data) and once with it, to understand what is a bias or perception embedded in the underlying model and what actually comes through the grounding process. But for regular monitoring, I think that's not the best way to go, because the things that come from the baseline model aren't going to change next week, they're not even going to change next month. But with grounding, of course, there's a chance to change that, right? If you work with sources and publish content, you can do this. So I believe this aspect works well. And there's a second aspect of testing when you ask about sources: for example, why did you choose this particular source? Is this the best source? Why didn't you recommend this source? And the problem here is that if you look at the real pipeline, the LLM that you're talking to might not know this, because there might be another LLM, a very small one, that just decides, "Okay, out of 20 sources, which three do we give to the main LLM that answers the question?" Or "What, um, paragraphs of content, what specific pieces of text are we going to extract?" Um, or for example, ChatGPT has brands that have partnerships with news resources, right? And um, they actually, ChatGPT doesn't even get all the content, just a static summary for each article. So at the time this resume was created, you may or may not have been in it, regardless of whether you were mentioned in the article. So there are many, many steps that the real model that you're talking to doesn't know about because it has a tool call, and that call says "web grounding." Perhaps it generates fan-out queries itself. Maybe there's another tiny model that just takes the initial query, looks at the user context, and quickly creates queries to branch out to, right? And then the model you're talking to doesn't know that. So she just uses the tool and gets the answer. So I still think it can provide interesting insights to understand, for example, why did you recommend this source? Why didn't you recommend this brand? Why is this brand in third position and this one in first? But you could achieve similar results by repeatedly asking 100 experts, "Why do you think this happened?", right? LLM doesn't necessarily have a real understanding of its underlying systems. It's a bit like when ChatGPT came out, people tried to ask what GPU the inference was running on to prove they were using an old NVIDIA chip or that DeepSeek had access to the latest NVIDIA chips, but that was just a hallucination, LLM can't know that. So I'm skeptical of people getting too deep into this, but I believe it can yield a lot of great insights that can then be used for monitoring prompts, fact-checking, and as inspiration for content that you create to highlight certain strengths of yours, or to correct misconceptions about your brand, and to call attention to weaknesses of your brand.

James Dooley: Yeah, I think, on that, I think, Jay, you can obviously add a little bit too. To better understand Jay John's point: he literally believes that more data, big data, is the key. So he said, do this for every major language model, multiple times a day, and for all the different types, to understand why I should trust this? Why shouldn't I trust this? He took everything one by one and said that one should never, never, never accept one query and one answer as the ultimate truth. It's about the amount of data for various large language models, for everything that needs to be done. Collect all this information, and when several language models start pointing out your shortcomings because a competitor has something better, then you can start working on it. But he was a firm believer in the importance of big data: do it many times with different queries, across different systems with positive and negative tone, and yes, do it all the time. I just wanted to add to this, but Jay, what are your thoughts on testing large language models?

Jay: I'll put it this way: testing is great, but again, as you said, you can't rely on just one answer. I mean, sometimes, let's say we have one query, we get 100 responses to it, and today we're mentioned in 36% of the responses, and tomorrow it's 71%. So the answer is constantly changing. They are very probabilistic in their approach, these language models. As Malte said, if you ask them what GPU they're running on, the model has to answer because it thinks it's the smartest in the world. So they will answer randomly. Even if they don't know, they will respond because they act very probabilistically. And then, yes, I would say, it should work. For example, you ask which XYZ software is the best, and then ask a follow-up question: why did you choose this particular source? Why did you choose this? But again, one answer will not be enough. I think for one particular query, you usually have to repeat it thousands of times and then build a plan and a strategy based on that, for example, if the model says, well, I mean, your software has been called less reliable, or let's say it says your customer support is really bad. This is the answer we get, say, after 5,000 requests, this is the conclusion we have after probing. Then you have to make some kind of decision and, you know, dig into what's really causing that mood. Is this internal memory? Is this pre-training data? Or maybe these are external sources? Then you'll have to figure it out. If it's third-party data, say social media, if it's a third party, you'll have to investigate that. If it comes from a previous training, then you have to, you know, do all the necessary things again, work with the press, social media, do a lot of things to get into the next training, and then you have to change that mindset. But yes, to your answer, yes, you would have to do it on a very large scale to learn about your brand and then change perceptions.

James Dooley: Um, and Malte, J, and even Jay leave. Remember how we recently ran basic tests on pulling branched queries, and it was so interesting to see what kind of queries Chat GPT was issuing for SEO experts. And we know that this is partly true for some SEO experts. The very first branched query is disputes. We know this guy has his share of controversy. Then for James Dooley, the branching queries included recent interviews. We know he's been on a lot of podcasts lately. And Brian Fishkin had a query "zero click marketing", and it was his book that was recently released. What kind of How, how does Chat GPT produce such novel branching queries? Because before, it was just "the best" or "top 10 best SEO experts" and very basic stuff, right? How do they manage to fulfill these complex requests that are so personalized, custom, and very specific to each person? Where does this come from, and does it give us an idea of where things are headed, or could it be part of our research?

Malte Landwehr: So, judging from what we see, ChatGPT does not run all its fan requests at the same time, but executes them one by one. So any content retrieved from the first fan query can already give some idea of what they include. Plus, they now have their own index, so they could very well build a knowledge graph where they just know that Rand Fishkin is associated with "zero clicks." So they just add it to the fan request. So I think the combination of these things—content for the first query and whatever they can get from their index—is going to be the driving force behind this process. And I expect this to continue: they will have more and more information in their own index and use it to get extremely fresh data. This is very personalized for each entity, which, of course, can have both positive and negative consequences, as there are brands for which ChatGPT adds the word "scandal" to fan queries in 25% of cases, and, of course, this highlights scandals associated with that brand. So this can be both good and bad. Another interesting thing I noticed during testing: when I searched for a name, a clear knowledge panel immediately appeared on Google without any questions or confusion with other experts with the same name. ChatGPT immediately started responding with proper fan queries, i.e. detailed queries. So I didn't even have to clarify: "James Julie, SEO expert." In that case, I simply said, "James Julie." But what if there were names that didn't immediately trigger Google's knowledge panels, and Google asked, "What exactly is David Quaid?" "Is it David Quaid this or David Quaid that?" In this case, ChatGPT also does not send a detailed fan request and does not provide an answer, but first asks who exactly it is about. But when I searched for David G Quaid, and there is no other person with the same query, it gives the corresponding fan queries: "Okay, SEO expert, his channels," and then the answer itself.

Jay: Have you found any correlation in the sense that people with clear knowledge panels that are not confused with other entities are more likely to initiate responses to fan queries in ChatGPT, or, say, does ChatGPT rely heavily on entities that have knowledge panels in Google? I believe that knowledge panels are purely Google's own data, and I haven't seen anything similar that correlates with ChatGPT search queries and their results that are related to knowledge panels. But to answer your previous question, yes, they like freshness. And as Malte said, first of all, if you search for someone's name, they first look for information about that person, and if something comes up in the first result, they just dig into it and do additional searches based on the first one. It just unfolds further, and that's how they find the information. Regarding the knowledge panel, I didn't find a connection, but in AI mode, apparently in AI reviews, there is some connection to the knowledge panel. Malte, do you have any thoughts on this?

Malte Landwehr: I mean, there's no specific data, but perhaps the same data that allows Google to have a knowledge panel prompts ChatGPT to have some statements in its own knowledge graph with similar information about this person. Of course, it could also be that they are scraping the knowledge graph, like many other things, so why not do it here as well. But I don't... it's very difficult to test, because correlation and causation are sometimes difficult to separate. Here I would say that it is probably definitely a correlation, but I am not sure that it is a cause and effect relationship. But looking at where search is going in general, everything you've done in the past to get a record in the knowledge graph, everything you've done to optimize it, everything related to connecting your profiles, having a master source of entity data, making sure that systems understand what entity you are, I believe that this is already and will be very, very useful for AI search in the future.

James Dooley: You both have your own tools, and I assume both tools have a content planning feature, and I believe that probing can be part of a content plan. What advice would you give our listeners on proper probing, what steps should they take to understand, perform the probing, and then plan their content accordingly?

Jay: Yes, I would say that probing, as you said, is very similar to tracking prompts, so you would have to track prompts as well, right? Just like you track prompts, you'll have to track responses, and then again, for content planning, I would advise looking at what's ranking in Google and what's being cited the most. So, just combine these two approaches, extract all the NLP terms, all the semantics from the pages that are cited the most and the pages that are in the search results, the ones that rank the best, and then combine them, extract all the terms and optimize them as much as possible in terms of semantics, NLP, and also try to understand what type of page it is. Let's say we pull up four search results. We also extract the top four citations and learn that the page type was in list format. So, we learn that the type of page here is about 60% of the content around these two is just "how to" or service pages. So let's just create a highly search-optimized service page that answers this specific query, and within that query you can add all the related subqueries about your brand. If something is negative, try to improve that as well and try to cover as many subqueries as possible within that content. This way, it will be optimized as much as possible for everything: NLP, content type, and subqueries. That would be my recommendation.

Malte Landwehr: I will say that too. Yes, I have something else you can do additionally, because in the end, it's not just about what attributes are associated with your brand. What is also important is what attributes LLMs care about the most, and you can actually test this with this probing mechanism. So, let's say you want to become the number one podcasting software. You are using a basic LLM without grounding and saying there are three companies. Khabus Podcasting has the best customer support. James Podcasting has the most reviews, and there is also James J Podcasting, and they meet the SOC 2 standard. Which one should you choose? And then you do it over and over again, changing names and attributes, adding other attributes, and if you do it hundreds of times, you eventually learn: yeah, for podcasting software, let's say, connectivity, reliability, and integrations—these are the attributes that Gemini rewards the most. And ChatGPT rewards good feedback and communication the most. And no one rewards SOC 2 compliance. Then you will know that whatever attributes are true for your brand, those are the ones you need to focus on. And we've already done this in several industries. And it's really very interesting that every LLM has some biases about which attribute is the most important. For example, when you ask LLM about the best enterprise CRM, SEO tool, or podcasting software, suddenly SOC 2 becomes extremely important, SOC 2 compliance. But if you ask about similar software solutions or services for agencies or startups, then SOC 2 becomes irrelevant. So, if you have a rough idea of what query you want to target, such as a general categorical query, you can really understand what attributes actually influence the LLM decision. So this is perhaps another clever way to find out not only what attributes are associated with you and your competitors, but also what attributes LLMs value the most. The bad news is that not all LLMs encourage the same attributes. And when a new Gemini or ChatGPT model comes out, you have to repeat this analysis because the results are not stable after model updates. But there are often certain attributes that they value. If anyone has clients in the automotive industry, I can say that safety in all LLMs is very, very overrated compared to other aspects like low consumption, energy efficiency, speed, design, etc. Unless you provide other context regarding the car, LLMs are looking for safety.

James Dooley: Malte, you mentioned this before and also in this answer. You mentioned that you need to run queries without grounding. What is the difference between executing requests with and without justification? Can you please explain this for those who don't have much idea about it?

Malte Landwehr: Yes, if you run them without justification, it effectively means that LLM cannot retrieve any relevant data from websites. No RAG, no search justification, no internet search. She has to respond only using the base model. So any answer you get is based on the biases of the underlying model or the knowledge embedded in the model's weights. And if you activate the justification, that's how most of us typically use LLM and how most consumers use it. You enter a query, and if it requires a search, the model performs the search. For most commercially relevant queries, it will perform a search. It will find websites and then form an answer based on what it finds on those resources. And for the most part, I would also look at the data with justification because that's the relevant thing, that's what people see. This gives you actionable insights into which sources work, which websites, formats, or topics are effective. But if you want to deeply understand LLM to develop a strategy for next quarter, it makes sense to run some ungrounded queries to understand the model's biases or how it perceives your brand and competitors.

James Dooley: And how do people turn off grounding? I mean, I'm using the API and not activating grounding as a tool.

Malte Landwehr: Previously, in many LLMs, at least in many consumer models, this could be deactivated in the frontend. I have to admit, I haven't tried to disable web search in ChatGPT or Gemini in months. Um, but API is usually the best option in such cases. The API also has the advantage that you can choose a specific model because you want to test it on the latest version. And if you're using ChatGPT now, you also have pretty good control over the model in the interface, but there were times in the past when ChatGPT would sometimes choose the model for you. So if you entered a very simple question, you could get an answer from the GPT-4 model, even if GPT-5 had already answered most of your questions. Or if you use AI mode and AI Overview, the version of Gemini you get sometimes depends on the country you are in, whether you are logged in, and whether you have a paid account. And if you use the API, you can very precisely tune it to the specific model you want to explore.

James Dooley: Regarding your question: remember earlier we talked about how you type in a name and it gives you different branch options, and you said that sometimes they hold the first branch, which leads to the second, which leads to the third? In your opinion, do you think that's why the set of queries for one person is different from another, or do you think that's part of what you were just talking about—the baseline data for grounding learning? Do you think they just do an internal search for information about this person and form query branches based on that, or is it a sequential process where they do a first branch that leads to a second, then a third, then a fourth, and it's the first one that determines different branches for each person?

Malte Landwehr: I think it's both. So, if it's a famous person like Ren Fishkin, then LLM already knows to add the word SEO. While SEO may be in his past, it's still what he's known for, right? Um, and if it's a lesser-known person or someone with an ambiguous name because there are a lot of people with the same name, then it's more likely that the first fanout query will identify it. And then you'll probably also see the first fanout query, such as "Malte Landwehr biography", "Malte Landwehr resume", or "Malte Landwehr interview". I mean, in my case, I think ChatGPT also knows that I'm involved with SEO and some other topics. Um, but yes, it's a combination of both factors. If they have knowledge, I don't know if it's their knowledge graph or something they quickly pull from the index, they'll already be using it. But if they are unsure about the substance, then only the results of the grounding process for the first fanout request can influence subsequent ones.

Jay: I have something to add to the previous point. You know, Malte said you could just turn off grounding. You can also adjust the temperature, i.e. how probabilistic you want the model to be in your approach. So you can do that too. And since Malte said you can turn off reasoning, I would advise a lot of people to watch the queries in real time, because the model is the users, how they use it. They just see something, ChatGPT selects the model they want on their behalf, and then performs a web search. So you should also consider this metric: how many times, let's say, for this particular query, I ran it 1,000 times, how many times was the web search performed? I had a client who was doing optimization. He works in the healthcare sector. He optimized the content, which was very informative. And I noticed that web search is used very rarely for informational queries. So for someone who wants to optimize information queries, it will be very difficult, because they are actually taken from the justification data. So you, because if they don't do web search, they don't read third-party sources, and you can't do any GE optimization because they just don't do any web search. So, this is one of the points: you should pay attention to the percentage of search queries on the web; if it's less than 30% or 20%, just don't do it. Just don't do optimization. I mean, just don't waste your money on it. That was my thought.

James Dooley: So, you both suggest using the API also to run a lot of checks and tests to get different answer options. Not for daily monitoring, but if you want to get an initial set of data to form a strategy or hypothesis, yes, that's one case where an API might be appropriate. And continuing the theme of researching large language models, we've talked a lot about different attributes here. Of course, there are many different attributes that you can start to cover. You have brand attributes. This is everything about the company and so on. You have service or product attributes that relate specifically to the products and services you want to sell. And then you have the attributes of trust. I consider these to be reviews, recommendations, awards, cases, and the like. How important are they, or which one would you choose if you had to choose one, and I know the obvious answer is, "it depends on the situation." But if someone came in and said, "We want to improve our visibility in AI," which attribute would you look at first: brand attributes, trust attributes, or service attributes? Where would you start? I'll start with you, Jah.

Jay: Yes. I would say that I strongly recommend tracking fan-outs (query branching) because they provide everything you need. If it's about security, they will search multiple times, for example, "security of XYZ service." Is this legal? Do you know what the risks are? What instructions are there, etc.? So the models, particularly ChatGPT, provide a lot of data in the branches, making it clear that security is a concern for ChatGPT when recommending this particular service. So let's create an attribute that answers this very question. This applies to the service. So, this falls under the service attribute. And with branding—if there is a search for your brand or other brands, comparisons, or alternatives, it clearly indicates that the brand attribute is important for that particular query. And again, the same thing: the same prompt, multiple runs, we see a certain set of results appearing many times, and based on that, we can conclude that, well, these are exactly the attributes that the model is focusing on when forming the answer. So let's just focus on creating content and making an impact with it. So, that's my answer.

Malte Landwehr: Do you agree, Malte? Hmm, nothing to add. I would take a similar approach. That sounds good.

James Dooley: So for everyone listening to this, I want each of you to summarize what was said. You have less than 30 seconds to summarize. I'll start with you, Jabez. You have less than 30 seconds to answer. Do you think LLM probing is a good strategy to try to get information that will help you in your work and that can be used as part of your content strategy? And I'm going to ask each of you to give an answer and explain why in less than 30 seconds.

Jabez: I will say 100%, we have already started implementing this in our campaigns. Probing should be done on a large scale and should be considered as a rough guide. It should not be considered a ranking factor. Be skeptical about it, understand where these LLMs are going, and it will give you a lot of ideas, but of course it has to be done based on big data to see a general consensus.

James Dooley: What do you say, Jay?

Jay: Agree with Jabez, you can't rely on just one data point or one answer, just do it multiple times. I would say that regular tracking of leads should be enough, you will get a lot of data about your brand as well as your competitors. I think that will be enough, but if you want to delve into these things and get more information, then I think probing is for you.

James Dooley: What do you say, Malte?

Malte Landwehr: For 80% of companies, I consider LLM probing to be a distraction and not the most important thing they should be doing right now. For the 20% who want the most advanced strategy with the most data, this is a very important key element of their strategy.

James Dooley: Yes, just expanding on Malte's point. I think many people don't even do the basic basics, let alone start probing to find the additional attributes of what is needed. If you fall into the 20% that Malte mentioned, and you are already quite advanced in AEO or GEO, seeking to build brand attributes, trust, and product or service characteristics, then probing can yield results that go beyond simply focusing on finance for inquiries. However, the quickest wins come from working with the basic financial requests you need to complete. So in this scenario where we were talking about safety in relation to cars and so on, you have to make sure that's the first thing they highlight, because that's the most important attribute that big language models give off. It was extremely interesting to see if we should be probing large language patterns. In my opinion, if you're going to do it, make sure you do it through ChatGPT, Claude, Perplexity, Grok AI—cover everything, use all the tools to get a consensus on what you should cover. This may also start to reveal certain feedback and controversial points that you will need to address, including from an online reputation management perspective. Malte, Jay, Jabez, thank you so much, and I hope you enjoyed this episode about LLM probing for information to increase visibility in AI.