Challenges of measuring PR’s influence on Generative AI – the giant Plinko machine

The PR industry is having an existential and serious debate about its future.  The rapid evolution in technology and Artificial Intelligence is changing not only how PR agencies and communications teams work but is asking questions about its very purpose.

The spread of digital and social media since 2010 has created a more complicated environment for PR and comms professionals to navigate.  Automated algorithmic feeds have led to audiences becoming more fragmented, isolated and polarised, creating reputational risks for organisations.

Stories of brands like Gillette, Bud Lite, Harley Davidson, Jaguar and Tesla alienating one section of their audience while attempting to appeal to the values of another have become familiar.

This challenging media landscape has been further complicated by the rise of generative AI chatbots like ChatGPT, Perplexity and Gemini which have become significant gatekeepers of information.

The logic that this is transformative for the PR industry goes:

1. People are increasingly going to Gen-AI chatbots to get their information.
2. ‘Earned media’ such as news websites are the primary source of content that feeds the training data for Gen-AI models.
3. PR’s core competence is in influencing earned media.
4. PR therefore has the opportunity to be the most strategically important marketing discipline.

An entire new industry of Generative Engine Optimisation (GEO) has been born with vendors claiming to have the magic formula of how to influence Gen-AI models.

PR Futurist Stuart Bruce and his organisation Purposeful Relations recently published an excellent and detailed white paper reviewing publicised research on GEO from different vendors.  The Impact of Generative Relations and Communications Report is available from the Purposeful Relations website and Stuart followed this up with an workshop for the PR Academy Student community (for students, a recording is available on the PR AcademyStudy Hub).  The white paper concludes:

“The theory that earned media drives AI answers is based on research published in numerous reports that claim to analyse AI answers to identify their sources. The problem is there isn’t an established methodology to do that so there are a huge number of challenges to ensuring accuracy and consistency…. Most of the reports have been published by companies selling tools that claim to monitor and measure AI answers.”

Discover the AMEC Diploma in Strategic Communication Planning and Measurement

The Gen-AI ‘Butterfly Effect’

The challenge is that measuring what affects Gen-AI outputs is hard.  Large Language Models are inherently unstable.

The outcome in a game of Plinko is difficult to predict because small changes in the ‘input’ of where you drop the marble leads to big changes in the ‘output’.

This is similar to the ‘butterfly effect’ in weather systems where a butterfly flapping its wings in Brazil can lead to a hurricane over the Atlantic.  And the variations in input can be large.

SimilarWeb data suggests that the average question to ChatGPT is 60 words long compared with just three for a Google search.  The number of degrees of freedom (or entropy) of longer natural language questions is many orders of magnitude larger than a short search.

The AI companies generally do not release data on how people prompt their chatbots, so most Gen-AI research is based on synthetic questions.  The models themselves are ‘black boxes’ that don’t let you see how they work.  There are big variations between models and individual models themselves change over time.  Some even change within a single prompt – the most popular model ChatGPT 5 will route to different underlying models depending on the question.  There will even be differences if the same prompt is asked in the same model by different people as many models include a memory function, altering the answers based on a personalised history of previous interactions.

Don’t let the facts get in the way of a good story!

Ironically, the same algorithmic rules around maximising engagement can incentivise comms thought leaders towards over confidence, hyperbolic claims and ‘click-bait’ headlines on blogs, LinkedIn posts and even industry ‘AI’ events.

Unfortunately, the real world is often not so simple.  As one of my former CFOs used to say, “don’t let the facts get in the way of a good story!”.

It is worth challenging some of the assumptions around the popular narrative.  So, for example, how fast is the transition towards consumers using Gen-AI chatbots to research information around products and services?

There is no doubt that there has been a significant rise in the number of people using Gen-AI since ChatGPT was launched three years ago.

According to SimilarWeb’s “2025 Generative AI Landscape: The State Of Gen AI” report, ChatGPT had grown to be the world’s 5^th most popular website by September 2025 with 5.9 billion monthly visits.

Google however remains the number 1 ranked website with 14 times more visits than ChatGPT and traffic has increased slightly year-on-year. More recent data released in November shows that the growth in traffic to ChatGPT has slowed to zero.

OpenAI boss Sam Altman reportedly issued a  ‘Code Red’ internal memo at the end of the month, warning that ChatGPT is in danger of falling behind the Gen-AI arms race against the likes of Google.

The SimilarWeb data shows that 95% of ChatGPT users also used Google, suggesting that users see these as separate tools.  The distinction between how consumers see Gen-AI and search is further demonstrated by the fact that 86% of users who started using Google’s AI mode stopped after three days.

Meanwhile OpenAI’s push to integrate Gen-AI and search with the launch of their own browser ChatGPT Atlas saw a 95% fall in visits to the download page after five days of the launch, indicating a lack of ongoing enthusiasm beyond initial early adopters.

So what about the assumption that consumers are using Gen-AI to research products and services?

Luckily, we do have real consumer data on this.  OpenAI released 2.6m real user questions on ChatGPT from June 2025 which was categorized and analyzed by researchers at Harvard and Duke Universities.  This showed the huge breadth of use-cases from creative writing and image generation to teaching and how-to advice.  Just 2% of prompts related to purchasable products and services with questions like “What’s the best streaming service?” or “Recommend a good laptop under $1000”.  This suggests that consumer research only forms a small part of how Gen-AI chatbots are used.

Search may well continue to be the primary tool to look for specific brands and products, with Gen-AI potentially being used earlier in the user journey for more general research.  Clearly this is a moving feast and there are clear incentives for the likes of Google to crack the user experience of using Gen-AI together with search to protect their market dominance.

From information fragmentation to integration

But surely we can be clear about earned media being the main source for Gen-AI training data?  Unfortunately given the volatility and opaqueness of the Gen-AI models, this is not straightforward to answer.

Research from companies in the PR and comms space give different degrees of magnitude – from 22% of source citations coming from earned media to over 90%!

There is even significant variability in the data from individual methodologies.  Semrush’s ‘2025 AI Visibility Index Study’ states that there is just a 32% overlap in the top 100 cited sources between ChatGPT and Google’s AI Mode.

Even within one platform, there are big differences in the top cited sources related to different sectors.  This will also change over time as models change their training – for example, there is some evidence that OpenAI is downgrading the prevalence of Reddit in its training data given concerns around content accuracy.

Given this volatility, a reasonable strategy for comms programs could be to broaden the out-reach to a wider range of different channels.  This is counter to the current thinking in a fragmented media landscape of targeting specific channels to influence specific audiences.

Before we get carried away and think that the ‘spray and pray’ of generating as much content as possible will come back into fashion – what is likely to count is the quality of that content.  Discipline around consistency of messaging is likely to lead to these messages coalescing and feeding through in Gen-AI answers.

And maybe that gives us some broader optimism on how information consumption could work in the future.

The algorithms behind social media serve to split information out into echo chambers and divide people.  The algorithms behind Gen-AI chatbots bring information from disparate sources together.

Gen-AI therefore has the potential to provide more balanced, integrated views that could broaden perspectives and break down barriers.  Yes, there will be important caveats around biases in both the user prompts and in the models themselves, but at least the overall architecture provides the opportunity for more truthful information, better balance and less division.

—

Coming soon: AI and tech for planning and evaluation

Look out for Paul’s next post early in 2026 which will focus on the different use-cases for AI and technology in planning and evaluation, which Paul also spoke to our AMEC Diploma students about.

About the AMEC Diploma in Strategic Communication Planning and Measurement

This postgraduate-level qualification is designed to enhance your strategic communication skills.

The course was developed by Distinguished Professor Jim Macnamara, PhD, FAMEC, FAMI, CPM, Chair of the AMEC Academic Advisory Group and an internationally recognised leader in evaluation of communication, in collaboration with AMEC and PR Academy.

Paul Hender is course leader so you are in excellent hands. Find out more about Paul below.

This article is based on a live session that Paul ran for the course.

Discover the AMEC Diploma

About Paul

With a background in physics, Paul Hender has spent three decades bringing science to the art of communications.

He has held leadership roles at several of the world’s most respected media intelligence companies, served on the board of the International Association for the Measurement and Evaluation of Communication (AMEC), and contributed to major industry initiatives including the Integrated Evaluation Framework, the Measurement Maturity Mapper, and the latest edition of the Barcelona Principles.

Paul is founding partner at CommsClarity Consulting. CommsClarity helps communicators and PR professionals use best-practice measurement, analysis and evaluation to be better informed and make the right data-driven decisions.