The answer is yes, now what is the question? How automation and AI are changing media intelligence

Douglas Adams was very prescient about technology.  In his Hitchhiker’s Guide to the Galaxy – originally a 1979 radio play before becoming a book, a TV series and a film – there are several uncanny predictions. From Marvin the ‘Paranoid Android’, a robot with a ‘genuine people personality’ synthetic persona to Deep Thought, a super-smart computer that can answer any question in natural language.

When Deep Thought first comes online, the operators ask the computer for the answer to “the ultimate question of life the universe and everything”. After ‘thinking’ about this for seven and a half million years, Deep Thought answers: “42”. Sensing that this is getting an underwhelming response, it follows up with: “had you thought about what the actual question was?”

Gartner’s most recent Hype Cycle for Artificial Intelligence has ‘AI Agents’ at the top of its ‘Peak of Inflated Expectations’ while ‘Generative AI’ is plunging into the ‘Trough of Disillusionment’. Gartner’s Hype Cycle model argues that it can take years before the oscillation between optimism and pessimism settles into a ‘Plateau of Productivity’.

When requirements are not specific they can undermine the effectiveness of professional services, even beyond the underlying tech itself. In a previous role at a media intelligence business, we inherited a sales positioning of: ‘The Answer is Yes, Now What is the Question?’.  Not surprisingly, this positioning led to both clients and internal staff being confused about what services we were supposed to provide.  I would argue that it is the responsibility of the professional service firm to define by asking the right questions and then work out the process that best addresses them.

Operational workflow frequently develops organically with comms professionals building muscle memory through on-the-job practice over time. Codifying unconscious learned experience can be hard, but for technology to be effective it must support specific, definable and repeatable tasks.

From Dickensian workhouse to automated SAAS platforms

As with much of the communications industry, media intelligence services have historically been labour intensive.  When my old media analysis business Metrica, merged with media monitoring company Durrants, we saw first-hand how many people were needed across multiple day and night shifts to monitor the media effectively for clients.

If this resembled a Dickensian workhouse, it was because that is how media monitoring businesses originated. William Durrant founded his ‘press cuttings’ business in the 1880’s, alerting aristocrats, socialites and business leaders when they appeared in the press. The process of manually reading every media outlet, physically cutting out articles and posting them to clients was preserved in amber for 100 years – the only difference being occasionally changing the colour of the card to which the cuttings were attached!  This only started to change in the late 1990’s as scanning and optical-character-recognition allowed newspapers to be digitized and client keywords to be identified using emerging search-engine technology.

For those of us running media analysis companies, things were not much better.  Despite the use of computers from the outset – one of the earliest media analysis companies, CARMA, an acronym for Computer Aided Research and Media Analysis – still required humans to read, interpret and codify articles.  This was costly to run at scale, even with human labour increasingly offshored to cheaper locations.

Venture capital and private equity money poured into the industry which turbocharged the growth of software-as-a-service online platforms, resulting in the media intelligence market we see today.

If AI is written on the tin, what is in the tin?

Since the emergence of ChatGPT in 2023, media intelligence companies have found themselves in an AI arms race – competition is incentivising firms to overclaim how much AI is involved throughout their workflows and to rush new developments out to market before they have been properly tested and validated.  This may be what investors want to hear, but it can leave customers confused and disappointed.

The hype around Large Language Models is leading to challenges in how the technology can be integrated into media intelligence systems.  While LLMs are good at some things, like summarizing content in natural language, they can struggle in others, such as turning natural language into the structured data at scale that is needed for quantitative analysis.  The costs incurred from using LLM APIs means media intelligence firms are reluctant to use the technology across the billions of items of content that they typically need to handle.

Many of the platforms we reviewed were still using legacy technologies for much of their workflow.  Common examples include Boolean keyword searches, which can produce poor precision or recall, and keyword-based sentiment analysis, which can struggle in sectors such as charities and cybersecurity where organisations are positively associated with negative topics.

Accurate identification of client-specific key messages – a must-have in many comms measurement programmes, especially where the objective is brand positioning – can be problematic using these legacy technologies.

As a result of these challenges, many of the media intelligence companies continue to rely on in-house and outsourced human teams of analysts to ‘code’ articles on their platforms for some of these more demanding metrics.

Ask me anything?

In the opening presentation of AMEC’s recent AI day, Ant Cousins, VP of Product for Meltwater, said that AI was likely to lead to “more of a UX revolution than a data science revolution”.  This seems to be the case with many of the market leading media intelligence companies launching new AI-driven interfaces.

The rise of dashboard-driven platforms led to the parallel rise of specialist roles within in-house comms teams and PR agencies looking to get the best out of them. More data and more functionality tended to make these platforms even more complicated to use. Media Intelligence’s dirty secret is that most comms professionals don’t often use the platform to which they subscribed and those that do tend to follow the same few basic tasks. A few experienced ‘power-users’ are often relied on to get beyond this rudimentary functionality.

The development of natural-language chat-bot style interfaces, potentially negates the need for deep knowledge, empowering all users to be able to access the same level of insights. But this potential benefit also risks becoming a drawback. Users can ‘agentically’ create media summaries and measurement reports from a simple prompt without needing to see the underlying content and data. But if they haven’t seen the underlying content and data, how confident they be that the outputs generated reflect reality?  This can be a serious problem when you are in a meeting with the CEO or an important client and you get challenged on your reported findings.

Large Language Models can struggle in the interface between natural language and quantitative data – the ‘counts and amounts’ needed for measurement. Many of the foundation ‘reasoning’ LLMs outsource quantitative tasks to separate engines (often python emulators) and then attempt to interpret the results using natural language. Because the LLMs themselves don’t have an internal worldview of the data, there are risks in interpretation in both directions – in ‘understanding’  what the user is asking for and then in making sense of the quantitative data that are returned.

When doing a deep dive review of different platforms, we have seen examples of agentic summaries and reports that have described different numbers from the dataset on which they were supposed to be based.  In one example, when asking the chatbot interface how the dataset was defined, the system refused to give a response!

We have also seen AI generated ‘insights’ on dashboards that merely describe the numbers in the charts (which you can see by simply looking at the charts) rather than adding value by interpreting and summarizing how the underlying content is affecting the numbers.

Principles for an AI-driven world

Professional trade body AMEC (Association for the Measurement and Evaluation of Communication) is working to establish principles and best-practice guidance on how technology and AI should be used in comms planning and measurement. The organization has teamed up with the Market Research Society and several other international research bodies under the Global Data Quality Initiative to “address the risks associated with poor accuracy and trust in unstructured social, media and related voice of customer data”.

AMEC Board Director, and founder and CEO of Converseon, Rob Key, is leading the initiative on behalf of AMEC and has stated: “finding truth and accuracy in unstructured data is challenging but we have an obligation to use responsible AI to accurately reflect the opinions of consumers and other key stakeholders”.

Last summer, AMEC launched the latest ‘4.0’ iteration of the Barcelona Principles with guidelines to address the rapid rises in technology and AI.  The culminating principle states that “Ethics, governance and transparency with data, methodologies and technology builds trust and drives learning.”  This principle includes the advice that “measurement and evaluation should be transparent about the use of AI and automation across the full workflow and that when using AI and automation there should be a commitment to data quality, performance tracking and human-in-the-loop governance”.

Technology and AI have already made a huge difference to media intelligence – automating large parts of what was a very manual process, increasing scale and reducing costs – and are likely to continue to do so.

In my view, the core competence of media intelligence, and comms measurement and evaluation more broadly, is the critical thinking needed to translate a communications strategy to definable quantitative and qualitative data – and vice versa.  This comes from asking specific questions.  What are the communications objectives?  Where does comms sit in the ‘marketing funnel’?  Who are the target audiences and what is the media universe that they are most influenced by?  What are the perceptions and actions that we want to affect?  How should the brand be positioned and what are the tactics of achieving this?  How is the organisation developing thought leadership and who are the spokespeople that will deliver this? What are the right key performance indicators, what does ‘good’ look like and what are the appropriate benchmarks to establish this?

Conversely, any resulting metrics, without understanding the drivers behind them and how they translate back to strategy, are at best meaningless, and at worst, misleading.  How can we be confident that the data is both accurate and relevant?  What is affecting the trends in the data?  What are the emerging issues that could affect our reputation?  What do the numbers tell us about achieving our objectives? What actions can we take as a result?  Where should we prioritise our efforts? What should we do more of and what should we do less of?

For media intelligence to be truly effective, it will need technology and people to work in partnership, thinking critically through these specific issues.  AI and automation can be part of the answer, but only if we take responsibility for ensuring that we are first asking the right questions.

————————————————————————————————————————

Find out more about the AMEC Strategic Communication and Planning Diploma where you will be working with Paul as course leader.

Discover the course