Measurement and evaluation: the story so far

About the author

Richard Bailey Hon FCIPR is editor of PR Academy's PR Place Insights. He teaches and assesses undergraduate, postgraduate and professional students.

Here’s a broad but selective overview of the first day’s sessions at the AMEC Summit for those not in attendance who want a quick summary.

What do you know about AMEC – the International Association for the Measurement and Evaluation of Communication? Probably little more than its famous dismissal of AVEs in the Barcelona Principles of 2010.

Well, the conversation has moved on since then because there was scarcely any mention of AVEs or the Barcelona Principles at the AMEC Summit on day one. (The day started at 8.30 and the first mention of AVEs came at 15.45. The context: that AVEs are now dead, though this assertion was later challenged).

The direction of travel is to look beyond media (literally, means) and towards ends. So what’s the end purpose of public relations and communication efforts?

There’s no easy answer to this, because it usually involves winning hearts and minds, and is not always equated to sales.

Narrative one: from PR to integrated communication

And yet I detect a dangerous narrative that seems to have taken hold. If we accept that public relations is about more than media coverage – and we must; if we also accept that there is increasing convergence between public relations and marketing – and it’s hard to dispute this – then can we make the leap to assuming that all public relations is marketing communication?

As consumers, we are merely objects to be learnt about and sold to. But what about our lives as citizens? Not all relationships can be understood as commercial and transactional. We’re also family members, voters, volunteers, employees, fans and hobbyists. We have our values and interests, perhaps even some remaining shreds of a private life.

What about alternative narratives around citizenship and community? Perhaps a conference dominated by vendors – rather than academics – is no place for these subversive thoughts.

Narrative two: from words to numbers

We understand the power of words and stories. We’re masters of this art. Yet can storytellers become coders and data analysts? Can a wordsmith become a geek?

We know that data is nothing if it doesn’t lead to insight, but data is the necessary starting point. One comms leader spoke of the need for her team to learn the Python programming language. Another gave a glimpse into her ‘war room’ (or ‘nerve centre’), with multiple screens monitoring reputational risk in real time.

There’s more to this change than an educational challenge. It requires a culture change, which is where we turn now.

Narrative three: from success to failure (and back to success again)

In the bad old days of media coverage, there only was good news. I’ve never heard of any AVE calculation that reduced value because of negative coverage. Coverage was assumed to be a good thing, and we as an industry tended to focus on the sunny side (one academic famously described public relations as hemispheric communication, because the sun only shines on one side of the globe at any time).

This gave us an unhealthy emphasis on the positive, and an aversion to the risky and the negative.

Now, in the new world of data and numbers, several speakers spoke of the need to embrace failure – even to mandate it as a KPI within comms teams. If we only focus on what we know, if we only know what works today, how can we cope with change and disruption? How can we ever learn? What about the unknown unknowns?

An experimental, agile approach is being encouraged that welcomes innovation and the lessons that uniquely come from failing fast – and learning from failure.

It’s counter intuitive, but statistically proven. We heard the story of wartime mathematician, Abraham Wald, and the danger of system bias.

He was asked to recommend where aircraft should be given extra armour, based on the evidence of the damage caused to returning aircraft. His conclusion: to strengthen the parts that were undamaged. This seemed counter-intuitive, but his argument was that they only had the evidence of those aircraft that returned safely – in which case the damage inflicted was non-fatal. If they had access to the ones downed by enemy fire, then surely that would give the best indication of the dangerous areas that needed extra protection.

In other words, the problem with data is that we’re biased by what we can see, and tend to discount what we can’t see.

Narrative four: from press to broadcast and from data to visualisation

Written words are searchable and are subject to content analysis techniques. But how to capture and categorise TV, radio and podcasts in a similar way? Several organisations are working on transcribing live broadcasts to turn them into searchable text.

There’s no lack of data. The problem is with extracting insight from this data. We saw examples of how data can be presented visually to make it more dynamic and more readily analysed.

On reflection

My thoughts returned time and again to a novel published 70 years ago: George Orwell’s 1984, because of its anticipation of a world of fake news (‘newspeak’) and oppressive surveillance (Big Brother). But you might find another piece of early twentieth century dystopian science fiction more apposite: is this a Brave New World in which our work becomes more precise, our results more impactful and our budgets increase accordingly?