Lessons in measurement and evaluation

There’s a clear narrative to explain the measurement journey that individuals, organisations and indeed the whole public relations business has been on. It’s about a shift from a focus on awareness raising through media coverage to an emphasis on measuring real world behaviour change outcomes.

Nothing tells this story better than the evolution of the professional association AMEC. Now standing for the International Association for the Measurement and Evaluation of Communication, AMEC began life as a rather more humble trade association: the Association of Media Evaluation Companies.

Memories may be fading of a world before Google, but this was a world of print coverage gathered from newspapers and magazines in the form of physical clippings and often presented in coverage books left for all to see in the headquarters reception area. One unfortunate side effect of this was that broadcast media coverage tended to be marginalised – certainly here in the UK – because it could not be clipped in the same way and presented more widely.

This emphasis on print coverage inevitably led to some crude measures based on fallacious assumptions.

Crudest of all was simply to count the volume of clippings and assume that 50 this month must mean a 25% performance improvement from 40 the month before. It’s a fallacy because it takes no account of the quality of the publications or the nature of the coverage (was it positive, neutral or negative? how prominent was the mention of company X or brand Y in the piece?)

The next commonly-used measure was the now discredited advertising value equivalent (AVE) calculation. This took each clipping and calculated the value if an equivalent sized advertisement had been bought in that publication. Then, on the basis that editorial endorsement is more valuable than an advertising message, a multiplier was applied (this was typically three or four times, though I’ve heard values multiplied by five). The fallacies in this approach are surely obvious to see:

• Not all media coverage is positive, nor are most articles only about one company, one product, brand or service. Yet the crude AVE calculation assumes both (because advertising is always a ‘best foot forward’ message).
• The comparison of editorial against the advertising rate card at least has some basis in evidence; but where is the evidence to justify the ‘PR value’ multiplier? 
• What’s the AVE calculation of a positive mention on the BBC? Logically, it’s zero because you can’t buy advertising on the BBC’s domestic channels. Yet the BBC certainly counts as ‘major media’.

But the most important fallacy of all is this: 

• Media coverage is not the end result of public relations activity, the purpose of which is not to be in the media for its own sake, but to be in the media for its ability to reach and persuade readers, listeners and viewers to think or do something differently. We used to describe this obsession with media coverage as ‘vanity publishing’, reflecting a time when self-published books were considered inferior to those produced by a reputable publishing house. Nor is this problem confined to the past; there’s a danger that much of today’s influencer marketing involves the same vanity publishing in a new guise, just with added data points. 

I can illustrate the problem of an obsession with media coverage with an example to show my own limitations when, long ago, I was UK PR manager of a prominent global software company. I was in my office admiring a piece about the company in the Financial Times I’d helped to generate when the boss came into the office. He did not stop to praise me but instead asked two key questions.

‘How many people read that article?’

‘I don’t know – though the circulation of the Financial Times is 300,000 and many of those copies are read multiple times.’

‘And how many people bought the product as a result of reading the article?’

‘We can’t know that.’

Don’t know, can’t know. No wonder managers schooled in ‘if you can’t measure it, you can’t manage it’ culture were so often frustrated by public relations.

Today’s practitioners should be able to answer these questions. They can monitor website hits in reaction to announcements or stories in the major media, and assuming there are no other variables at that time, they can attribute spikes in website traffic to the PR activity. If that website visit then leads to a goal (a sign-up, a registration, an enquiry, an order) then there’s a perfect audit trail. Good times!

There are some clear lessons from this decades-long evolution away from media evaluation. I don’t claim this is an original or controversial narrative; indeed, it’s notable how in this area academic authors are in full accord with leading practitioners. There’s a clear thread from Jim Macnamara’s Pyramid Model originated 30 years ago to AMEC’s Integrated Evaluation Framework and the Government Communication Service’s Evaluation Framework 2.0.

So here are five lessons. These are not presented for the cognoscenti who by definition already know this. I’m not claiming to break new ground. They’re for those we teach on degree courses and on the intermediate level CIPR Professional PR Certificate course and for the many more unqualified public relations officers and account executives who need to get up to speed if they’re to turn their public relations jobs into a viable career.

Lesson one: Awareness is insufficient – behaviour change is key

Just as media coverage is merely a staging point towards awareness, so awareness is not in itself the end point of a public crelations campaign. Once we’ve made people aware of something, what do we want them to do with that information? Buy something? Quit something? Change their attitude towards it? Change their behaviour and do more of less of something?

It’s easy to expose the limitations of awareness as an objective. I’m aware of Porsche, I tell my undergraduate classes. This sometimes raises the desired titter because they can clearly assume that the shabbily-dressed older lecturer in front of them is hardly in the target demographic to drive a gas guzzling sports car.

On the other hand, I’m certainly aware of racism and am conscious of my white privilege. That puts me in a position to modify my attitudes and my behaviour.

The true measure of the success of public relations is whether you’ve shifted the dial in some way among target publics.

Lesson two: Media coverage is an outtake from public relations, not its outcome

This point has already been made. But I need to add that if your public relations measurement stops at media measurement, then you’re not measuring what matters, to borrow the phrase used by so-called Queen of Measurement Katie Paine. Think of your news release as an output of public relations; any resulting media coverage as an outtake of the same. You’re measuring what’s easy, but you’re nowhere close to proving the effectiveness of public relations, and so enhancing your status, salary and team budget.

Lesson three: Evaluation starts at the beginning, not at the end

Put simply – and it really is beautifully simple – if you have established SMART objectives at the outset, you have already built measurement into your planning.

The longer version is that evaluation is part of a continuum that starts with research, moves into objective setting, embraces stakeholder and issues identification, refines messaging and channels, and comprises monitoring and measurement as well as evaluation. In other words, evaluation is not an adjunct to public relations, it is the total process of public relations.

While it would take a book to describe this in full, the good news is that book has been recently published. We strongly recommend Mark Weiner’s PR Technology, Data and Insights.

Lesson four: How I Learned to Stop Worrying and Love the Data

There was a time when the key recruitment criterion for a public relations role was ability with the written word. And we know that because of the longstanding educational divide between arts and sciences, wordsmiths tended to shy away from numerate disciplines (maths and science).

Times have changed, though alas not within education.

Since data cannot be avoided, it should be embraced. Nor is it really about numbers; it’s about insights (Sam Knowles is a key author here; we’ve reviewed his data and insights trilogy: Narrative by Numbers, How to Be Insightful and Asking Smarter Questions).

Put another way, and to pick my data from an earlier example, how do we know that 40 or 50 are significant numbers? Is 25% a good increase? The raw numbers don’t tell us this so we need to present them and contextualise them. Graphs, charts and infographics are helpful. So in addition to being a wordsmith, you will now need to be a data geek and a visual communicator to make a strong start in public relations.

Lesson five: Repeat after me: Input, Output, Outtake, Outcome, Impact

These are the building blocks of every recognised evaluation model and by definition every public relations campaign. If that’s too much to remember, the key distinction is between outputs and outcomes (see lessons one and two above).

None of this is hard; I’ve not told you anything new or surprising.

But here’s something that is surprising. Advertising agencies long ago established a division of labour based on the very different skill sets needed for a winning team. There were the creatives, who you never allowed near the clients. Clients were handled by account managers. Then there were the planners who did the heavy lifting on research; the media buyers and so on.

Digital PR teams may be moving in that direction as the people who are expert in technical SEO are not the same people tasked with the softer skills of ideas creation, outreach and link building.

But in more traditional public relations teams, we’re expected to be versatile all-rounders. Perhaps we need to become more self-aware and recognise that no one individual can do everything and that we need a combination of right-side of brain creativity with left-side of brain planning within our teams.

Progress at last

A century on from people drawing salaries to work in a new discipline called public relations; 30 years on from the first iteration of Macnamara’s measurement model; around a quarter of a century on from the founding of AMEC and the birth of Google and twelve years on from version 1.0 of AMEC’s Barcelona Principles, perhaps this is a sign that we’re evolving as a profession. At last.