Meaningful Measurement: Today and Tomorrow

As leaders in data and marketing, we should be constantly striving to discover new and rich data sources to contribute to both our understanding of marketing effectiveness and the measurement of this value.

Last week, while our DMA Awards category chairs were debating which of this year’s Gold winners would go on to take the Grand Prix, there was an equally animated discussion happening just next door. Hosted by the DMA and Salesforce, Judging Partner for the DMA Awards, the roundtable discussed effectiveness insights uncovered in the ‘Meaningful Marketing Measurement 2021’ report and what marketers should do to improve measurement today and tomorrow.

A special thanks to all the attendees (listed below) as without their openness and willingness to share their experiences, this roundtable and round-up would not be possible. The session was conducted under Chatham House rules and, as such, we’ve not attributed any of the comments or quotes – we may have slightly paraphrased these in some cases.

“I’m not surprised”

There is now a myriad of data and sources that many marketers have to contend with just to get a superficial picture of their marketing activity, let alone its effectiveness. The most recent ‘State of Marketing’ research from Salesforce found that the average number of data sources used by marketers is already around 10 in 2021, but more concerning is that this is predicted to only continue to rise in the future – making it ever-more complex. Even now, only 33% of marketers are completely satisfied with their ability to use data to create more relevant customer experiences.

This complexity in the data ecosystems marketers must navigate may go some way to explaining the current over-use of campaign delivery metrics as found in the ‘Meaningful Marketing Measurement 2021’. Currently, 41% of the measures seen in that analysis fell into this group and while these metrics are useful, they tell little about the true impact or effect of marketing – whether on consumers, customers or businesses.

As such, many of the roundtable attendees commented that they were “not surprised” but this over-use of campaign delivery metrics in place of true measures of effectiveness. It was agreed that these campaign delivery figures are important hygiene factors but aren’t good measures of effectiveness – particularly when left in a silo without any business, brand or response effects measured either. However, if you want to understand the true picture of effectiveness and measure it, creating and validating these metrics is also not a simple thing to do.

“There seems to be a lack of curiosity”

The wealth of digital channels – either entirely or in part – is also making access to these campaign delivery metrics easier and easier. A number of the attendees, as such, felt digital is a bad influence because precisely because it’s so cheap and easy to seemingly achieve such big numbers.

This seems to be more of a human issue, rather than a technological one, with a “Lack of curiosity” at the heart of it. By this, we mean the lack of interest in understanding and unpicking the measures that really matter and define success for a campaign.

Such attitude is combined with the ease of access to metrics that look, on the surface, impressive. This has led to the obsession and over-reliance on these ‘vanity metrics’. One possible solution discussed was the availability of industry benchmarks and a framework to help codify what good measurement looks like. This is something already underway with the DMA’s Media Council, following the launch of its ‘Making Measurement Meaningful’ whitepaper earlier in 2021, which is now working on a glossary of terms, measurement framework and best practice case studies for meaningful measurement.

“We have to stop HARKing”

A further challenge to improve measurement identified was the lack of focus on the ‘consideration’ phase when we come to measuring marketing. Many businesses focus on the ends or awareness and then selling, without measuring the how and why in the middle.

In addition, strategic explanations and justifications are all-too-often sought after the fact. This was described as ‘HARKing’, meaning “Hypothesis After Results Known” or essentially retrospectively tweak your claimed strategy once you have metrics that can match your story.

With so many measures (albeit largely vanity metrics) to choose from, this makes it all too easy for marketers to fall into the HARKing trap. But it requires faith and patience to create a solid investment in the more general strategy (and how you’re going to measure success) to support and build a brand for the long-term.

“Bad metrics lead to bad decisions”

Clearly defining objectives and goals is another strategic challenge many attendees identified. Alongside the need to avoid HARKing there is a need to ensure measures are aligned with these objectives is also required. Otherwise “Bad metrics lead to bad decisions” and can cause more challenges for the business in the long run.

One example shared by an attendee went as far as a brand creating a new customer offering/product specifically to maximise vanity metrics. This came at a direct cost to the customer experience and long-term loyalty, highlighting the danger of chasing metrics over strategic objectives.

This could also be down to a focus on short-termism, with many brands falling into discount traps to acquire customers that have long-term implications on brand perception. For example, leading customers to only shop with a brand when they have a discount code or an offer will create the expectation that there will be always one.

Brand investment, it was agreed by all those in attendance, is long-term but this is not widely understood within many businesses. Being all-too-easily overlooked by the need to meet monthly, quarterly or annual targets – even when these are measured using vanity metrics too.

Whether acquisition or retention is the objective, it was agreed that measurement and strategy need to go hand in hand. Measurement is the solution to prove the value of marketing, as well as the value of new and existing customers – whether respectively or in competition.

“Need to educate and talk about value, not cost”

To this end, there was also agreement that many “Need to educate” on the measures that really matter. In addition, we need to “Talk about value, not cost” as an industry. This education and focus on value goes across the industry too, meaning agencies, clients and service providers need to commit to this shift.

This shift would also have additional benefits in allowing marketers to truly understand and learn from their marketing too. Rather than chasing vanity metrics and targets, they can ask themselves “What did we learn?” and understand their campaigns in context to better understand what their customers want or need in the future too.

Bringing non-marketing leaders on this journey is also important, to ensure they understand the difference between meaningful and meaningless measures. In addition, educating them on the benefit of campaigns that didn’t go as expected, as a learning opportunity for the business.

This allows for more honest and open discussions across the business about success and a truer understanding within marketing teams of what some channels can and can’t do.

“Is effectiveness just more difficult to measure?”

Finally, when asked whether effectiveness is simply hard to measure the answer was unanimous: “Yes”. Even just defining what it is can be difficult - for some it’s engagement, sentiment change, sales or any of the other meaningful measures (not vanity metrics) available.

One attendee commented that it won’t be easy for some that have spent years talking about how important vanity metrics are. So, it won’t happen overnight, as the industry has spent too much time saying these vanity metrics are important, but we can ween businesses off them if we have better measures to replace them.

The latest figures from Salesforce’s ‘State of Marketing’ research shows some initial encouraging signs too. The pandemic has changed the way we all live, work, play and shop, but for many in the data and marketing industry it has meant their organisation have changed or re-prioritised the metrics they use (78%) and the same proportion of customers (78%) say the crisis should be used as a catalyst for improvement.

Finally, it was agreed that “We all need to push back when we see vanity metrics” and if we all commit to changing the culture and away from these meaningless measures to more meaningful ones. Afterwards, we will all be able to focus on true effectiveness measures that are aligned with strategic goals we, as an industry, can move the needle.

To find out more about the DMA’s Intelligent Marketing Databank and insights from the first phase of analysis, read the ‘Making Measurement Meaningful’ report.

Roundtable Attendees:

• (Chair) Jonathan Beeston, Product Marketing Director, EMEA, Salesforce
• Caroline Mastoras, Senior Account Executive, Media, Salesforce
• Emma Wicks, Head of Editorial Insight and Analytics, The Telegraph
• Fay Miller, Corporate Marketing, Marketing Cloud Campaigns, NEMEA, Salesforce
• Francesca Goldsmid, Account Executive, Datorama UK&I, Salesforce
• Matt Dailey, Chief Performance Officer, Havas Media UK
• Shahina Meru, CRO Director, Merkle
• Tim Bond, Head of Content Strategy and Insight at the DMA
• Tom Hunter, Head of Planning, Guardian News and Media