Life isn't always simply good or bad, true or false or black or white, and it turns out the same can be said for social media evaluation.
We’re often seeking to measure the positive or negative sentiment of what people are saying on social media, but does this really tell us anything useful?
DECC’s #BackClimateAction Tweetathon, held in November 2014, demonstrated to us just how complex and nuanced measuring the success of social media activity can be. So I thought I’d share some of our insight in case it helps with the evaluation of your own social media campaign.
Thanks to the Science and Society team at the Department for Business, Innovation & Skills (BIS), we were able to work with Ipsos MORI who evaluated our campaign as part of their on-going work with BIS looking at public attitudes to science. So, thank you BIS!
Measuring the numbers of our campaign was relatively easy and we got impressive figures for potential impressions and reach (250 million and 100 million respectively), but that didn’t tell us what people were saying, thinking, feeling or doing and whether it was good or bad as a result of our campaign - this is where Ipsos MORI’s expertise came in…
Two key conclusions emerged from their analysis, both of which would apply to other campaigns using social media:
- If you’re measuring sentiment, clearly define what you mean by positive, neutral or negative before you get going. Consider what it is you want people to be positive about? Using automated analysis tools to measure sentiment revealed two things: a) our campaign was overwhelmingly positive (great!) but that’s probably because our hashtag is inherently positive, so anyone using it is likely to be positive about the cause, and b) where we did see the tool categorising tweets as negative, on closer inspection they were often wrong, picking up on tone incorrectly when the tweet was in fact neither negative about tackling climate change or our campaign. So remember, tools can’t do it all for you.
- Before you start, think carefully about the type of engagement you want to see. Given the positive / negative split wasn’t hugely insightful for our campaign; Ipsos took a different approach to measuring sentiment going beyond positive or negative and instead broke it down into what people were doing with the information. Our results were then able to give us a spectrum of engagement, from simply sharing knowledge to pledging action – a much more meaningful set of results that revealed the range of opinion and response to the campaign.
There is much more that can be said on these two points – and also on our evaluation more broadly – so I’d recommend reading the full report (which includes an Executive Summary).
Overall, the Tweetathon was a huge success for us; we had over 60 partners involved; over 16,000 unique authors on Twitter; trended above Christmas in the UK and saw a 14% increase in people pledging action against climate change. And thanks to the approach and insight from Ipsos, we now have a new, more robust method in place to measure activity in the future, and we’re starting with our #BackClimateAction Instagram competition which opens today! So if you’re snap-happy, get involved.