Hidden Evidence – The problem with positive case studies

When we put series of case studies together to alleviate clients anxieties about strategies we are employing we can run into trouble. The problem with case studies is that they are supposed examples of the kind of thing we proposing but they dont take into account all of the cases that followed this route but failed, so we cant take what is evident in these successful case studies as proof our strategy will work. There is also a ‘confirmation bias’ here as we can always find cases that support our hypothesis but again this cannot account for all of the variables that are at play.

Say we want to create a piece of content in the intention of earning views and hopefully reaching that critic mass we refer to as ‘viral’. We can find a bunch of case studies that will confirm we are on the right track and we should but we should not blindly accept this study as comprehensive and should take an extra step by also looking for unsuccessful cases which is much harder to do which is why it is sometimes referred to as ‘hidden evidence’. Then we can analyse these as to the reasons we believe they were not successful, something that the advertising community does much less of than it does celebrating and attempting to emulate successful cases. If we look at both why some succeeded and why others failed we will then have a much better understanding of the variables at play and can then have more confidence in our strategy.

In his book ‘Black Swans – The Impact of the Highly Improbable’, Nassim Taleb talks about how corporate culture’s obsession with focussing on successes obscures its view from learnings from the unsuccessful, which when you think about it should be treated with equal importance.

I would like to see a repository of unsuccessful case studies with commentary on perhaps why it didn’t work and what learnings could be applied ‘next time’. What a great strategy planning resource that would be ! It would however take a fair amount of courage to use such an idea as it means making our failures visible and public. Who wants to go first?