Big Data’s Big Debate: Does Intuition Have a Place in Data?

Nate Silver – “Data-driven predictions can succeed — and they can fail. It is when we deny our role in the process that the odds of failure rise. Before we demand more of our data, we need to demand more of ourselves”

No doubt you’ve heard ‘the hype’ about Big Data and that we need it…RIGHT NOW! The claim that you need Big Data is not the case, what you need is the tools and brain power to analyse Data (whether it’s Big or not). So why are businesses taking their time, hesitating and dawdling around when it comes to hiring in the Data Specialists and implementing the tools?

Winterberry Group produced a study that surveyed over 150 senior figures throughout the digital world. 77% claimed that in the long-term, data management tools and platforms will play a critical role in their business. This figure is surprisingly low don’t you think? How could the analysis of consumer data not play a business critical role in any organisation; no matter how big or small the organisation, or the size of their data. Why does senior management avoid involvement with data?

It can’t be because Data Analytics is a new, scary and intimidating thing that only super geniuses can understand. Using algorithms, statistical techniques and coding to make sense of multi-channelled data in enterprise data warehouses with millions lines of data within various data sets, has been done for years’. Almost 6 years’ ago, back in 2008, Chris Anderson from Wired made the point that “this is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear”. With regards to the consumers, “with enough data, the numbers speak for themselves”.

With enough data, you can present facts about how your consumers behave. You can start to predict how successful your marketing, sales and financial strategies will be. Nonetheless, there have been many successful businesses that have thrived under the influence of intuition, where ‘naturals’ have lead the way creating products and services that supersede competition, but they weren’t really sure how they did it.

Guy Cuthbert, Managing Director at visual analytics firm Atheon Analytics, has tried to help many companies become data driven. He has seen “a huge number of opinion-operated businesses that don’t get why decisions could be made on data. I’ve listened to executives spout all sorts of opinions with no fabric or no substance behind them at all”.

So is intuition bad? Should we blindly follow data? NO! When intuition is used in conjunction with Data, invaluable insight is born.

Samuel Arbesman makes the point that “Big Data might be deep…but it’s not wide”. We can pull together snippets of information about an individual from their smart phone, PC, online behaviour and social media but, on the whole, this information doesn’t reflect how this person thinks.

Mille Findlay from Sense Worldwide agrees that “it might be possible to track how someone shops, or how they interact with their peers on Facebook, but it would be a mistake to imagine that this represents how they are all the time, in every situation. We also need to be aware of assuming that this data even represents all people – 76% of people globally still don’t use smartphones, and while it may seem that Facebook is everywhere, only 14% of the world’s population are active users”.

Do not take away the human element of instinct and intuition away from data, as after all, this is data about humans. In your business, when you’re looking to build an Analytics, Data Science or Insight function, look for those people who can perform the technical graft but can use their own intuition to pull out innovative discoveries.