For many organisations there is an elephant in the room – ‘Big Data’. Big Data isn’t new and the term can be misleading. Large data sets have been in existence for many years, built and used by major companies for decades. In fact, many commentators have argued that ‘Big Data’ really just means more data. It has become so big, it is now a problem. The categorisation of Big Data is then formed.
At Experian, we process more than 1,151 billion records annually, with a global segmentation of more than 2.3 billion consumers in more than 30 countries, and demographic data on 700 million individuals and 270 million households combined – that’s pretty big data.
There is no fully agreed definition of Big Data. Simon Rogers at The Guardian took a tongue in cheek view: “Big Data is one byte more than you are comfortable dealing with.” But actually, this is rather close to the truth as Big Data is data that is so big it won’t fit on a single machine. It has to be spread over many machines. And it can come from anywhere, so it might be in strange and exotic formats. And it’s added to all the time.
The Four V’s – a term which IBM coined in 2012 to describe the phenomenon of Big Data, expanding an earlier version of Gartner’s three V’s. This is still a very useful starting point for thinking about data. Let’s break them down:
The vast amount of data generated every second, by organisations and consumers, is hard to comprehend. Many data sets are too large to store or analyse using traditional database technologies, and are also continuously being added to or updated. This volume is set to increase exponentially as the Internet of Things matures, though this eruption could also bring a greater volume and depth of insights into customer behaviour, if harnessed effectively. Either way, the challenge of increasing volume is clear.
Data comes in numerous shapes and forms, from geo-spatial data to website logs, from tweets to visual data like photos and videos. Though often overlooked compared with the well-publicised issue of ‘volume’, the variety of data out there is likely to be a bigger problem for most businesses. The ability to harness the different forms, together, can be the key stepping stone to the unified insight that many businesses are crying out for. A ‘holistic view’ is what companies should strive for, and dealing with data variety is the key.
While data volumes grow, the speed of data creation and use is increasing too. This means processing, storage, and analysis must accelerate in tandem – business advantage lies in having and acting on the most up-to-date information, which means receiving data and insight as soon as possible, then acting with equal speed. This high velocity also means that data is becoming more and more perishable, as it’s updated or made obsolete faster than ever before. Real-time analysis helps to deal with velocity, allowing businesses to make decisions based on the most up-to-date information.
Veracity is about ensuring the reliability and validity of the insights derived from data. Inaccurate data is virtually worthless, even damaging in some cases. The flipside is that chasing veracity can lead to over-cautiousness, as organisations and individuals wait for perfect, clean data before making any decisions – something which is very often impractical. Veracity often depends on individual users, meaning engagement with data and a commitment to cleaning it up – and keeping it clean – are critical on a person by person basis.
While these four V’s represent significant data-related challenges, we’ve reached a point where they no longer encapsulate all of the major issues. We would argue that there are two further V’s to add to the list, to help provide a more complete picture. ‘Vulnerability’ and ‘Value’ should be considered by organisations when assessing their data landscapes.
The proliferation of data has left many people feeling exposed and vulnerable to the way their data is being used. As people think about the issues and learn more they don’t always become more reassured – in fact, it makes some more worried because they were not aware of how much data is being collected and used.
Conversely, a growing number of ever more tech-savvy consumers are willing to sacrifice some privacy as a trade-off to the benefits of digital, personalised technology, but under their own preferences and conditions. Increasingly, people want to be informed about data use and have the ability to easily opt-in or out at any point in time.
The challenge for organisations is to find a way of addressing their customers’ concerns and exceeding their expectations. In order to alleviate confusion and apprehension, a growing number of companies are moving towards privacy by design and becoming far more transparent around data usage and value.
In this context the provenance of the data, right back to its origin, will be key. Organisations know that data is exploding all around them; mobile data generation, real-time connectivity and digital business have changed the scene entirely and made things more difficult in many respects. Analytics have an increasingly important role to play in data security and are already transforming intrusion detection, differential privacy, digital watermarking and malware countermeasures.
However, security is also about building brand reputation and trust. Strong security practices, including the use of advanced analytics capabilities to manage privacy and security challenges, can set businesses apart from the competition and create comfort and confidence with the public.
At the most simplistic level, data has no intrinsic value. It only becomes useful when you’re able to extract the insight needed to solve a particular problem or meet a specific need. Once you can do this, the data acquires value through the business impact and consumer value this insight delivers.
Consumers are looking for value in terms of convenience, better products and better service. Organisations are seeking value via more engaged customers, lower costs and reduced business risk. For both parties to be successful there has to be a fair exchange – each needs to feel satisfied.
This means that data – and data analytics – is only valuable if it generates some form of payback. Thankfully, advances in analytics are helping businesses to achieve this more consistently, combating the challenges of data volume, variety, and velocity, and delivering the all-important value. This is being done by:
- Tying insights more closely to business decisions.
- Drawing on, integrating, and analysing new data sources.
- Moving beyond simpler business intelligence and analysis, towards diagnostic, predictive, and prescriptive analytics.
- Developing data strategies that relate closely to clearly defined business goals, outcomes and use cases, rather than simply deciding a ‘Big Data Strategy’ is required, and proceeding in a business vacuum.
Value exists in a two-way model between both parties. The organisation provides added value in the form of superior, more relevant content, products, and user experiences, and the recipient engages with that content, consuming it, buying products and signing up for offers and promotions.
To harness the full value that data can offer, organisations, governments and regulators will need to invest time and money in educating both the general public and businesses about how to manage the vulnerabilities and value opportunities that data presents. In addition focus and attention should be made to:
- Invest time to clearly understand the value buried in data and develop a clear data strategy.
- Monitor regulatory change carefully, particularly with regards to security requirements.
- Consider how the use of data could be more transparent to customers.
- Proceed cautiously and ensure that you develop an appropriate business model.
Big data is big. But, it needn’t be a problem. As outlined in this blog, cautious, careful and considerate use of data can help leverage the value and insight buried within it. The result is a uniformed and synchronised platform that enables better outcomes for the business, and the customer.