October 26, 2012 - FILED UNDER Agile Business
Unlocking Big Data Insight
We hear a lot about big data and its future role in the enterprise and rightfully so. Companies now have the tools and capacity to track and store tons of information about their customers. However as companies try and make sense out of the volumes of data they now collect, it’s become apparent that the data collected is too voluminous or too unstructured to be managed and analyzed by traditional means. Just take a look on the marketing front alone, it’s not uncommon to see reports overflowing with data covering existing channels like display, email, website, and search. For today’s business data collection is no longer the challenge, it’s turning that data into meaningful aids to assist with decision making, that’s the real difficulty.
Here, There, Data Everywhere
According to the MIT Sloan Management Review, “Every day, Google alone processes about 24 petabytes (or 24,000 terabytes) of data.” Companies are now able to collect more data than ever before. However turning those 24 petabytes of data a day into useful insight, is a new challenge facing companies. According to Irfan Kamal in a recent Harvard Business Review piece, “insights are relatively rare.” Today’s managers understand the power of data, but they lack the skills to accurately analyze and turn it into helpful insights. According to Kamal, data and insights are two entirely different animals writing “Delivering them [insights] requires different people, technology, and skills.”
The Netflix Prize
Take for example Netflix. The entire company rests on their ability to have sound data and logistics. Back in 2009, Netflix wanted to try and improve the company’s film recommendation engine. Dubbed the Netflix Prize, it was an open contest to search for the best collaborative filtering algorithm – this the primary algorithm used in the company’s movie recommendation engine. Despite this contest being open to the entire planet¸ only two teams could achieve the 10% improvement that Netflix set to qualify for the $1 million prize. This proves that “even with great data and tools, insights can be exceptionally tough to come by.”
Tips to be more Insightful
Given that coming up with insights is not an easy task, I’ve searched the web and distilled several tips that may help you and your team turn all that data into something that creates business value for your organization.
- Data Collection – Irrespective of what you are trying to analyze, making sure you gather the best data available is important. Think of it like the foundation of a house. The quality of the house doesn’t matter if you have a poor foundation to rest it on. Traditionally companies look at data as a static set of data stored in a warehouse. This traditional approach hurts companies. The speed of business today necessitates looking at data as a continual flow. “Streaming analytics allow you to process data during an event to improve the outcome,” notes Tom Deutsch, program director for big data technologies and applies analytics at IBM.
- Connect – Some data collected will simply be useful in the aggregate (for example, to look at broad trends), other data is more actionable if it’s connected to specific segments or even individuals. Importantly, the linking of social/digital data to individuals will require obtaining consumer consent and complying with local regulations.
- Analysis – Given the speed and volume that data is now collected, managing big data requires special techniques, algorithms and storage solutions. Kamal points out that the best way to generate accurate analysis is through collaboration, “Using statistics, reporting, and visualization tools, marketers, product managers, and data scientists work together to come up with the key insights that will generate value broadly, for specific segments of customers and, ultimately personalized insights for individual customers.”
There you have it. Big data is an important pillar for an organization’s future success. It’s time you stopped treading water in this ever expanding sea of data and started turning it into useful insight.