Big Data: Ripe or Hype?
There is, perhaps, no better way to put forth the importance of Big Data in today's world than as Google's Eric Schmidt put it: Every two days now we create as much information as we did from the dawn of civilization up until 2003. That's something like five Exabytes (one Exabyte is 1 billion Gigabytes) of data. Of course it is a bit of a hyperbole in the way it is put because the internet, as we know it, exists only since the last 40 years.
Even so, in four decades, the cyberspace has become an integral part of our lives. The explosion of mobile networks, cloud computing and new technologies has given rise to incomprehensibly large worlds of information. In 2005, there were 30 billion Gigabytes of video, emails, Web transactions and business-to-business analytics. YouTube claims that around 24 hours of content is uploaded on its servers every minute.
Moreover, the internet has now become portable. According to a Cisco study, mobile internet alone registered 18 Exabytes of traffic in 2013. Compare that to the scenario 13 years ago, when all of the internet registered a total traffic of only 1 Exabyte. The same study predicts that in 2018, the average smartphone will generate 2.7 GB of traffic per month. This will be 5 times the 2013 average of 529 MB.
As much as data is being sold as a commodity, it is also being valued more and more as a factor of production; much like human or physical capital. The use of Big Data to discern patterns and make better decisions is becoming the basis of competition and growth for individual firms, enhancing productivity and increasing the quality of products and services. Firms are being analysed on their data equity, just like brand equity. While the sophisticated companies have preferred to develop their in-house data analytics capabilities, the smaller, analytically challenged ones prefer to outsource. Either way, the general notion is that firms that can harness big data will trample the data-incompetents. That such kind of analysis would propel a company’s business and result in growth was unimaginable a decade ago. And now, any business running without accurate data is running blind. The Economist reports a strong link between a well-defined data management strategy and financial performance.
Big Data is what happened when the cost of keeping information became much less than the cost of throwing it away.
Undoubtedly Big Data is evolving the way businesses strategize and garner higher sales, but how much of it is just puff? In a 'hype cycle of emerging technologies' report published by Gartner Inc., the authors give buzzwords a reality check by analysing trends of over 2000 technologies and gauging their maturity, business benefit and future direction. The report proposes that technologies follow a cycle which starts with an innovation trigger, reach a peak of inflated expectations, come down through a trough of disillusionment, rise slightly through the slope of enlightenment and finally witness a plateau of productivity. The report puts Big Data at the top of the crest of inflated expectations, a period essentially characterized by early publicity making vendors flock the market, customers getting anxious that they don't yet fully understand the technology, and the expectations about what Big Data can do for an organization being over-inflated.
A supply-side perspective on Big Data would reveal that the revolution is all about storing, processing and analysing data that was previously discarded as being too expensive to store and process. This includes the traditional Web, network and server log data, as well as new data sources such as sensor and other machine-generated data and also social media data. Essentially, Big Data is what happened when the cost of keeping information became much less than the cost of throwing it away. As an example, an advertising firm Neustar moved from earlier paying 1,00,000 dollars for storing 1 Terabyte of data at an enterprise data warehouse appliance to only 900 dollars in Hadoop, an open source big data storage framework.
The World Economic Forum has heralded Big Data as a new asset class, and put it on the pedestal by deeming it as 'the new oil'. While this supply-side burst that the Big Data industry is going through has led to a lot of innovative start-ups, it is a bit of an exaggeration to call it 'the new oil'. When oil was discovered, and converted to gasoline, there was a revolution in transportation as cars replaced horses and commercial air transportation replaced railroads. Big Data is, at best, a possible revolution in the making and only time will tell when and how this new technology will become an integral part of our lives in the truest way. At the risk of sounding overtly cynical, calling Big Data the new oil seems a bit like undermining the achievements of the last century.
Although it is conceivable that Big Data will be a bust for the things people expect it to be useful for, and this is strictly being said in a very short-term perspective (around 5-10 years according to Gartner's report), it might be possible that the economic punch of Big Data is in the making as engineers trained in data manipulation make their way through college. There is no disputing that Big Data is providing that extra edge to a wide spectrum of businesses, but some entirely new use will have to turn up for it to be exploited to its full potential.