You Can Never Have Too Much Data

Big Data Has Huge Network Implications

You Can Never Have Too Much DataNot everyone has been swept up by the Big Data hoopla which only showed up a year ago in Gartner’s five-stage hype cycle. While it has rosy expectations for the future, the research firm had Big Data approaching its “peak of inflated expectations,” to be followed by trough of disillusionment, slope of enlightenment, and eventually plateau of productivity. That meant we were all about to be disappointed while vendors scrambled to create Big Data products and services,  while current and prospective customers grappled with what it all meant, how much would it cost, and if it was worth it.

Fast-forward to 2013 and Big Data is already starting to shake the foundations of the IT industry, with huge implications for the network market. According to new research from IDC, Big Data technology and services will grow at a 31.7% compound annual growth rate, around seven times the rate of the overall IT market, with revenues reaching $23.8 billion in 2016.

“The Big Data technology and services market represents a fast-growing, multibillion-dollar worldwide opportunity,” said Dan Vesset, VP Business Analytics, and Big Data research at IDC. “It is an important topic on many executive agendas and presents some of the most attractive job opportunities for people with the right technology, analytics, communication, and industry domain expertise.”

In a time of economic uncertainty and corresponding tight corporate purse strings, I’d think a seven-fold compound annual growth rate was pretty impressive, but not to GigaOm analyst Derrick Harris, who thinks that $23.8 billion by 2016 could start looking a lot bigger when analytics, SaaS applications, and other products are included. He said defining Big Data isn’t always an easy task, and breaking it out into a group of separate technologies might not be either, as IDC isn’t including the market for analytics software.

In July, IDC predicted that the analytics market — which is a critical piece of the overall Big Data picture — would hit $51 billion by 2016. While IDC segments its predictions into hardware, software, and services, it becomes a lot more difficult when you try and determine where cloud-based Big Data hosting and SaaS services revenues fit in, Harris said.

“However you define it, though, there’s no denying that Big Data will be a huge market. Whenever you hear terms like Hadoop, analytics, machine learning, machine-to-machine, data warehouse, NoSQL, data science, visualization, data management — you name it — someone somewhere is associating them with big data revenue.”

As for networks, a recent article wondered whether Big Data is a killer app for Software-Defined Networking: “A true killer app for SDN would be one with significant economic benefits to lots of companies and one that requires SDN in order to be viable on a broad scale. I think big data may be one such application area.”

If SDNs can enable applications to make better use of distributed processing resources, then Big Data applications can deliver acceptable performance using much less computing hardware, making them affordable for more organizations. That could make Big Data a killer app for SDNs.

A new Wall Street Journal item said technology leaders are predicting that 2013 will be all about networks, driven in large part by the advent of Big Data. Tim Watkins, VP of Western Europe for Chinese telecom equipment maker Huawei, predicted, “2013 will see the network continue its transformation — transformation to smarter, and bigger ‘pipes’. […] Bigger pipes refer to the anticipated surging demand for a 60-fold bandwidth increase in the next five years.”

Hyper-connectivity is set to redefine the relationships between individuals, consumers, and enterprises, according to both Ralf Schneider, group CIO for Munich-based insurance giant Allianz and Oliver Bussmann, CIO of German software maker SAP. “Gartner talks about the Era of Nexus and I like this because it describes the coming together of Big Data, constant access, and connection through mobile and social and new delivery capabilities through the Cloud,” said Bussmann.

Schneider said, “Companies have to find ways to exploit the potential of the massive amount of unstructured data available in the environment, e.g. from the Web, link it to their structured data and convert it into meaningful information, available for instant analysis, interpretation and use in real time.”

A 60X bandwidth increase in the next five years? Who said networking isn’t fun!

Image credit: JD Hancock (flickr) – CC-BY-2.0