The creation of Hadoop in 2005, with its ability to store and process Big Data, helped companies and governments to identify anomalies and trends in the huge volumes of user activity data they collect as more and more transactions and interactions go on-line, even in brick-and-mortar shops. The ability to exploit the ‘wisdom of crowds’ (as exemplified by Google’s PageRank algorithm) by collecting, storing, and analyzing large volumes of data provides the basis for business intelligence (BI). However, most BI data processing uses data at rest — dipping into historic data lakes — and is thus reactive at informing what the outcomes and values of specific activities and business processes were.
Storage, Process and Access
If companies want to have more predictive analysis that processes data in real-time so as to optimize individual customer interactions, then Big Data Analytics needs to shift gears and be able to analyze data as it is being generated. Real-time streaming data analysis is just emerging from companies like Vitria, Guavus, SQLstream, and MindStream Analytics, shifting the focus from BI to operational intelligence (OI) — analyzing the streams that feed into data lakes. In the call center space, companies like Genesys are adding real-time analytics to their call center platform, offering to achieve similar personalization gains.
The positive outcome of deploying streaming analytics is a more personalized customer interaction, and — hopefully — a more efficient servicing of customer needs. The classic example in retail is the offer of additional products and services to a customer when buying a specific item. Purchasing analytics, based on big data analysis of customer interactions involving the specific product, alerts the checkout person to provide additional information to the customer relating to the specific item being purchased. The result can be fewer return visits (because the customer gets all they need in one visit), fewer calls to help desks (because the customer actually has the instructions needed), higher sales volumes (because the customer buys all the components needed to complete the operation in the store), and higher customer satisfaction (because the customer actually completes their task).
Today, streaming analytics is still a feature reserved for large service providers, primarily in government, telcos, financials, and retail — in part because of the considerable processing power required, as well as because the corporate infrastructure and business processes must be adequate, and also because deployment still requires professional service consultants to map out the specific system integration processes required. In order to extend streaming analytic capabilities to a wider range of enterprise customers, the ‘time-to-value’ needs to be shortened. More packaged solutions must emerge, and public cloud vendors must offer services that are easy to configure and deploy.
Packaged solutions are widely available for real-time network management, including the Cisco Prime Analytics platform that delivers high-performance, real-time analytics with high scalability and low latency (of course tightly integrated with Cisco’s own hardware and software). Reasonably-priced public cloud services are now available from operators like Amazon Kinesis and Google BigQuery, where developers can send up to 100,000 rows of real-time data per second, and on-demand queries cost $5 per terabyte. To ensure real-time available capacity, customers can buy 5GB/sec reserved query capacity for around $20,000 per month. To reduce costly SI support, many customers are adopting more automated data discovery solutions such as Splunk to collect, index, and harness machine-generated big data in order to create actionable information from widely distributed, high volume data streams in near real-time (for more details see the Quocirca report).
The problematic aspects of this enhanced personalization relate to personal privacy. The algorithms that predict our individual behavior and our needs are commercially owned, and operate completely outside the customers’ insight or control. There is no opt-out clause in the real-time world.