Jun 19, 2012
In a post that I made on my Network World blog, I spoke about the challenges of running Hadoop over the WAN. What I didn’t get a chance to cover is why Silver Peak optimization is such a great complement to big data, like Hadoop.
Hadoop requires a predictable, low-latency network, which is why most implementations tend to reside at a single site. The Large Hadron Collider experiment, though, generates so much data of interest to researchers all over the world that the Hadoop network spans hundreds of universities and laboratories and over 2000 WAN links.
The University of California, San Diego (UCSD) is one of those organizations and it’s connected via dual, 10 Gbps links back to Switzerland housing the primary data. GridFTP is used to move up to 15 Gbps of data at any one time between locations.
The research uncovered a number of problems of running Hadoop over the WAN, problems which Silver Peak’s optimization technology largely addresses:
And since Silver Peak optimization is just software it can be downloaded in minutes from our Virtual Marketplace and deployed across most servers running a major hypervisor. As computational loads move between nodes or sites, optimization instances can follow. So while Big data may require big optimization, it most definitely does not require big iron.