Oct 17, 2014
At 5:00 PM on October 17, 1989, a bank manager in Santa Cruz was waiting for her tellers to close for the day. Heading to her car, she walked down the sidewalk next to a historic brick building.
Up in the San Francisco Bay Area, a technician for a major Telco provider had just commuted over the Bay Bridge from San Francisco to his home in Berkeley. He was getting ready to watch the World Series, and had gone into the kitchen to grab some snacks.
At 5:04 PM, the earthquake began; though it lasted only 8-15 seconds, witnesses say it seemed like the shaking would never end.
The manager had made it home before the quake struck. If she’d left a few minutes late, she would have been hit by a pile of bricks that fell from the window under which she’d just passed.
The Telco technician heard a snap and a crash, and saw trees outside bobbing vertically instead of swaying from side to side. If hadn’t left work early to watch the game, he would have been snarled up in traffic behind or in the middle of the collapse of the upper level of the bay bridge.
Both people recalled how the Loma Prieta earthquake of 1989 shattered the world as they knew it. And they recount how restoring a town, or a telecommunications network from a natural disaster required rapid response and arduous planning, mixed with a dash of serendipity.
In Santa Cruz, the damage was estimated at $1 billion. In the downtown area alone, 20 buildings had collapsed, displacing 50 businesses. Total damage from the quake is estimated at $6 billion.
In the Bay Area, the Telco technician’s clients at the time relied on several key data centers. One client—a retail store—had a data center conveniently located right next to the Hayward Fault.
His banking clients backed up their data by shipping reels of tape to a “safe” disaster recovery site in Sacramento. Bandwidth was typically limited to a fractional T1 line, around 128 kbps. Tape backups were updated with incremental data transfers over these small bandwidth lines, with batch backup jobs running nightly from midnight to 4am.
When the earthquake struck, a major termination center on Folsom Street in San Francisco went completely offline; main power and backup generators failed. Data was supposed to reroute to the Sacramento data center, but the Sacramento data and voice circuits were completely overloaded. Some carriers couldn’t reroute their backup data at all. In Northern California alone, 154 out of 160 central offices also lost power.
Meanwhile in Santa Cruz, the bank manager and her husband immediately set to restoring commerce in the downtown. Within days, her husband, who ran the Chamber of Commerce, formed a planning committee consisting of business owners, local government, and city residents to create a plan to restore the city as quickly as possible. Several argumentative meetings and stroke of good luck later, and many downtown businesses were back up and running. A committee member had had a chance encounter with someone from a company in Canada that made large structural tents. The City ordered five of these large tents—each holding five businesses—and placed them in parking lots behind Pacific Avenue. Many of the local businesses that had temporarily shuttered after the earthquake were able to resume operations within weeks of the disaster, and flourished there for a few years while the downtown was rebuilt. The original plan to restore Santa Cruz was formally adopted two years later, and revised seven more times.
In the Bay area, the Telco reevaluated its disaster recovery and backup plans. They could no longer consider Sacramento, California as an earthquake-free zone, and extended their Western disaster recovery sites to places like Salt Lake City, Phoenix, and Reno. Some businesses even extended backhaul T1 circuits to the east coast. Since 1992, they have conducted annual disaster recovery exercises.
Today, 25 years, $22 billion in local infrastructure upgrades, and several network paradigm shifts later, how quickly will the Bay area, and Silicon Valley bounce back from The Next Big One?