How the Internet Kept Humming During 2 Hurricanes
By James Glanz, NY Times, Sept. 18, 2017
At one node of the industrial backbone that keeps the internet running, employees sheltered from the worst of Hurricane Irma in a stairwell of a seven-story building in downtown Miami. When the power had gone out, diesel generators instantly kicked in to keep the lights on and prevent the internet from going down.
In Houston, at another of these nodes, called data centers, Hurricane Harvey pushed waters so high that a live fish flopped in a loading dock, but the physical defenses held.
Yet another data center, west of Houston, was so well prepared for the storm--with backup generators, bunks and showers--that employees’ displaced family members took up residence and United States marshals used it as a headquarters until the weather passed.
“It wasn’t Noah’s ark, but it was darn close,” said Rob Morris, managing partner and co-founder of Skybox, the company that runs the center.
For all their seeming immateriality, the internet and the cloud rely on a vast industrial infrastructure consisting of data centers linked through a sprawling network of fiber optics. The facilities are stacked with servers--boxlike computers that crunch the data for everything from hospitals, law enforcement agencies and banks to news websites, email and weather reports--that cannot be without electricity and cooling for even a fraction of a second.
Yet even as millions of people lost power across Florida, and thousands of homes and businesses were flooded out in Miami and Texas, the heavy digital machinery at the heart of the internet and the cloud held firm.
Though the storm disabled some cellphone towers and local connections, Jeff Eassey, a manager for Digital Realty who hunkered down in the Miami building, said the center never stopped processing and transmitting data. It lost utility power around 8:30 a.m. on Sunday, Sept. 10, but supplied its own electricity with the generators.
When the storm eased and he walked outside, Mr. Eassey said, he immediately saw the effect that Digital Realty and other data centers had by keeping the servers going. “Everyone was talking on their phones, searching on their phones, and commerce that uses the internet to do their business was up and running,” he said.
A list of Digital Realty’s top customers shows why those operations are so important: Clients include IBM, Facebook, CenturyLink, Oracle, Yahoo, Morgan Stanley, AT&T and JPMorgan Chase, among others.
Inside the centers, the drill during a natural disaster can be overwhelming. One company, EdgeConneX, lost utility power for three days at a Miami data center, two days at a center in Jacksonville, Fla., and two days at an Atlanta location. But Rich Werner, director of operations at the company, said that backup generators turned on and service was never interrupted.
“Data center operations, to me, is 362 days of boredom,” Mr. Werner said. “And then you get these hurricanes coming through, and it’s three days of pulling your hair out.”
There are no legal standards for data centers unless they house servers for clients like government agencies, which require special protections.
But a professional organization, the Seattle-based Uptime Institute, rates the facilities on four tiers of resiliency for events like storms, earthquakes and run-of-the-mill power failures--possibly one reason the internet performed creditably during the hurricanes. The institute’s engineers “go to the site and pull the plug,” said Mark Harris, senior vice president of marketing, “and to be certified, it has to keep running.”
“So when a hurricane or a tornado or an ice storm happens, the entire data center has been designed to withstand these external factors,” he said.
During Harvey and Irma, those measures seemed to work, said Doug Madory, director of internet analysis at Dyn, a web performance company with servers at a Miami data center. “I’m not aware of any core internet services going down,” he said.
One necessity is building above the 500-year floodplain, according to Gary Wojtaszek, president of CyrusOne, whose facilities include a data center in the mostly flooded Galleria area of Houston. The water stopped a few feet short of the building, which ran on backup power for several hours, Mr. Wojtaszek said. He added that “practically every single large oil and gas company in the world” operates servers in its data centers--an indication of how commerce of all kind relies on the facilities.
Depending on local conditions, some centers got off easier than others. In Houston facilities run by Data Foundry, automated systems worked so well that employees pooled resources and watched the Floyd Mayweather-Conor McGregor fight on pay-per-view, said Edward Henigin, the company’s chief technology officer. When everything is working, he said, “what else are they going to do?”
Everyone in the business says that extensive preparation is critical. James M. Palmieri, Digital Realty’s vice president for data center operations, said that as Irma approached the building in Miami, engineers topped off diesel generators, called Mr. Eassey down from Atlanta to direct operations and waited to see how strong the hurricane would become.
“The storm got worse,” Mr. Palmieri said, adding that at that point, “the real issue is, how do we make sure that our employees are protected?”
Mr. Eassey said that about 14 people in the building--including employees, customers and security personnel--met in a sixth-floor conference room that Friday morning and realized that the worsening conditions, heavy traffic and overcrowded shelters in the area meant it was too late to leave. With reinforced concrete walls and heavy glass, the building was the safest place to be, they decided. The mood was calm.
“Most of these guys are seasoned Miamians,” Mr. Eassey said. “I was the out-of-towner. I would be exaggerating if I said anyone was nervous or panicked.”
The peak of the storm passed over the weekend, and when the power went out, the center was ready with diesel generators and a 10,500-gallon diesel tank on the site.
In Houston’s so-called energy corridor, the Skybox campus stayed above water, Mr. Morris said. Utility power never went down. But the site was stocked with thousands of gallons of diesel fuel, as well as food and water, emergency medical kits, showers, bunk rooms and flares.
The site, which Mr. Morris calls a “modern-day fortress,” began hosting not only employees but several of their families, whose houses were flooded. Then, Mr. Morris said, the building doubled as an emergency response center for the United States marshals. He said that roughly 50 people used or stayed on the campus at one point or another during the storm.
Asked if there were anything in his operations that he would change before the next storm hit, Mr. Morris did not hesitate. “We’ve decided to purchase a washer and dryer to keep on site,” he said.
