#comatose servers

One of the most surprising things about the data center industry is how cavalier it is about the number of servers sitting around using electricity but doing nothing.  We call such servers “comatose”, or more colorfully, “zombies”.

In 2015 we did our first study of this issue using granular analysis on a small data sample (4000 servers) for a six month period in 2014, using data from TSO Logic.  Now we’re back with a sample four times bigger, covering six months in 2015, and with additional detail on the characteristics of virtual machines. 

My colleague at Anthesis, Jon Taylor (with whom I conducted the study) wrote up a nice summary of the work here. You can also download the study at that link.

Here are a few key paragraphs:

Two years on the data set from which the original findings were drawn has grown from 4,000 physical servers to more than 16,000 physical servers and additional information on 32,000 virtual machines (VM) running on hypervisors. The new findings show improvements, as well as an alarming wake-up call.

On the upside: when an enterprise acted to remove physical zombie servers when presented with evidence of the problem’s magnitude, they were able to reduce the amount from 30 percent to eight percent in just one year. On the downside: new data show that some 30 percent of VMs are zombies, demonstrating that the same discovery, measurement, and management challenges that apply to physical servers also apply to VMs.

The study confirms that the issue is still not being adequately addressed. New data indicates that one quarter to one third of data center investments are tied up with zombie servers, both physical and virtual. Virtualization without improved measurement technologies and altered institutional practices is not a panacea. Without visibility into the scale of these wasted resources the problem will continue to challenge the data center industry.

Here’s a key graph from the report:

There are some complexities in comparing the new data with the older data, because one facility in the 2014 sample decided not to allow its data to be used for the 2015 sample.  The remaining facilities in the 2014 sample, when shown evidence that one third of their servers were comatose, took action and moved from more than 30% comatose to 8% comatose in just one year.  

We corrected for these changes in an attempt to estimate the percent of comatose servers for enterprises that haven’t dealt with the problem, and the result is an estimate that about one quarter of servers in such companies are comatose (see the middle bar of the figure above).

Surprisingly, the percentage of virtual machines that were comatose was about 30% (see the right most bar above), indicating that the same management failures that lead to high percentages of comatose servers also afflict virtual machines. Virtualization without institutional changes is not a panacea!

Image:  Critical data centre at University of Hertford. Licensed under a Creative Commons Attribution Share-Alike Unported 3.0 license.

The Wall Street Journal today has an article by Bob McMillan highlighting my work with Anthesis and TSO Logic on zombie servers (those that are using electricity but delivering no useful computing services).  To download our most recent report on the topic, go here.

Here are the first few paragraphs:

There are zombies lurking in data centers around the world.

They’re servers—millions of them, by one estimate—sucking up lots of power while doing nothing. It is a lurking environmental problem that doesn’t get much discussion outside of the close-knit community of data-center operators and server-room geeks.

The problem is openly acknowledged by many who have spent time in a data center: Most companies are far better at getting servers up and running than they are at figuring out when to pull the plug, says Paul Nally, principal of his own consulting company, Bruscar Technologies LLC, and a data-center operations executive with experience in the financial-services industry. “Things that should be turned off over time are not,” he says. “And unfortunately the longer they linger there, the worse the problem becomes.”

Mr. Nally once audited a data center that had more than 1,000 servers that were powered on but not identifiable on the network. They hadn’t even been configured with domain-name-system software—the Internet’s equivalent of a telephone number. “They would have never been found by any other methodology other than walking around with a clipboard,” Mr. Nally says.

Read more (subscription required)...