Yesterday I went to an Aperture-sponsored event in downtown Chicago that Andrew Fanara from the federal Environmental Protection Agency spoke at. Much of it was information that he has spoken about before and that we’ve reported, all of it around the data center energy efficiency issue that the EPA has gotten more involved with in the past couple years.

A major focus of the event was measurement. Leaders in the industry say that data centers must learn to measure how much power they’re consuming in order to reduce it. Then they can have before-and-after accounts of their Power Usage Effectiveness number, which is an efficiency metric dividing your total facility load by the IT load.

Your PUE number is like golf — the closer to 1, the better. At least that has always been the common wisdom. The goal, says experts, is to reduce your PUE. But sometimes an IT energy efficiency project can play games with that number.

Steve Yellen from Aperture said a virtualization project can temporarily hurt your PUE number. Take this example: You have 10 megawatts coming into a facility, and 5 of them are taken up by the IT load. You virtualize and consolidate servers, thereby reducing your server footprint, and thereby reducing your IT load. So now your IT load is only 4 megawatts even though your facility load is still 10 megawatts. So your PUE would go from 2 to 2.5.

Presumably there would be an adjustment. You would see that the IT load had decreased, and so you would adjust your facility load accordingly. According to Yellen, everything would be hunky dory again, right? Wrong. Your PUE would still take a hit. Let’s take the same example:

• Your facility load is 10 megawatts and your IT load is 5 megawatts, so your PUE is 2.
• You virtualize and consolidate so that your IT load becomes 4 megawatts, a one-megawatt reduction. Your PUE is now 2.5. Uh-oh.
• So you adjust, reducing your facility load by one megawatt to match with the IT load reduction. So now your facility load is 9 megawatts while your IT load is 4 megawatts. Your PUE is now 2.25, which is still worse than the PUE of 2 you had before you virtualized and consolidated. Still uh-oh.

In fact, the more energy you save with your virtualization/consolidation project, the worse it could be for your PUE. Say your project reduced your IT load by two megawatts instead of one. So you reduce your facility load by two megawatts as well. That means the facility load is 8 megawatts and the IT load is 3 megawatts, yielding a PUE of 2.67. Uh-oh.

Taking it a step further, any project that improves your IT load alone will yield a worse PUE. If you buy those new super-duper efficient servers, that could make your PUE worse. If you install blanking panels and move perf tiles around the right way, that will improve your PUE.

Let’s not panic here, because there is a good side to this. If I consolidate servers, I have fewer servers to cool. That presumably means that I’ll be able to reduce my facility load further because I might be able to shut down one of the cooling units. And maybe fewer servers means I can get rid of one of my uninterruptible power systems (UPS) units. In the end, it might all even out, but it may just leave you with a zero-sum game instead of an improved PUE number, which is what you think it would do.

In the end, what’s most important is reducing your overall power load, and if you can document how it all happened, all the better.

 Trends in data center power was the topic at AFCOM New England Chapter’s meeting this week, and apparently it’s a subject that resonates with members—at least judging by the nearly 100 attendees who showed up. (As the New England chapter enters its third year, President Rocko Graziano, whose real job is manager of infrastructure operations and services at L.L. Bean, said this was the largest meeting yet). Two speakers gave the audience their take on the some emerging trends they see taking shape.

Rudy Kraus, CEO of Validus DC Systems, a provider of direct current (DC) power infrastructure for data centers and telecommunications facilities, naturally sees a bright future for data centers powered by DC rather than AC-based electricity. Kraus cited a number of statistics from the likes of the Uptime Institute and McKinsey outlining just how much power data centers can save by switching to more efficient DC power. If data centers in the United States converted only 10% of their capacity to DC power, that would eliminate $1 billion in electric bills. The co2 emissions for a 10 megawatt data center with 17,500 servers would drop from 99,776,400 pounds to 59,865,840 pounds. Kraus invited members of the audience to do their own comparison by visiting an online calculator offered by Intel that analyzes facility-level efficiency of AC and DC servers.

The other speaker, Brian Ouellette, of J.S. Fleming Associates, a provider of power and cooling systems, spoke about the five power trends heading to a data center near you. The top trend, that energy efficiency is gaining importance, is pretty self-evident. The other four trends centered on ways to make data centers more efficient: New ways to scale UPS architectures into adaptive models that can adjust to changing power requirements; two-stage power distribution that reduces restrictions to cooling air flow, among other benefits; increasing use of monitoring with tools such as smart power strips (that monitor in-rack power) and branch circuit monitoring (that monitor each PDU output circuit). Ouellette also pointed out that data centers don’t have to go high-tech in order to become more efficient. When Ouellette asked the audience whether they use blanking panels in their data centers, only five people raised their hands. “Blanking panels are a great way to get the air where you need it,” he said. “Otherwise, you’ll get cross-contamination of air from your hot aisle and cold aisle.”

Since the announcement by Microsoft at AFCOM that they would be using containers like Rackable’s Ice Cube in their new data centers, much attention has been focused on the container-based set-up. Sun led the way with their Blackbox, and now Dell is releasing their own containerized data center.

But not everyone in the world of data centers think that these boxes are the cat’s meow. I asked Pete Sacco, President of PTS Data Center Solutions, Inc. for his thoughts. He referred me to his blog post on the topic, “Reflections on the Data Center in a Box”, and shared some additional comments:

Many data center-in-a-box providers only offer a homogenous data processing environment, which for most users won’t fly. Even those that are equipment neutral are available but have not sold well (e.g., APC’s Data Center on Wheels). The facility limitation of a data center-in-a-box hamper their adaptation. Access to sufficient power and chilled water (a heat rejection source) has always been, and will remain a primary cost driver of data center development. Therefore, the savings are not as dramatic as portrayed. Besides, due to form-factor and other attributes they are a lower availability solution than a traditional brick-and-mortar data center. In my opinion, their use will be limited to large enterprise and fringe applications at best.

Sacco succinctly points out the criticisms that were levied at Microsoft’s adoption of the concept in Computer World’s article. But beyond Sacco’s points about cost and flexibility, the CW article also picked at specific design flaws that some interviewees perceived existed. These were responded to in a thorough post from Mike Manos.

With Dell jumping into the pool with Sun, Rackable, and Verari, and IBM planning a container using it’s iDataPlex server, one must ask the question: Are all these companies making the right move? Competition is key to a market economy, but is market saturation going to occur? If the uptake on containerized systems isn’t going to be widespread, are these companies taking too big a risk in developing their own container systems?

Ron Hughes, President of California Data Center Design Group pointed out that the container approach is in fact, nothing new:

They’ve been making these for at least 10 years. Like everything else, they have their place. In the right setting, for the right client, they can be very useful and they work very well. We typically see them as either a short term solution until a data center can be built, or as a permanent solution if building a data center is not an option. The upside is speed to market, potentially energy efficiency and portability. The downside is cost and capacity.

I’m not convinced that it will be a widespread solution due to the high cost and limited server capacity. The solutions also vary as to whether they require separate infrastructure support systems or not. I haven’t seen any cost comparison to a standard data center, but for a medium to large application, I doubt that it is cost competitive. It will be interesting to see how these play out long term.

For an end-user company like Microsoft that is in need of large amounts of servers, the plug-and-play and interchangeability of the containers is attractive. But for other data centers, this solution may cost too much or be too large for their limited needs.

What do you think? Are containers here to stay — are we looking at the future of data center technology? Or is it just one tool in the toolbox?

The Environmental Protection Agency’s Energy Star program is trying to collect information from anyone with a data center to help the EPA get a sense of what’s happening nationally with data center energy consumption. In this video from the Uptime Institute Symposium, the EPA’s Andrew Fanara discusses how data center managers can participate in the program, by measuring their energy efficiency in a standardized way.

You can download the forms for the National Data Center Energy Efficiency Information Program from the EPA’s data center Web site.

Last week at the Uptime Institute Symposium, I met with Ken Ostereich of Cassatt and we chatted about the need for a radical change in data center design and operations in order to achieve the Uptime Institute’s challenge for end users to reduce energy consumption in the data center by 50% over the next 36 months. Ostereich observed that the proposed solutions are so incremental, a 50% reduction isn’t likely.

Christian Belady, Principal Power and Cooling Architect at Microsoft, actually compared the level of innovation to the movie Groundhog Day, 1993 Bill Murray classic where he has to repeat the same day over and over again.

In the video below, Ostereich calls on the data center industry giants like Yahoo, Google and Microsoft to serve as a “Data Center Space Program” helping best practices and new technologies trickle down into the more traditional companies. Ostereich has an excellent rundown of the week’s events on his blog, Fountainhead.

While the pace of change can be painfully slow, I argue that the innovations Ostereich and Belady are calling for are already happening.

Microsoft is exposing its internal operations to the public, and while they might not be giving away the secret sauce, if you’re paying attention you can still see the shape of what they’ve got under the sheet. They’re proving it can be done.

I too have heard the cries for metrics, hot-ailse/cold-aisle and virtualization ad nauseum at every conference over the past two years. But end user panelists at companies like Ford Motors and Boeing are talking about using cloud computing for non-critical applications and shutting down servers during the off hours.

Granted, many companies are putting blanking panels in their racks, virtualizing a few servers and calling it the best they can do, but those companies likely won’t be running data centers at all in five years. Google and Microsoft will be running their IT for them.

This week I sat down with Microsoft’s Senior Director of Data Center services, Michael Manos at the 2008 Uptime Symposium to talk about Scry, the company’s data center analytics tool that they touted at AFCOM. According to Manos, this tool allows him to look at his data centers’ energy use, carbon footprint, power bill and more — all at an incredibly granular level. It also allows him to slice and dice data to make decisions, for example Microsoft can look at the energy consumption of an individual product like Hotmail. The program is especially slick in that it ties into Microsoft’s CMDB and assett management tools. Microsoft has been touting this tool at various conferences throughout the past few months, but it’s not likely to become a commercial product for other companies since so much of the tool was built around Microsoft’s specific homegrown internal software. But the main point of Manos’s data center road show is to prove to people that it can be done. Microsoft is measuring and improving its energy efficiency in the data center and Manos is not waiting for someone to hand down the perfect metric or the perfect tool. Check out the video below, where Manos outlines how he uses Scry and in the second video he talks you through a 3-D rendering of the new containerized data center being built in Chicago.

One Tuesday, April 29, at the Interop 2008 conference in Las Vegas, Mellanox Technologies Ltd. was granted the Best of Interop award in the Data Center and Storage category for its ConnectX EN 10 gigabit Ethernet (GbE) server and storage I/O adapter with Fibre Channel over Ethernet (FCoE).

The Santa Clara, Calif.-based interconnect product supplier announced its ConnectX EN 10 GbE network interface card adapters for VMware- and Citrix XenServer-based virtual environments in February.

According to Mellanox, adapters maintain 9.6 Gbps throughput as the number of virtual machines in VMware ESX Server 3.5 scales up to 16 in multicore CPU environments. This improves server utilization because more VMs can be deployed per physical server while maintaining application I/O performance.

ConnectX EN is also the first adapter to support FCoE hardware offload and Priority Based Flow Control, both of which boost performance.

With support for PCI Express 2.0, ConnectX EN dual-port FCoE adapters are available today in silicon form for LOM (LAN on motherboard) applications and as PCI Express adapter cards that plug into server and storage systems with various media interconnect support including XFP, SFP+, CX4, and 10 GBase-T.

The finalists for the 2008 Best of Interop Awards were selected by InformationWeek’s panel of judges. Other finalists in the same category include Foundry Networks Inc.’s BigIron RX Module and Imation Corp.’s SSD PRO 7000.

Alan Warn, a data center manager at ABN AMRO who I spoke to last week for a story on CFD modeling in the data center, also had some things to say about Cisco switches and EMC storage devices.

Neither are new issues for data center facility managers, but they bear repeating. A popular Cisco LAN switch, the Catalyst 6500, blows its hot air out the sides, wreaking havoc on facility managers who are trying to arrange a hot-aisle, cold-aisle configuration in their data center.

“Cisco switches are getting harder and harder to cool,” Warn said. “Their processing equipment, the cooling runs side-to-side. It’s just madness.”

Last year, we asked Doug Gourlay, Cisco’s senior director of data center solutions, about that exact issue in a Q&A we had with him. His answer? The Catalyst 6500 is actually offered in a front-to-back airflow configuration as well, but many data centers don’t buy it because you can’t fit as many ports into it as in the side-to-side version. The bottom line is that if you buy Cisco switches, you have sacrifice either proper airflow or port density, neither of which is very attractive.

Warn said he is looking at some APC cabinets that have special fans to take air from the sides and force it out the back.

Terry Rodgers, an associate partner at data center consultancy Syska Hennessy, recently wrote to us regarding our story on computational fluid dynamics (CFD) modeling. He said that CFD modeling is too expensive to be used by data center managers simply to tell them that they should be following best practices on reducing bypass airflow.

“Why pay tens of thousands of dollars to find out that yes, you should follow accepted best practices?” he wrote. “CFD is not necessary to identify and seal cable cutouts, holes in the data center perimeter, use blanking panels in racks, and remove perf tiles from the hot aisle.”

He added that CFD isn’t even necessary to finding hot spots in the data center. “Just walk around and check and take a temperature sensor with you.”

The better strategy, he argued, is to correct all of those airflow issues that you can, and then use CFD modeling to help you rearrange server cabinets, for example, or determine whether your air conditioning units are providing the airflow pressure that they should.

“Another use for CFD is to validate design strategies before proceeding with costly construction,” he wrote. “But to tell me to seal cable cutouts???”

We agree. A CFD analysis can cost a pretty penny and should be used primarily to pick up the more nuanced cooling issues in your data center, the ones not easily seen with the naked eye. When half of your floor tile is gone to get some cables through it, it doesn’t take a rocket scientist analyzing complex algorithms to determine that you’ve got to fill them up. Thanks to Rodgers for writing in.

Since yesterday was Earth Day, I’m sure all you data center managers were out hugging trees, riding your bicycles to work and watching An Inconvenient Truth. But that was yesterday. Today, reality in the data center hits again. And the reality is that many of you aren’t that concerned with data center energy efficiency. I know, it’s a downer. But there are some hints of a silver lining if you can hold on a bit.

First, the reality: 37% are not in the middle of a data center energy efficiency project and have no plans for one. Perhaps some don’t realize that a server consolidation and virtualization project is an energy efficiency project, so that could be skewing the numbers somewhat. Pretty much everyone nowadays is doing virtualization and consolidation.

Second, almost half are getting their information about data center energy efficiency from vendors, either on the IT or infrastructure side. These vendors have one primary goal — to sell their equipment. If making their server or UPS 1/10% more efficient can help sell it, they’ll do it. Meanwhile, data center managers might be able to fill up cable cutouts and put in blanking panels for pennies on the dollar compared to what these vendors are selling, and save more energy.

“I’m not sure that’s surprising, but it’s telling,” Jay Fry, the VP of marketing for Cassatt, said. “They’re getting their data from interested parties, well-entrenched vendors who have a stake in incremental change, but not in a more innovative and potentially market-disruptive way.”

Cassatt, which provided these numbers from its own survey of users, is a server power management software company. And yes, Fry is a marketing guy. They’re a vendor just like the others, and Fry’s job is to sell, just like the other vendors’ marketing guys. So what is Cassatt pushing? Their software allows data center managers to power up and power down servers according to scheduled demand, thereby saving companies power costs they don’t need to spend when their servers aren’t being used. But not everything in Cassatt’s own survey is so rosy for them either.

In the survey, more than 41% said they cannot justify turning their servers off. Period, end of story. Sorry Cassatt, you’re not welcome here. Ouch.

But Fry said that was actually a lot lower than he thought; he was expecting that number to be about 90%. That’s what it has been in past years, he has found, so at least some people are thinking about changing.

“There’s still a lot of work to do,” he said, “but it does seem like people are thinking about ways to do things.”

There are some good things from this survey. About 40% said that either their primary or secondary reason for doing an energy efficiency project is “environmental responsibility.” See, so there are some tree-huggers out there. Also, in a response that seems to contradict the one about shutting off servers, almost half of respondents said they would be comfortable with an automated solution that “power-controlled” at least some of their production servers. Most likely they think that “power-controlled” means something less than shutting them off.

Finally, in a response that reflects results from a recent Uptime Institute survey, it seems that the IT-facilities gap is shrinking. About 54% said there is no gap or a small one. This result probably doesn’t have a direct effect on Mother Earth, but overall if communication between facilities and IT is better, power in the data center can be better monitored, thereby keeping power in check, thereby reducing strain on coal-fired power plants, thereby reducing their output, thereby making the shrinking gap an environmental concept. It took a while to connect those dots, but yes, we did it.