Yesterday I went to an Aperture-sponsored event in downtown Chicago that Andrew Fanara from the federal Environmental Protection Agency spoke at. Much of it was information that he has spoken about before and that we’ve reported, all of it around the data center energy efficiency issue that the EPA has gotten more involved with in the past couple years.

A major focus of the event was measurement. Leaders in the industry say that data centers must learn to measure how much power they’re consuming in order to reduce it. Then they can have before-and-after accounts of their Power Usage Effectiveness number, which is an efficiency metric dividing your total facility load by the IT load.

Your PUE number is like golf — the closer to 1, the better. At least that has always been the common wisdom. The goal, says experts, is to reduce your PUE. But sometimes an IT energy efficiency project can play games with that number.

Steve Yellen from Aperture said a virtualization project can temporarily hurt your PUE number. Take this example: You have 10 megawatts coming into a facility, and 5 of them are taken up by the IT load. You virtualize and consolidate servers, thereby reducing your server footprint, and thereby reducing your IT load. So now your IT load is only 4 megawatts even though your facility load is still 10 megawatts. So your PUE would go from 2 to 2.5.

Presumably there would be an adjustment. You would see that the IT load had decreased, and so you would adjust your facility load accordingly. According to Yellen, everything would be hunky dory again, right? Wrong. Your PUE would still take a hit. Let’s take the same example:

• Your facility load is 10 megawatts and your IT load is 5 megawatts, so your PUE is 2.
• You virtualize and consolidate so that your IT load becomes 4 megawatts, a one-megawatt reduction. Your PUE is now 2.5. Uh-oh.
• So you adjust, reducing your facility load by one megawatt to match with the IT load reduction. So now your facility load is 9 megawatts while your IT load is 4 megawatts. Your PUE is now 2.25, which is still worse than the PUE of 2 you had before you virtualized and consolidated. Still uh-oh.

In fact, the more energy you save with your virtualization/consolidation project, the worse it could be for your PUE. Say your project reduced your IT load by two megawatts instead of one. So you reduce your facility load by two megawatts as well. That means the facility load is 8 megawatts and the IT load is 3 megawatts, yielding a PUE of 2.67. Uh-oh.

Taking it a step further, any project that improves your IT load alone will yield a worse PUE. If you buy those new super-duper efficient servers, that could make your PUE worse. If you install blanking panels and move perf tiles around the right way, that will improve your PUE.

Let’s not panic here, because there is a good side to this. If I consolidate servers, I have fewer servers to cool. That presumably means that I’ll be able to reduce my facility load further because I might be able to shut down one of the cooling units. And maybe fewer servers means I can get rid of one of my uninterruptible power systems (UPS) units. In the end, it might all even out, but it may just leave you with a zero-sum game instead of an improved PUE number, which is what you think it would do.

In the end, what’s most important is reducing your overall power load, and if you can document how it all happened, all the better.

 Trends in data center power was the topic at AFCOM New England Chapter’s meeting this week, and apparently it’s a subject that resonates with members—at least judging by the nearly 100 attendees who showed up. (As the New England chapter enters its third year, President Rocko Graziano, whose real job is manager of infrastructure operations and services at L.L. Bean, said this was the largest meeting yet). Two speakers gave the audience their take on the some emerging trends they see taking shape.

Rudy Kraus, CEO of Validus DC Systems, a provider of direct current (DC) power infrastructure for data centers and telecommunications facilities, naturally sees a bright future for data centers powered by DC rather than AC-based electricity. Kraus cited a number of statistics from the likes of the Uptime Institute and McKinsey outlining just how much power data centers can save by switching to more efficient DC power. If data centers in the United States converted only 10% of their capacity to DC power, that would eliminate $1 billion in electric bills. The co2 emissions for a 10 megawatt data center with 17,500 servers would drop from 99,776,400 pounds to 59,865,840 pounds. Kraus invited members of the audience to do their own comparison by visiting an online calculator offered by Intel that analyzes facility-level efficiency of AC and DC servers.

The other speaker, Brian Ouellette, of J.S. Fleming Associates, a provider of power and cooling systems, spoke about the five power trends heading to a data center near you. The top trend, that energy efficiency is gaining importance, is pretty self-evident. The other four trends centered on ways to make data centers more efficient: New ways to scale UPS architectures into adaptive models that can adjust to changing power requirements; two-stage power distribution that reduces restrictions to cooling air flow, among other benefits; increasing use of monitoring with tools such as smart power strips (that monitor in-rack power) and branch circuit monitoring (that monitor each PDU output circuit). Ouellette also pointed out that data centers don’t have to go high-tech in order to become more efficient. When Ouellette asked the audience whether they use blanking panels in their data centers, only five people raised their hands. “Blanking panels are a great way to get the air where you need it,” he said. “Otherwise, you’ll get cross-contamination of air from your hot aisle and cold aisle.”

The Environmental Protection Agency’s Energy Star program is trying to collect information from anyone with a data center to help the EPA get a sense of what’s happening nationally with data center energy consumption. In this video from the Uptime Institute Symposium, the EPA’s Andrew Fanara discusses how data center managers can participate in the program, by measuring their energy efficiency in a standardized way.

You can download the forms for the National Data Center Energy Efficiency Information Program from the EPA’s data center Web site.

Last week at the Uptime Institute Symposium, I met with Ken Ostereich of Cassatt and we chatted about the need for a radical change in data center design and operations in order to achieve the Uptime Institute’s challenge for end users to reduce energy consumption in the data center by 50% over the next 36 months. Ostereich observed that the proposed solutions are so incremental, a 50% reduction isn’t likely.

Christian Belady, Principal Power and Cooling Architect at Microsoft, actually compared the level of innovation to the movie Groundhog Day, 1993 Bill Murray classic where he has to repeat the same day over and over again.

In the video below, Ostereich calls on the data center industry giants like Yahoo, Google and Microsoft to serve as a “Data Center Space Program” helping best practices and new technologies trickle down into the more traditional companies. Ostereich has an excellent rundown of the week’s events on his blog, Fountainhead.

While the pace of change can be painfully slow, I argue that the innovations Ostereich and Belady are calling for are already happening.

Microsoft is exposing its internal operations to the public, and while they might not be giving away the secret sauce, if you’re paying attention you can still see the shape of what they’ve got under the sheet. They’re proving it can be done.

I too have heard the cries for metrics, hot-ailse/cold-aisle and virtualization ad nauseum at every conference over the past two years. But end user panelists at companies like Ford Motors and Boeing are talking about using cloud computing for non-critical applications and shutting down servers during the off hours.

Granted, many companies are putting blanking panels in their racks, virtualizing a few servers and calling it the best they can do, but those companies likely won’t be running data centers at all in five years. Google and Microsoft will be running their IT for them.

This week I sat down with Microsoft’s Senior Director of Data Center services, Michael Manos at the 2008 Uptime Symposium to talk about Scry, the company’s data center analytics tool that they touted at AFCOM. According to Manos, this tool allows him to look at his data centers’ energy use, carbon footprint, power bill and more — all at an incredibly granular level. It also allows him to slice and dice data to make decisions, for example Microsoft can look at the energy consumption of an individual product like Hotmail. The program is especially slick in that it ties into Microsoft’s CMDB and assett management tools. Microsoft has been touting this tool at various conferences throughout the past few months, but it’s not likely to become a commercial product for other companies since so much of the tool was built around Microsoft’s specific homegrown internal software. But the main point of Manos’s data center road show is to prove to people that it can be done. Microsoft is measuring and improving its energy efficiency in the data center and Manos is not waiting for someone to hand down the perfect metric or the perfect tool. Check out the video below, where Manos outlines how he uses Scry and in the second video he talks you through a 3-D rendering of the new containerized data center being built in Chicago.

Since yesterday was Earth Day, I’m sure all you data center managers were out hugging trees, riding your bicycles to work and watching An Inconvenient Truth. But that was yesterday. Today, reality in the data center hits again. And the reality is that many of you aren’t that concerned with data center energy efficiency. I know, it’s a downer. But there are some hints of a silver lining if you can hold on a bit.

First, the reality: 37% are not in the middle of a data center energy efficiency project and have no plans for one. Perhaps some don’t realize that a server consolidation and virtualization project is an energy efficiency project, so that could be skewing the numbers somewhat. Pretty much everyone nowadays is doing virtualization and consolidation.

Second, almost half are getting their information about data center energy efficiency from vendors, either on the IT or infrastructure side. These vendors have one primary goal — to sell their equipment. If making their server or UPS 1/10% more efficient can help sell it, they’ll do it. Meanwhile, data center managers might be able to fill up cable cutouts and put in blanking panels for pennies on the dollar compared to what these vendors are selling, and save more energy.

“I’m not sure that’s surprising, but it’s telling,” Jay Fry, the VP of marketing for Cassatt, said. “They’re getting their data from interested parties, well-entrenched vendors who have a stake in incremental change, but not in a more innovative and potentially market-disruptive way.”

Cassatt, which provided these numbers from its own survey of users, is a server power management software company. And yes, Fry is a marketing guy. They’re a vendor just like the others, and Fry’s job is to sell, just like the other vendors’ marketing guys. So what is Cassatt pushing? Their software allows data center managers to power up and power down servers according to scheduled demand, thereby saving companies power costs they don’t need to spend when their servers aren’t being used. But not everything in Cassatt’s own survey is so rosy for them either.

In the survey, more than 41% said they cannot justify turning their servers off. Period, end of story. Sorry Cassatt, you’re not welcome here. Ouch.

But Fry said that was actually a lot lower than he thought; he was expecting that number to be about 90%. That’s what it has been in past years, he has found, so at least some people are thinking about changing.

“There’s still a lot of work to do,” he said, “but it does seem like people are thinking about ways to do things.”

There are some good things from this survey. About 40% said that either their primary or secondary reason for doing an energy efficiency project is “environmental responsibility.” See, so there are some tree-huggers out there. Also, in a response that seems to contradict the one about shutting off servers, almost half of respondents said they would be comfortable with an automated solution that “power-controlled” at least some of their production servers. Most likely they think that “power-controlled” means something less than shutting them off.

Finally, in a response that reflects results from a recent Uptime Institute survey, it seems that the IT-facilities gap is shrinking. About 54% said there is no gap or a small one. This result probably doesn’t have a direct effect on Mother Earth, but overall if communication between facilities and IT is better, power in the data center can be better monitored, thereby keeping power in check, thereby reducing strain on coal-fired power plants, thereby reducing their output, thereby making the shrinking gap an environmental concept. It took a while to connect those dots, but yes, we did it.

Last year we wrote about the IT-facilities gap. What is it? A lack of communication between IT workers and facilities employees, which can often lead to mismatched timelines that cause a data center to run out of space and power while servers are still waiting on the loading dock.

Oftentimes it comes down to the refresh rate. The IT refresh rate is usually only a few years — that’s when servers start being replaced. In larger organizations, new servers are coming in all the time, sometimes dozens or hundreds every day. Meanwhile, it can take years to plan, permit and build a new data center facility. And so data center capacity planning becomes key.

According to a user survey this year by The Uptime Institute, relations between IT and facilities folks seem to be improving. More than 57% of respondents said communication between IT and facilities were good or excellent, a number that Institute founder and executive director Ken Brill didn’t foresee.

“These results paint a rosier than expected picture of IT/Facilities relations,” Brill wrote in a report on energy efficiency strategies. “It may be that businesses are being forced by the changing economics of the data center to forge more productive relationships between these departments.”

But what kind of data centers responded to this survey? About one-third had data center floor space exceeding 30,000 square feet. So many of them are large organizations that may be more advanced in their data center development. Where the IT-facilities gap probably exists just as strongly is in smaller companies where the facilities guy focuses on the entire company and all its office buildings, and not just the data center.

Brill realizes there is still a ways to go, writing that “it is clear a large number of organizations still need to build bridges between the Facilities and IT departments in order to approach data center energy consumption holistically.”

Northern California utility Pacific Gas and Electric Co. has made headlines by finding novel ways to offer rebates for energy efficiency in the data center. Last week PG&E hosted utility companies from all over the country to brainstorm ideas for new rebate programs and other methods to curtail IT energy consumption.

According to the PG&E Website, the Utility IT Energy Efficiency Coalition comprises over 24 utilities from across the US and Canada that are primed to address the high tech, data center, and IT infrastructure markets. California utilities are well represented, with Southern California Edison, San Diego Gas and Electric, the Sacramento Municipal Utility District, City of Palo Alto, and Los Angeles Department of Water and Power attending the meeting. Utilities from the Pacific Northwest, Texas, New York, and Canada also participated.

We talked to PG&E’s Mark Bramfitt, supervisor of the customer energy efficiency program for the high tech market about the programs that they’ve had in place so far. Surprisingly, the server virtualization program hasn’t been quite as sucessful as I would have imagined. “The uptake on the virtualization program is a little slower than we wanted,” Bramiftt said. “We’ve received 60 applications in so far.” PG&E has had a number of calls from customers that say “we just did it”. But unfortunately, customers need to come to PG&E before a virtualization project to get the funding. Bramfitt is working on offering a rebate to customers so they can get incentives after the fact.

On the upside, PG&E is seeing a lot of success with its new construction incentives. The Bay Area is currently experiencing a data center construction boom. And Bramfitt is convincing many of them to install air-side and water-side economizers. For more on economizers, check out our case study on United Parcel Service.

“We’re about to pay a customer a $1.4 million incentive [for economizers],” Bramfitt said. “It’s going to be one of the biggest checks we’ve ever written for a new construction.”

The people I speak with about virtualization projects always list the same reasons for going virtual; they don’t have enough space in their data center to add more physical servers, they can’t afford power and cooling bills, they want to consolidate physical machines, and they want to consolidate physical people.

That’s right; the majority of people I speak with - employers and employees alike - say nonchalantly that they deploy virtual machines to avoid deploying more IT staff. While this is great for corporations, it doesn’t sound so good for IT job seekers.

A few examples; I went to a VMware Inc. User Group meeting in Boston on March 27, and one user gave a presentation about the virtualization project he oversaw at the paper manufacturing company called SAPPi in Maine.

“One reason we wanted to virtualize is we needed to lower our IT headcount. We needed to get rid of high end support and just keep desktop support,” the systems engineer/presenter said.

Similarly, at the growing law firm Owen Bird Law Corp. in Vancouver, British Columbia, Stephen Bakerman, the sole IT staffer, went with Virtual Iron virtualization to avoid adding more physical servers and having to hire more staff to help him manage it all.

“The cost savings is probably $100,000, and the time savings for me are incredible. Once everything is virtualized, I can run everything from my desktop remotely from my office or at home. I don’t have to hire someone else, and I would have if we kept adding servers,” Bakerman said.

Another company called QualComm Inc virtualized 60% of its data center environment and saw a similar side effect. At the VMware Virtualization Seminar Series in Providence, RI Feb 26, VMware presented a case study of the wireless technology company showing how it started with 1,200 servers and consolidated down to 100 (12:1 ratio) physical servers, increasing data center space and cutting back on power and cooling. That’s great. And the cherry on top? They have not had to increase their IT staff at all in 2.5 years.

Sure, I get how cool virtualization is, and the benefits it brings from a savings and management stand-point, but is anyone else concerned those IT college kids who dream of days spent engineering systems won’t be able to find a job? or is anyone worried about those system administrators who might get consolidated from many to few along with their servers?

I’m interested in hearing from IT folks; is virtualization leading to a virtual job market?

The federal Environmental Protection Agency, which is in the midst of developing an Energy Star rating for servers, also has its sights set on creating one for data center facilities.

The Energy Star program is a familiar one now to American households — they’re the yellow stickers on televisions, washing machines, ceiling fans and dozens of other appliances that show how energy efficient they are. But a lesser known program is the Energy Star’s rating system for commercial buildings. And now the EPA wants to bring that system to computer rooms in a program it is calling the Data Center Infrastructure Rating.

To develop it, the EPA must collect 12 months of data on the energy use of existing data centers, and it’s looking to the industry for help. It is asking organizations to monitor and submit details about their data center energy use starting in the beginning of June and lasting until June of 2009. All the companies participating will be kept anonymous for the purpose of the study, and will be required to submit energy use data to the EPA on a quarterly basis from an Excel spreadsheet.

All of this data center energy use data the EPA collects will then go into its formation of the Energy Star data center protocol next year. For more information, visit the Energy Star’s data center site.