We write a lot about optimizing data center cooling — engineering advice from thermodynamics masterminds with perfect data centers. But what about the worst data center cooling you’ve ever seen? We asked readers to send us their duct-taped data center cooling horror stories. This week we’re publishing responses in the data center blog.

The neighbors froze our data center

Our office is underneath a laboratory where they manufacture cryogenic equipment housing liquid nitrogen. On May 24, 2006, one of the pipes in the lab above us burst and there was so much condensation in our storage facility that it took the disaster recovery unit three days to scrape off chunks of xenon tetrachloride residue from our rack mounted servers. Our downtime was over two weeks and we lost 13 terabytes of backup data. Fortunately our original data was able to be used to backup the backups. Our attorneys are still involved in the insurance claim!

Since then, we have switched to HP blade servers and virtualized storage at a remote location. I don’t know a horror story worse than ours, but I am anxious to read some of the others.

S.F.

CRAC Band-Aid: It’s ugly but it works

Our data center had become so crammed with servers that the two CRAC’s were unable to maintain temperature. When one went down we had to open all of the doors to try and keep the servers from overheating. The data center was also expanded without regard for cooling and power (another nightmare issue with only two PDU’s). The new size really needed an additional unit or two but none was purchased. The attached photos are the Plant Engineering department’s solution to our cooling problems. Ugly as it is the solution worked, however we are embarrassed to show the space to anyone.

We just completed construction of a brand new data center building so we are abandoning this space for that purpose, thankfully. The new space has six CRAC’s with redundant power sources and chillers each capable of maintaining temperature for the space. In addition the power problems have been solved with six PDU’s with lots of room for growth.

M.R.

Mini-data center meltdown

The worst cooling I ever had to deal with was in a tiny little company that moved all three of its servers, plus communications equipment, to the last empty office upstairs — with no vents and no cooling. All I had was a single desk fan to keep the air moving around in the hope that some of it would escape under the door. I couldn’t even open the window — it faced south, and the merest zephyr of a breeze would blow the vertical blinds all over the place, allowing the sun to fall directly on the sides of the servers. They were IBM servers, too — very black, very capable of absorbing radiated heat.

Then I discovered that the ceiling tiles could be removed and the heat could all just vent into the roof space.

M.M.

This old data center

A few years back we were moving to a “new” office building. We rented the whole floor in a turn of the century (20th, not 21st) building. The whole floor was gutted, right down to bare cement floors, walls and ceiling and refitted with modern materials. That included new HVAC throughout. On the blueprints two rooms were designated specifically for computers. Our computing needs weren’t all that big, but we did have a huge legacy (ie 1960’s, 4ft wide, 2ft deep, 6ft high) power conditioner.

The first sign of problems was the minor issue of 20 missing workspaces (turns out the floor plan wasn’t square and planners worked from the largest measurements). The next problem was the “computer room”. Within a day of moving in, the room temperature was up around 100 deg F. The locked door had to be left propped open and an old painted over window had to be forced open.

The computer room solution was to retrofit an exhaust fan from directly outside of the building. Even with the “new” retrofitted HVAC system the whole floor had heating and cooling problems year round. In the winter just walking down the corridor between the cubicles you could walk through a 20F degree range in 10 feet, from 65F down to 45F. Yes, some of us had to wear jackets and gloves sitting at out desks!

The final solution was that we moved to a newer building at the end of the 5 year lease.

R.S.

Names have been abbreviated to protect the innocent! If you have a great cooling story, leave feedback in the comments section.

Intel Corp. made an announcement today saying it is joining 40 other server technology providers in support of the new Server Systems Infrastructure (SSI) industry specification for modular server platforms. 

The group plans to ”lower the cost of product development by providing design guidance that enables server builders to develop compliant and interoperable building blocks at the blade, chassis and manageability software level.”

Microsoft is in the vendor mix, but some glaring ommissions  from the group are the industry giants; Hewlett-Packard, IBM, Sun Microsystems and Dell.

In talking to these vendors in the past, I’ve learned they aren’t keen on standardizing blades.

As of today, users who buy blades are locked into that vendor because blades from different vendors have different “ecosystems,” and blades from one vendor will not fit into another vendor’s chassis.

This is a fact that won’t likely change anytime soon, according to IBM’s world wide marketing manager of IBM’s BladeCenter Scott Tease. “There has been a push toward standardizing blade chassis, but I don’t ever see that happening,” Tease said. “I couldn’t imagine IBM getting together with Dell and HP to share chassis. It is what makes each unique.”

I think SSI has an uphill battle when it comes to standardization.

AlarmPointtoday announced an upgrade of its IT event notification software.Version 3.1 has more functionality with Blackberry devices and Voice overIP telephones through partnerships with Research in Motion and Avaya, respectively.

“The primary driver behind 3.1 was expanding communications and focusing on enterprise communications,” AlarmPoint CEO Troy McAlpin said.

The company claims it is also now easier to import and export relevant data to and from AlarmPoint for better integration into company’s customized applications.

The company has about 770 customers. The new version, available today, has three different price ranges: $5,000-$15,000; $25,000-$50,000; and $75,000 up.

The data center hosting facility 365 Main in San Fransisco lost power last Tuesday and its generators failed to back up, causing downtime for the companies who bank on promises of uptime there.

After the outage, 365 Main had experts working non-stop to figure out why the generators didn’t fire up and found diesel engine synchronization failure and PDU issues. Hitec and 365 Main and trying to figure out the root cause of the failures.

It’s unfortunate that companies like Craigslist who pay for hosting services at 365 Main experienced downtime there. I’m sure the piece of mind they had disappeared when their data centers went dark last week.

At least 365 Main is going to route of full disclosure, continually updating the public on this issue on its website.

A San Francisco power outage caused a number of Web 2.0 companies to go quiet when it affected one of 365 Main’s data centers. Web 2.0 companies like Craigslist, Technorati, Yelp, TypePad and LiveJournal all got hit, according to Artur Bergman at O’Reilly.

You may remember when MySpace shut down last year because of the heat wave in the Los Angeles area.

However, power outages in and of themselves shouldn’t shut down data centers like 365 Main or MySpace. They’re supposed to have rock-hard reliability and redundancy built in. In the MySpace case, backup generators failed.

Meanwhile, 365 Main said their generators also failed to start when the power surge hit. It added that the data center was without power for about 45 minutes. The company has a good question-and-answer portion of the update where it reveals details such as that up to 40 percent of its customers were affected by the surge.

That can be one of the risks of outsourcing your data center: There’s a certain lack of control you have over your own IT equipment. Although if you were to build a data center with the redundancy that a place like 365 Main has, it would be quite a pricey venture.

dbaDirect, which provides managed services for companies’ database software, has now added support for DB2 on the z/OS mainframe.

The Cincinnati-based company, which has 102 customers, also supports Oracle, Sybase, SQL Server, MySQL and even DB2 for distributed environments. Now it says customer demand is causing them to support DB2 on z/OS.

“We do a number of customer surveys throughout the year and they began to express concern about having skills in their organization to do high-level operational support for z/OS DB2,” said Chief Technical Officer MarkVorholt. “What we hear a lot is that the mainframe isn’t fading away but the people are starting to.”

So how does support for z/OS DB2 differ from distributed DB2 support? According to Vorholt, it’s all about degrees. With distributed, dbaDirect typically has more freedom to go in and do close monitoring of the database system. On z/OS, most companies already have some monitoring in place, and sodbaDirect’s goal is to be able to process and handle that information for the customer.

And according to Vorholt, it won’t start putting in-house staff out of a job.

“In the vast majority of cases we go in as a supplement,” he said. “Most customers are doing 24/7 monitoring. They want to take staff and put them on new projects and we free them up to do that. We give them freedom to put someone else on a project.”

If you are upgrading to multi-core chips, you may need to drop some extra cash on your Ethernet to get the performance you expect. 

Intel Corp. introduced this week two new Ethernet controllers suited for the high traffic flow of multicore processor systems and virtualized datacenters.

The Intel 82598 and Intel 82575 Ethernet controllers distribute workloads across all available processing cores to achieve optimal system performance.

The Intel 82598 10 Gigabit Ethernet Controller addresses the networking bottlenecks associated with server consolidation and demanding enterprise applications as in high performance computing, Intel reports.

The Intel 82575 Gigabit Ethernet Controller is available now, and the Intel 82598 10 Gigabit Ethernet Controller will be available for volume shipment in September. More information is available at Intel.

The federal Environmental Protection Agency’s Energy Star program released a framework today for a server energy efficiency benchmark.

The two memos look very preliminary in terms of details but the EPA hopes it’s a starting point toward having Energy Star ratings for servers just as it has them for desktop computers, clothes dryers and ceiling fans.

The memo describes three “building blocks” for developing the standard:

• Definition - to spell out what is covered by the new Energy Star specification.
• Eligible Product Categories - to divide the general server category into subcategories of servers; for example, 1U rack servers vs. mainframes.
• Energy Efficiency Criteria and Test Procedures - to craft a process that determines how energy efficient a server is. The EPA is suggesting a breakdown into two different “tiers” of testing, one that goes into effect immediately and another down the road. The first tier could include criteria such as the efficiency of the server’s power supply and examining the computer’s power management features. The second tier would look at the entire power consumption of a server and might be based off work that the Standard Performance Evaluation Corp. is doing to develop a server efficiency standard.

The EPA is also finishing up a study to Congress on the power consumption of data centers and suggestions for improvement. It was initially scheduled to be done by the end of June but is now scheduled to be released this month. There is more information on both of these initiatives at the EPA’s Enterprise Server and Data Center Energy Efficiency Initiatives Web site.

Timothy Sipples writes that in certain situations, software licensing charges for zVSE 4.1 on the mainframe will actually drop. The operating system, which stands for Virtual Storage Extended, is less common than z/OS, the mainframe’s major operating system. It tends to run on smaller versions of Big Iron but has been around for more than 40 years.

We wrote about zVSE 4.1 in March when it first became generally available. We quoted Pete Clark of CPR Systems, which provides education on mainframe operating systems, as saying that some customers might be able to get cheaper licensing costs under particular schemes.

The sub-capacity option allows mainframers to pay for the MIPS that they use on z/VSE, rather than being charged for how many total MIPS are on the box. Sipples really spells out those different schemes in his post.

Basically, by using the sub-capacity pricing option on zVSE, which is new with version 4.1, mainframe users can reduce their costs by staying within certain MIPS ceilings. You can dig into it yourself by checking out IBM’s Midrange Workload Licence Charges.

…if you buy a B01 and run z/VSE V4.1’s Central Functions, you can actually lower your software bill if you can live within a roughly 20 MIPS softcap. Previously the best you could do with z/VSE V3 was about 26 MIPS (z890 110 or z9 BC A01) full capacity. A 20 MIPS softcap inside a 38 MIPS machine is quite nice: you have room to grow, and softcaps allow temporary spikes above 20 MIPS.

Sipples concluded by saying that “it’s nice to see IBM offer a much better package for its smallest VSE customers.”

Glenn Keels of Dell’s server team wrote an interesting blog about the blade market and how to use them in an efficient manner in data centers.

Keels said, “We believe a “Blade Everything” philosophy is not in the best interest of our customers.  In fact, we believe that applying that philosophy could actually increase IT complexity.”

There are some good points in this blog to consider if you are thinking about using blades, like the following:

“Wholeheartedly adopting blade servers would require some customers to build entirely new data centers with greater and different power sources such as 220V to each rack which isn’t standard in many data centers, and cooling capabilities.  That makes no sense.  Customers want to build their business not new data centers. ”