We write a lot about optimizing data center cooling — engineering advice from thermodynamics masterminds with perfect data centers. But what about the worst data center cooling you’ve ever seen? We asked readers to send us their duct-taped data center cooling horror stories. This week we’re publishing responses in the data center blog.

The neighbors froze our data center

Our office is underneath a laboratory where they manufacture cryogenic equipment housing liquid nitrogen. On May 24, 2006, one of the pipes in the lab above us burst and there was so much condensation in our storage facility that it took the disaster recovery unit three days to scrape off chunks of xenon tetrachloride residue from our rack mounted servers. Our downtime was over two weeks and we lost 13 terabytes of backup data. Fortunately our original data was able to be used to backup the backups. Our attorneys are still involved in the insurance claim!

Since then, we have switched to HP blade servers and virtualized storage at a remote location. I don’t know a horror story worse than ours, but I am anxious to read some of the others.

S.F.

CRAC Band-Aid: It’s ugly but it works

Our data center had become so crammed with servers that the two CRAC’s were unable to maintain temperature. When one went down we had to open all of the doors to try and keep the servers from overheating. The data center was also expanded without regard for cooling and power (another nightmare issue with only two PDU’s). The new size really needed an additional unit or two but none was purchased. The attached photos are the Plant Engineering department’s solution to our cooling problems. Ugly as it is the solution worked, however we are embarrassed to show the space to anyone.

We just completed construction of a brand new data center building so we are abandoning this space for that purpose, thankfully. The new space has six CRAC’s with redundant power sources and chillers each capable of maintaining temperature for the space. In addition the power problems have been solved with six PDU’s with lots of room for growth.

M.R.

Mini-data center meltdown

The worst cooling I ever had to deal with was in a tiny little company that moved all three of its servers, plus communications equipment, to the last empty office upstairs — with no vents and no cooling. All I had was a single desk fan to keep the air moving around in the hope that some of it would escape under the door. I couldn’t even open the window — it faced south, and the merest zephyr of a breeze would blow the vertical blinds all over the place, allowing the sun to fall directly on the sides of the servers. They were IBM servers, too — very black, very capable of absorbing radiated heat.

Then I discovered that the ceiling tiles could be removed and the heat could all just vent into the roof space.

M.M.

This old data center

A few years back we were moving to a “new” office building. We rented the whole floor in a turn of the century (20th, not 21st) building. The whole floor was gutted, right down to bare cement floors, walls and ceiling and refitted with modern materials. That included new HVAC throughout. On the blueprints two rooms were designated specifically for computers. Our computing needs weren’t all that big, but we did have a huge legacy (ie 1960’s, 4ft wide, 2ft deep, 6ft high) power conditioner.

The first sign of problems was the minor issue of 20 missing workspaces (turns out the floor plan wasn’t square and planners worked from the largest measurements). The next problem was the “computer room”. Within a day of moving in, the room temperature was up around 100 deg F. The locked door had to be left propped open and an old painted over window had to be forced open.

The computer room solution was to retrofit an exhaust fan from directly outside of the building. Even with the “new” retrofitted HVAC system the whole floor had heating and cooling problems year round. In the winter just walking down the corridor between the cubicles you could walk through a 20F degree range in 10 feet, from 65F down to 45F. Yes, some of us had to wear jackets and gloves sitting at out desks!

The final solution was that we moved to a newer building at the end of the 5 year lease.

R.S.

Names have been abbreviated to protect the innocent! If you have a great cooling story, leave feedback in the comments section.