Last week we asked readers to offer up their best duct-tape data center fix-it stories, and the response was great. Data center managers are the real MacGyvers of IT. Here are some of the stories:
About four years ago, my small data center was located on the top floor of an office building. After we had been there for about four years, we noticed that the roof had developed a leak when some ceiling tiles started getting wet. At first, we thought it was possibly some condensation from the A/C unit that we used to keep the equipment cool and dry, but we could not find the problem. So, we called building management. Without telling us, they sent someone up on the roof with a water hose to see if they could find a leak. They did.
Fortunately, the new ceiling tile soaked up that water before it could run onto the equipment. Supposedly, they fixed it, but after another wet ceiling tile during the next rainstorm, we invested in some very nice bright blue tarps that we installed over the server racks to make sure that any other leaks were diverted from doing an damage. That turned out to be a really good investment since the leak got worse and we moved to another building before it was repaired. The data center tarps were formally listed as one of our risk mitigations in our Sarbanes-Oxley reporting that year.
DP
My example (seen at a client’s site) is an emergency drip tray made from plastic milk bottles to catch the run-off from an air-conditioning unit. The A/C unit had been installed and placed over a fully populated 42u rack by non-specialists. On a very humid night the base of the A/C unit started dripping and the run-off had nowhere to go but into the server rack. To keep the IT services running, the night shift put together a gutter made from plastic bottles to carry the fluid away from the rack and into a bucket.
CR
A number of years ago, at a previous employer, there was an electrical fire above the ceiling tiles in a server room. The room, in the past, was a large mainframe and DASD room. Since the company had been acquired by another company the mainframes and DASD were removed. The large room was relegated to a half dozen racks, of servers and network gear, for the existing building business units.Apparently the fire smoldered for quit a while and thanks to a closing shift security guard a major disaster was avoided. At that time there were no smoke senors “above” the ceiling tiles. Minor water damage, from the fire department, to unoccupied areas of the room was the result.
As part of the decommission of the mainframes and disks a decision was made to remove the Halon fire suppression systems. Water suppression was installed. However the server administrators were concerned about “water damage” if the system should be set off or happen leak. So they commissioned the Facilities team to install large plexiglass panels to be hung above the equipment racks, like an indoor roof. The panels were hung at an angle that allowed the water to run off on to the floor behind the racks.
So now they have a fire suppression system that will not suppress a fire that may begin in any of the equipment racks. That is of course until the flames get big enough and melt the plexiglass.
JJ
When a broken pipe led to opening all the raised floors in the data center, we discovered vermouth, gin, olives and very cold martini glasses.
CN
At our off-site location, we had a flaky NIC card connector in one of our main servers. It wasn’t the cable, we tried a number of them. The connector was loose, and would only work if the cable connector was pushed up a certain way.We didn’t have a spare, and since that data center is locked down, and you can’t bring in any equipment or parts that aren’t pre-authorized, we were stuck. We ended up having to fold up some stiff cardboard that fit perfectly between the server case and network cable to keep pressure on the connection. Worked like a champ until we could return and replace.
I would love to have photographed it, but taking pictures of shared cages is a no-no. Oh, and it was my boss that installed the solution.
DH
When I was at Sprint one of my data centers had “rolls” of plastic sheeting next to each server complex. What for? We were located in a basement right under a restaurant. It had frequent water overflow problems so in an emergency we would cover the servers with plastic until the water stopped dripping. I put together three business cases to move the data center, but never got funded.
RH
Funny thing happened as I was reading your email about Duct Tape Data Centers… I heard a shout out for duct tape and got the attached pictures as our IT Manager patched a torn link to our fix for a server room with inadequate AC.
WS
We could also call this “green computing”. I have been with a small software development shop for a couple of years now. My boss is one who believes that HVAC comes with the building, no matter how many servers we pack into an 8 x 10 room that is our “data center”.Since I’ve been with the company, we’ve managed to destroy four wall units that cool the server room, and prior to my arrival, I’m told that two more had been worked to death.The building owner put his foot down that he would provide no more wall A/C units and went to the trouble to remove the unit from the server room wall, leaving a hole in the wall approx 4′ by 2′. My boss also chose not to back down, and has not provided his own cooling unit.
We are located in suburban Chicago, so needless to say that since the A/C removal and the arrival of winter, it has become quite cold in that server room. My Dell servers now go into alarm as their internal temps drop to 4 degrees centigrade.
I was able to ask for and get a blanket (actually a used painter’s tarp) to cover the missing A/C unit hole in the wall - prior to that, I stuffed it with shipping foam and newspapers (what we call the “homeless mode”) of temperature regulation.Today was a warm day in Chicago - up to the mid 40’s, so I had to remove the tarp. I don’t know what’s coming, but spring is going to be interesting.
BF
At a hospital data center I managed many years ago (no photos, sorry) we had a roof that leaked directly on a large Tandem system. After repeated calls, attempts by building engineering to patch the leak, and a few very close calls with water dripping on the systems, the building engineers installed the obvious high tech solution; a bucket sitting on top of the ceiling tiles directly under the leak.Ahh, I see you thinking this through… what if the weight of the water in the bucket exceeded the strength of the ceiling tile? Disaster awaits!? No problem! Our brilliant engineering team cut a hole in the bucket not far from the bottom, taped a plastic hose from the bucket down through the ceiling tile into an “overflow” bucket on the raised floor behind the Tandem system. This worked much like the overflow tank for the coolant in your radiator. Then, when we saw water accumulate in the bucket on the raised floor, we’d empty both buckets and reconfigure for the next rainstorm.A $5 engineering marvel that protected our $1,000,000 system for well over a year before the roof repairs were properly executed.
PB
Have a duct-tape data center horror story? Tell your tale of woe or heroism in the comments.