Data center raised floor vs. solid debate
I just slogged my way through Douglas Alger’s 5-page excerpt from a Cisco Press White Paper purportedly discussing the merits of raised floor versus non-raised floor designs for data centers. It spends four paragraphs of the first page telling you why overhead distribution on a solid floor is not good, then rambles on for the next 4.5 pages telling you all about raised floors. It appears by that fact, and from several statements by the author sprinkled throughout the paper, that he has a strong preference for raised floor. Some of his statements about overhead infrastructure are just plain wrong, or easily mitigated. Perhaps he’s never even managed a solid floor facility? So much for a thorough analysis!
Given that I am involved in the management of two facilities, both designed at the same time, but one using raised floor and the other a solid floor with overhead infrastructure, I feel like I can present a more balanced viewpoint. I agree with most of what Mr. Alger says about raised floors, both their strengths and weaknesses. He neglects a few glaring issues with raised floors, and highlights a few of their annoyances quite well, such as tile/cabinet drift. What Alger fails to do is explore the benefits of a solid floor data center; therefore let me lay those out for you:
Floor Load
Alger is living in the past when he talks about “heavy” racks weighing 1500lbs. In today’s high-density reality, 1500lbs is a lightweight installation. The average installation we are seeing in our facilities today is 1800 lbs. We have several cabinets that exceed 3000lbs! I don’t see this trend changing any time soon. When people have 42RU to use, or to put it more bluntly, 42RU that they are paying for, they are going to stuff it with as much as they can. This is where a solid floor really shines above raised. Got a big, heavy load? Roll it on in and set it down wherever you please. No ramps to negotiate, no risk of tiles collapsing and your (very expensive) equipment falling down into a hole.
Stability
Steel reinforced concrete slabs don’t rattle, shake, shift, or break, …at least under normal circumstances. If your data center is located in an geographic region known for what I like to call “geological entertainment” your data center is likely better off with a solid floor. You can solidly secure all your infrastructure to a solid concrete slab far better than to a raised floor. The stress, shaking, and shuddering of a seismic event can displace floor tiles. The last place I want to be in an earthquake is in a raised floor data center… tiles popping, racks swaying, and the whole floor structure wobbling around underfoot does not make for a confidence-filled rollercoaster ride. I’ve been inside a solid-floor facility in a 7.1 earthquake; the overhead ladder-rack and server racks all moved in unison, creating an eerie wave, but the floor remained solid throughout, much to my relief.
Calculations of point loads and rolling loads become irrelevant, except for maybe your UPS gear if you are off the ground floor of your building.
Fire Suppression
Fire suppression technologies in today’s data center focus on isolation of smaller zones and release of a clean agent to extinguish the fire in that area. If you have a raised floor you instantly double the number of zones you must monitor, and deploy fire suppression systems into. The server spaces as well as the plenum spaces. Zone isolation is achieved through dampers in the air handling system and solid walls. These are trivial to build and secure in a solid floor facility. Air supply and return plenums and ductwork can have automatic dampers driven by the fire suppression system. Try that in a raised floor environment of any scale and it is prohibitively expensive and in some cases just flat out impossible. In the facilities I am involved with the solid floor datacenter is protected by FM-200 and Ecaro-25 fire suppression systems throughout its entirety, whereas the raised floor data center’s fire suppression is limited only to the UPS rooms.
Data center fires are unlikely, but the presence of suppression systems is a requirement for some users of data center facilities. If data centers are kept clean, dust-free, and combustible materials are kept out (almost impossible as the presence of servers is a guarantee of cardboard proliferation!) then risk of fire is low, but it can not be completely eliminated. The under foor plenum spaces are a magnet for the collection of dirt, dust, loose change, and various bits of paper, cardboard, etc. I’ve never seen a raised floor plenum space that wasn’t dirty after a year or so of installation. How many of you have seen fire suppression extended to the plenum space under the floor? What good is it to deploy in one part of the data center and not another?
Cleanliness
The above point leads directly to this one. Data centers should be very clean environments. Solid floor facilities are much easier to maintain to a very high standard of cleanliness. Raised floors are not. Periodic removal of all tiles is required to clean the plenum spaces. This not only is a messy hassle, it also reduces the effectiveness of the cooling systems during the maintenance interval, it also exposes your cabling infrastructure to risk of damage. My car always needs washing, and my wife will tell you I’m a slob, BUT my data centers are clean enough to eat off of… but don’t even THINK of bringing food or drink into one of them! I can stand in my solid floor facility and visually scan for dirt and dust with the efficiency of The Terminator. Not so with a raised floor. Unless it was installed yesterday, all manner of dirt, dust, and debris lurks beneath every raised floor used in actual production. The raised floor advocates will try to deny this, but no raised floor will pass the repeated scrutiny of a white-glove test.
Raised floors also provide a false sense of order. If a single cable is out of place, or some rat’s nest of shameful cabling lies beneath… it is hidden. No difference to the casual observer. The CEO that tours through once a year may not know whether it is the one cable or the rat’s nest, but YOU will… and YOU are the one that has to manage it. Every production facility is under constant change management, and if things go unchecked for even a little while what started as a well-ordered cable plant can turn into a rat’s nest pretty fast. Tracing cables under floor tiles is one of the biggest pains in the posterior any data center manager has dealt with. I have found that with all the infrastructure in plain sight, keeping it in order is at least easier. There are no surprises lurking when everything is in plain sight.
Density and Growth
The reality of high-density computing is that the data center must be able to support far more cable, power, and number of servers-per-rack than ever before. The days of eight 4U servers, a patch panel and maybe a few bits of 1U network hardware in a rack are long gone. Todays racks each need hundreds of cat-5 ports for multiple NICs, various storage connections, etc, room for forty-plus 1U servers, or maybe even a half-dozen blade chassis, and enough power to drive a Tesla Roadster from San Francisco to Seattle. If your raised floor was built even as recently as five years ago there likely just isn’t enough space in your plenum to handle that much cable anymore, at least not without seriously compromising your airflow. Once you build your raised floor, you are locked in to that design. You must peer far into the future and assume infrastructure needs way beyond what is expected today. With a solid floor and overhead infrastructure, you can keep adding network and power without any compromise to cooling or air flow.
At my two facilities, I work with of both raised and solid floor data centers. The raised floor one has hit the limit of what it can power and cool, based on a seven year old design, but it still has empty spaces that will remain unused, forever. The solid floor facility is currently being expanded, while still remaining on-line and operational. It will soon be capable of more than double the watts-per-square-foot its original designers planned for in the year 2000. It’ll be able to pack every rack full to 42U. The cooling system, which originally was giant air-diffusers up in a 15′ ceiling are being modified with ductwork to concentrate cold air right in front of each rack, with hot-air return plenums being routed out of the hot aisles and back into the the HVAC system on the roof. The ladder rack cable trays are not even at 20% of their capacity. This scenario is not possible with raised floor data centers, unless you can shut them down for a complete overhaul.
Access
Contrary to Mr. Alger’s claim, every solid floor data center I have worked in has had power and network terminations within reach of an average sized human being, no stepladders required. In the current solid floor facility I manage, the ladder rack is substantial enough, and the ceiling high enough to enable workers to walk on the structure itself. Ladders are only needed to ascend to it, once up you can walk around the entire facility quite safely, nine feet off the floor. The only time one needs to go up there is to install new cabling, or access the HVAC ductwork, which is rare. Working beneath the floor tiles by comparison is a miserable chore.
Having worked in both environments over the years, I’m leaning towards avoiding raised floor in the future, and sticking with solid floor facilities. To me raised floor stands as an echo of older days, when “The Data center” contained a handful of mainframes, a minicomputer or two, and men with white shirts and pocket protectors loading tapes and sitting at terminals. Entirely raised floor design just does not effectively scale to the density needs of a modern facility. I have seen hybrid facilities with raised floor plenums used solely for cooling and overhead ladder rack for power and network delivery, and that seems like a good compromise to me. But the overall benefits of a solid floor have convinced me to never look back at raised floor except as nostalgia. I suspect that I am in the minority though, as so few people have had the opportunity to experience both options first-hand. Inertia has lead people to only think of data centers in the context of raised floors.
Do you agree? Or do you think I’m wrong? Let me know in the comments.
Posted: July 10th, 2007 under Data center physical infrastructure, Data center room design and site selection.
Chuck,
I have the best of both worlds. I have a raised floor, with no cable under the floor. The only thing under the floor is cold air. Everything else is in overhead ladder racks in an open ceiling plan with little or no ductwork. This keeps the critical cabling easily visible with no worries about cold air distribution, as I only need to change the tile change air coverage to an area. We lucked into “used” 5000 pound tile for the same price as new 2000 pound tile, so we don’t have to worry about weight limits.
David
Comment by David — July 10, 2007 @ 2:46 pm
David, you forgot another advantage of the hybrid system: at least where I’m at, fire suppression is not required under the floor, as there is nothing besides the cooling to start a fire.
Also, power overhead doesn’t require an EPO switch for the facility, I believe.
Cooling underneath, and power and network overhead, could possibly be the wave of the future.
Comment by Johnny Mnemonic — July 10, 2007 @ 3:08 pm
To paraphrase Mark Twain:
The reports of the death of raised floors is premature.
Comment by Robin — July 10, 2007 @ 3:17 pm
I personally agree with your comments on raised flooring. I maintain three networks in three different cities. Two of them are slab flooring, one raised. I have also seen in the past with other places pest problems. Moving items you constantly have to make sure you are not putting too much presure on one tile or on a cross bar that holds up a tile.
Not only that, but dust and such. It collects in the perforated tiles and it is a hasle to clean them. Simple, slab floor with a tile, no ramps, no stairs, nothing….
We also had to contend with a heavy safe to put in the server room for backup tapes, etc. etc. This safe is almost as big as a Dell rack and requires 3-4 people to move it.
Comment by Dan B — July 10, 2007 @ 3:21 pm
Thanks for commenting guys!
David,
I mentioned the hybrid overhead cabling/raised floor right at the end of my article, mostly because I remembered a facility I on another floor of my building here that uses that exact approach. It is a reasonable alternative. Most of my gripes about raised floor still apply though, just not the parts about cabling and cable management. The big one for me is air volume. Your floor density will always be limited by the size of your plenum.
Of course I know of one facility here in my region where they built a raised floor 14 FEET high to deal with that very issue. I haven’t seen it yet, but I imagine the ramp alone is impressive.
“Johnny”
You are correct, our overhead power facility does not require an EPO. Thanks for remembering that. I should have as it does allow me to sleep better at night.
Robin,
I never said it was dead, far from it. Raised floor seems to be pervasive in the datacenter business. What I postulated however is that it is due to inertia, rather than clear thinking. People who build datacenters assume they will have raised floors without really performing any serious analysis or honest comparison. In my case, I inherited my facilities rather than built them, so my analysis above is based on usage rather than design.
Dan,
I didn’t even consider “unauthorized personnel” in the form of rodents or insects. Thankfully I’ve never had to deal with that issue! Thanks.
–chuck goolsbee
Comment by cgoolsbee — July 10, 2007 @ 4:33 pm
Chuck, I was the one asking if raised floor was dead. I figured that would stir up some response. Looks like it worked. ALSO: Last week we ran a Q&A with colocation company Equinix in which the cheif business officer said no more raised floors.
Comment by Matt Stansberry — July 10, 2007 @ 4:52 pm
Matt! My article in no way suggested that raised floor was “dead”… (a statement only the likes of John Dvorak would make.) I was just saying there are alternatives to the old way of thinking.
–chuck
Comment by cgoolsbee — July 10, 2007 @ 5:07 pm
I agree wholeheartedly with your points. I too have had experience with both raised floor designs and overhead designs. If a client asks me my recommendations (rather than telling me what they would like)I always recommend overhead air distribution. I give the same reasons you’ve outlined - but at the top of my list is another important one that you did not address.
For overhead air distribution, the temperature profiles within the cold aisles are more likely to fall within the ASHRAE Thermal Guidelines of 68 deg F and 77 deg F. To state in the most simplistic terms, the servers and compute devices are better served by overhead air distribution. With overhead air distribution, there will be fewer hot spots and fewer areas that are overcooled. (Yes - there is such a thing as overcooling from the point of view of the data equipment.) From the point of view of the facility, overcooling is wasteful of energy.
There were two independent symposium papers presented at the ASHRAE conference in 2005 that addressed this very issue, and they both came to the same conclusion. For the sake of full disclosure, I should state that I was one of the presenters; Magnus Herrlin of Ancis was the other. I’d be happy to direct you to either or both of those papers.
Vali
Comment by Vali Sorell — July 10, 2007 @ 5:52 pm
Hi Chuck,
Thank you for the analysis, and interesting points to debate on before deciding to ‘have’ or ‘not have’ raised floor in a Data Center (DC). Infact, I have come across a few people referring Raised Floor (RF) as ‘False Floor’ (FF) following its counter-part ‘False-Ceiling (FC).’ The very intent of FF or FC are to hide the ugly things out of eye-sight!
There are some articles on the cold air blowing from bottom-up is far more efficient to top-down flow. While it is great that you have an opportunity to manage both a RF DC and a non RF one. But we know 2 DCs are alike, due a wide mix of IT Infrastructure, do you have an opportunity to analyse the pros and cons for the same installation?
Could you also share some details on the airflow management, ceiling heights etc? Further, is it a type when you mentioned somebody built a RF of 14Feet High? Is there any such 8th Wonder in the World!
In all the DCs I built or designed, we invariably provisioned FM200 below RF along side the server space. Of course, the zoning is a really challenge.
I am bit curious to realise that EPO is a by-product of RF. Is there any explanation why EPO is mandated in a RF environment?
Thanks again to all the people who had provided valuable insights.
Regards… Saradhi.
Comment by Saradhi Motamarri — July 10, 2007 @ 7:35 pm
Hi,
I manage a DC with the traditional raised floor setup where cool air is pushed through perforated floorboard in front of the racks. The network cabling is overhead though.
Question: How do you tackle issue of cooling and channeling hot air back to the CRAC/HVAC unit, for solid floor setup?
Rgds
Roy Ong
Comment by Roy Ong — July 10, 2007 @ 8:01 pm
I agree with the hybrid approach of using raised floor just for cold airflow. Can someone point me to articles/documents which describes how I should run overhead power through trays?
Comment by Sartha — July 10, 2007 @ 11:42 pm
Vali,
By all means share the URLs.
Saradhi,
EPO is usually mandated by municipal, or similar fire code when the risk of electrocution by high voltage is present for fire fighters. This is certainly the case with 3-phase wiring in a plenum underfoot. It will vary by municipality or other governing body, but I do know that our solid floor facility has been built in such a way that the Fire Marshall waived the requirement for an EPO.
As for hot air return, in our case the datacenter is on the top floor, and the air handlers are on the roof. Hot air returns are located directly underneath each AC unit. We are currently installing hot air return plenums from each hot aisle back to the returns in order to increase our density. Cold air is delivered by duct work directly to th front of the racks in every cool aisle.
Sartha,
In our case AC power is run from PDU’s out to the racks via conduit and wiremold. DC power is run via the second tier of overhead ladder rack.
–chuck
Comment by cgoolsbee — July 11, 2007 @ 9:08 am
Hi all,
I manage a large non-raised floor environment using both HACS and standard hot aisle/cold aisle. I cannot imagine any true data center that could show better efficiency through forced cold-air under raised floor vs. overhead forced cooling. Laws of gravity prove better efficency, by letting your cold air fall and your hot air rise. Granted the buildings infrastructure needs to allow the hot air to rise high enough without getting trapped around the cabinets.
-Tom
Comment by Tom — July 11, 2007 @ 10:54 am
One of the key purposes of the raised floor is static electriciy dissipation. The tiles prevent personnel from building up a static charge and ‘zapping’ sensitive electronic components. How do you prevent static in a data center without raised floor?
Comment by egallant — July 11, 2007 @ 11:13 am
Conductive floor tiles, conductive adhesive, and all of the above connected to the facility’s electrical ground system. Mitigating static is the same with both raised and solid floor, provide a path for continuous discharge.
I can shuffle around in our solid floor facility in wool socks all day and not build up static on myself.
–chuck
Comment by cgoolsbee — July 11, 2007 @ 1:07 pm
We own 4 former AT&T switch facilities built underground in the late 60’s. I was very concerned about the cost and need of putting in RF. I feel more confident that slab is the way for us to go. I appreciate your article and the comments from others.
Comment by Jack McDonald — July 11, 2007 @ 3:57 pm
I am an electircla engineer and I only got involved in the IS field in 2001 after spending 20 years in a refining & mining operation. Big change.
Approximately 18 months ago I put together a specification for our computer rooms, PABX rooms and radio sites. With my background I am completely in favour as I easily see the advantages of a solid floor over a raised floor. Unfortunately I was told to take the section out on solid floors but next revision it goes back in.
Even if you have to get up on ladders to run new cable it is a lot more cost effective than underfloor running of cable. I get about a 20% saving in labour costs. So if I have my way “the raised floor is doomed”.
Cheers … Bob
Comment by Bob Lawrence — July 11, 2007 @ 4:55 pm
For power in a non-raised floor colocation facility we ran wireway separate from the cable tray for our power. The key to this process was to estimate the quantity and type of power required at the cabinet level so that the wireway could be properly sized and derated. This system allows us to add or change circuits relatively easily.
The challenge of a non-raised floor facility is that duct work is expensive. It really requires careful planning of your room including hot / cold aisles and all but eliminates the ability to change your room configuration in the future. Also, much of the CFD software out there is currently geared toward raised floor facilities, although some of the players are coming around.
Comment by Aaron — July 12, 2007 @ 9:05 am
In a dynamic colo environment, ducting and the use of blanking panels enables cold row/hot row uniformity.
Comment by Dom — July 12, 2007 @ 3:24 pm
This is a great thread. I’ve heard people say that it’s NOT a data center if it doesn’t have a raised floor. Wow! The psychology behind the use of raised floors is amazing.
On another note, I was surprised that no one picked up on the HACS post #13. Hot Aisle Containment Systems capture the hot exhaust air from IT racks and duct into a row-based cooling unit. I may be bias, but in my opinion row-based and rack-based cooling is the only current method able to address the challenges in today’s data center. Things like increasing power densities, dynamic power consumption, and virtualization make it nearly impossible to sustain a data center using only raised floors.
Not to mention the fact that row and rack-based cooling are much more energy efficient than room-based cooling at power densities above 4 kW / rack. This will really hit home with upcoming legislation to decrease data center power consumption.
Comment by Victor Avelar — July 15, 2007 @ 7:58 pm
[…] If the name Douglas Alger sounds familiar, it should. Chuck Goolsbee wrote a post on this blog criticizing a recent white paper that Alger wrote about raised floor vs. solid floor in the data center. […]
Pingback by How fast could God build a data center? — Server Specs — July 17, 2007 @ 3:26 pm
The datacenter that that I manage has raised floor with DX -based CRACs pushing cold air under the floor - we wanted to increase the density of cooling. The “standard” answer was to raise the floor to increase volume of air through the plenum (which would also reduce the data center space due to the increase in space required for the ramp), all in a live, production data center. And alternative was to build out some additional space that we have, leaving the existing data center with underutilized investments in FM 200, VESDA, security etc.
The approach we took instead of the “standard” was to utilize in-row cooling, which increased our cooling density from 1.5 kW up to 6 kW per cabinet. The specific approach selected was from APC, however Liebert also has a similar approach.
APC - http://www.liebert.com/dynamic/displayproduct.asp?id=1077&cycles=60hz
Emerson/Liebert - http://www.liebert.com/dynamic/displayproduct.asp?id=1077&cycles=60hz
We are currently in the planning phase of building additional data center space, which will not include raised floor.
Included below is an excerpt from The Uptime Institute’s white paper titled “2005-2010 Heat Density Trends in Data Processing, Computer Systems, and Telecommunications Equipment”.
“The installation planning guides of several computer manufacturers optimistically assume 700 cubic feet per minute (20 m3/min) from each perforated floor tile on a raised floor. However, field measurements in many sites indicate the actual air flow to be 200 cubic feet per minute (6 m3/min) or less from commonly used 25% opening perforated floor tiles. Under typical computer room conditions, each 200 cubic feet per minute (6m3/min) of cold circulated air will dissipate about 1,000 watts. In some locations within a computer room, air is actually being sucked down into the plenum rather than being pushed up for proper cooling. The actual field conditions should be verified before assumptions are made about a particular site.
Several hardware manufacturers currently have next-generation products on their drawing boards for release in the not-too-distant future that will consume 30 to 50 kilowatts of power. Assuming the width of these new products is 4 feet (two floor tiles), and assuming two facing rows of equipment are to be installed, and further assuming the perforated tile air flow is 200 cubic feet per minute (6m3/min), which is typical of most data centers, the resulting aisle required between rows would be 16 perforated floor tiles or a width of 32 feet just to allow sufficient cooling. This is an unsatisfactory solution and drives the need for senior executives to pay increasing attention to critical physical layer issues. Failure to address these issues will have tremendous cost and availability consequences.”
Comment by Chris — August 24, 2007 @ 11:59 am
I am going to have to disagree, significantly, about the merits of a raised floor. My problem is that there is not enough standardization in several aspects, which I have been driving now for 5 years. I am still writing the white paper to address the challenge, but it really becomes the individual’s past experience that will cause one to lean in either direction.
There are two basic design styles; Teleco and for lack of a better term “IBM White Floor”. Being in a wide array of electronic facilities, I have seen the benefits and faults of each. The “Teleco” design has always favored the above cable plant, therefor, there was no need fro a raised floor, it was always about connectivity. The “IBM White Floor” was always about the versatility of placement of the big iron, lots of whips and clutter under the floor.
I manage 6 data centers, from 3,500ft2 to 72,000ft2, tier 2 through tier 4. These facilities have a total of over 35,000 servers alone and we tech-refresh 30 percent on an annual basis. This will test even the best designs. It will also drive the need for basic standards, otherwise your CAPEX and OPEX will go through the roof.
When I get done with my paper, I hope that we can have a productive and intuitive debate on the technical aspects. Plus the need to have an adaptive environment that lasts more than 5 years without always going back to the well for more money.
Comment by Keith Ward — March 3, 2008 @ 10:28 am
Keith I look forward to reading your paper. Thanks!
Comment by cgoolsbee — March 3, 2008 @ 3:44 pm
I too look forward to Kevin’s white paper. The standards discussion is new to me, and as well, I’m not that familiar with “big iron”, but couldn’t wips be treated the same as other power - ie: run overhead in power-racks?
Cabling (with respect to floor type):
Does the floor type change in any respect how cable infrastructure is best run?
–Eric
Comment by Eric Swanson — March 5, 2008 @ 9:21 am
Apologies, I think my 2nd question can be ignored. It was specific to data cable (and its probably best to do the same and run overhead in a data-rack).
Comment by Eric — March 5, 2008 @ 1:23 pm
We ran a large DC at our university. The building was set up to support a number of mainframes but eventually it was only used for smaller computers set up in racks. Cool air, network cabling and power was done under the raised floor. And that was where the fire entered the DC on November 20, 2002. It used the old (mainly unused) cabling that was put into place for the mainframes and never removed. It was to much of a hassle to remove old cablings. In the 80’s and early 90’s every year between Christmas and New Year a number of employees would get under the floor and clean up. But when it started to take more time than we had between those two days, we stopped.
After the fire we build two new DC’s. Both with raised floors but all cabling overhead. One DC was build in an old telco room. It turned out the floor was to soft to support the supports and where the cabinets where the old floor dropped half an inch. Over time making it impossible to open some cabinets.
I have also advised another company with building their DC. They have no raised floor and the cold air is dropped in front of the cabinets. I find that DC much cleaner and I have the impression the cooling is better.
There is one big disadvantage though. It can be very chilly to get the cold air blown down into the back of ones sweater.
I am asked to give a second opinion on a plan for a new DC at another university and I am not yet sure if I would advice them to drop the floor or not.
Peter
Comment by Peter Peters — March 7, 2008 @ 8:55 am
hello - I will keep it brief..
i have been advised that we could run into issues trying to cool a 20,000 sf datacentre that we’re planning, at 60w psf with a 14ft slab to slab height. When I suggested not having a raised floor, the room was stunned, before claiming it would not work.
any thoughts would be appreciated.
Comment by Dave — March 20, 2008 @ 11:18 am
I put in our computer room at Mercy Medical Center. We have a raised floor with only electrical running under to each rack. I love this setup. The tiles are easily accessible with a suction puller. Our cabinets run around 2000lbs each we have 20. The structure is very strong. All our data cabeling runs overhead on ladder trays.
Comment by Mark Latham — September 8, 2008 @ 11:58 am
You sir have a brain located between your ears! The raised floor concept is simply illogical.
Comment by rebel_kid — October 14, 2008 @ 3:03 am