A Myth Busted: 1U servers do not provide greater density.
It sure would make life a lot easier if the data centers we manage existed entirely within two dimensions; which seems to be the world that server manufacturers think we live in. To them, the only two specifications of size that matter are height and width. They could care less about that forgotten dimension, DEPTH.
Every time a new server appears on the market the very first spec I check is depth. Width is a given, and heights are limited to a very narrow range (1U, 2U, etc), but the makers of gear destined for the data center seem to think they’ve been given a free pass go as deep as they please. This drives me and I can only assume my peers in the community, crazy! Nothing would please me more than to get a bunch of Dell, IBM, HP, Apple, etc server hardware engineers into a room … and then flood it with Halon. Oh, OK maybe not halon… I’d probably hit them with FM200 and then when they come to take them for a tour of a data center and show them the error of their ways.
My biggest beef here is the 1U servers that seem to be growing to absurd depths. The worst offenders I’m dealing with at the moment are Dell’s 1950 and Apple’s latest version of the Xserve. Both arrive at 30″ (76.2cm) or longer. I’m sure there are others that have reached these lengths too. They have roughly the same form factor as the flight deck of an aircraft carrier. 1U x ~18″ x REEEEAAAAALLLLLLLYYYYY LOOOOOOONG. Attach a catapult and you could be launching Maverick and Iceman in their Tomcats to intercept the inbound bogeys.
When 1U servers started appearing they were rather compact, akin in form factor to your average Ethernet switch. 1U x ~18″ x ~12″. Some, such as the old Cobalt “RAQ” web servers (remember those?) could be stacked on both sides of a 2-post rack for a total of 84 servers in under 6sq’ of floor space. When the larger players started shipping 1U boxes, they ranged in depth from 20″ to 24″ (51cm - 61cm) on average. This was the same, or a bit longer than the average 2U and 4U boxes that preceded them, but still manageable. They could fit in 2-post or 4-post racks. But about five years ago 1U servers started getting longer and longer.
How does this affect us? Density of course. While having nothing but 1U servers would seem to be a step towards higher density that is really only true if you live in a two-dimensional world. If your servers are now over twice as deep as they once were entire rows of datacenters have to be moved farther apart to accommodate them. Logically if your rows of racks are farther apart, the number of racks you can install in your datacenter shrinks.
Additionally cabinets keep getting deeper to accommodate these longer and longer servers. It used to be that any server could be mounted in a 2-post rack. Cabinets were only needed if extra security was desired. Now the manufacturers of server EXPECT you to mount them inside cabinets. No flexibility in mounting is offered… except maybe cage nuts or tapped.
I remember when a cabinet averaged 32″ (81cm) deep. Many of today’s servers won’t even fit in a 32″ deep cabinet. Well, they might fit, but you won’t be able to close the doors anymore! The cabinets we’re buying for our data center are now 42″ (107cm) deep. That adds almost two feet (61 cm) to every aisle in the facility. That means you can fit fewer aisles. By my top-of-my-head math that means you lose two full aisles for every 5000 sq’ of data center. Depending on the number of racks-per-aisle, that can add up to a LOT of servers you lose by having these outrageously long boxes.
There are many facilities, primarily older colocation datacenters that limit how much power-per-rack you can use, so frequently you see 42U cabinets with MAYBE 14U of space that is usable. Why even bother with 1U servers then? Your cooling is messed up by all the empty space. You might as well go back to the big 4U servers of yesteryear and pack ‘em in. But nobody makes those anymore. The only time you see servers larger than 1U is when they are serious power hogs, packed with drives and CPU. So we’re back to square one.
Two recent events triggered this rant:
A customer sent a new Apple Xserve to us to replace their old Apple Xserve. The old one was a G5 unit, the new one a Dual-Core Xeon unit. Both we and the customer thought this would be an easy swap… power down, unplug, and pull out the old one, slide in the new one plug in and power up. Minimal downtime. Unfortunately the new one is two inches longer, the ports (network and power) have swapped sides, and the rack mounting hardware is completely different. What should have been a 5 minute operation turned into a multi-hour ordeal.
The next event, which sent me over the top is a new client had 32 Dell 1950’s shipped to our facility, along with an APC Netshelter cabinet and powerstrips to plug it all in. Upon arriving for assembly we noted that the rackmounting rails provided by Dell stuck out in the back of the 32″ (81cm) deep 1950 by an additional 3″ (7.6cm). So now the total depth of the servers amounted to 35″ (89cm). There was no longer enough room at the back of the cabinet to mount the power strips. Comparing these to previously installed Dell 1950 servers they did not have rails this long. What does this mean? Is Dell planning on making their next rackmount servers even LONGER? How long before we see 36″ long servers?
Do server designers ever try to actually rackmount their gear? Do they account for cables, power strips, etc? It seems to me that the unrestricted lengthening of the standard 1U server is becoming completely counter-productive to the original design goal of the 1U server, namely density of computing in the minimum amount of space. They’ve gained rackspace at the expense of usable FLOOR space. In the balance sheet of datacenter operations floor space is WAY more expensive than rackspace. I want my floor space back.
Does this frustrate you as much as it frustrates me?
Posted: September 17th, 2007 under Data center physical infrastructure cabling, Capacity planning, Data center standards and metrics, x86 servers.
I like your rant! The company I work for makes racks (I haven’t linked to their site as I don’t want this to be seen as a marketing thing), and it is a constant battle for us trying to make everything deeper and deeper. One rant I would add is that we have had a nightmare with Dell’s quick fit slides. They are a great time saver, and a really nice design. Trouble is that they don’t comply with SSI guidelines which Dell signed up for. What this means in practise is that they don’t fit in to some perfectly compliant racks. We have had to change all our design and manufacture system for this simple thing, and even now we still get some that don’t fit! I’m not sure how you feel about this but try telling a data centre manager it’s Dell’s fault!
I have reported it to them, but……
To be honest I have measured these and I think it looks like they made a simple mistake, but I wish they would fix it!
Comment by Simon Griffiths — September 18, 2007 @ 1:36 am
Thank you Simon.
We’ve been bitten by the Dell “Quick Fit” rail issue as well:
http://chuck.goolsbee.org/archives/408
Drove us absolutely insane. It of course happened in a late-night rush-job install as well. You are right, it IS Dell’s fault.
–chuck
Comment by cgoolsbee — September 18, 2007 @ 10:36 am
HI Chuck
This seems like a bait and switch scheme from hardware vendors — offering servers that promise density, but only in terms of height and width, not depth. Are they trying to get users to buy deeper racks after unknowingly purchasing deeper servers?
This is a great story for Seacrhdatacenter.com to get to the bottom of. Shoot me an email at bbotelho@techtarget.com.
Thanks
Comment by bbotelho — September 18, 2007 @ 12:34 pm
I too work for a server rack manufacturer and I am glad you bring up this topic that most are unfamiliar with. Our noise reducing server rack cabinets have 32+” rail to rail distance which has been plenty long in the past, but some of the new servers are starting to push the limits.
The use of a cable management “arm” that sticks out the back of the server a couple more inches has been my biggest issue, the arm is not always mentioned up front in server marketing materials (and/or customers don’t always notice it) and can become a big “gotcha” during install time.
Comment by Ken Vitto — September 18, 2007 @ 12:54 pm
An absolutely beautiful article - articulate and informative.
Comment by David — September 20, 2007 @ 11:39 am
It seems that those seeking density should be thinking of blade-based servers. I work for a software company, but we have over 1,000 servers for load testing, QA, dev, etc.
A good number of this are rackmounted blade servers, and one of the appeals of going that way was exactly this depth issue.
That would not help your Apple customer - as far as I know, there are no Apple server blades - but for the Linux and Windows crowd, there are any number of choices, with good options from HP and Verari, and an OK option from IBM. The blades also cut down on the cabling (depending on which one you go with, and whether they have embedded switches, used the blade backplane for networking, etc).
The big downside to blades is that they only make sense for customers who need at least n servers. If you want just a couple, the overhead of the blade chassis makes cost prohibitive - and that puts the problem of racking, cabling, and depth on the shoulders of the hosting site provider…..which is you.
I feel your pain!
Comment by Ursa George — September 20, 2007 @ 11:39 am
Much to the point…with Dell 1950’s in APC racks using dual power distribution (two 0U powerstrips) the Dell cable management mount points overlap the power strips, blocking out
a pair of receptacles and also making it hard to mount the rails.
There are reasons…cooling the higher performance machines means stuffing the box full of fans, and dual power supplies mounted end to end don’t help either. At least the 1950’s don’t have their network ports buried at bottom of 3/4″ wide 2″ deep slot requiring strange 3″ extender cables.
Comment by Eugene Day — September 20, 2007 @ 12:16 pm
Remember how big monitors were getting before LCD’s came along?
Remember how big servers were getting before drive enclosures came along?
One of the few things you can count on in IT is change.
Don’t get me wrong, though. Gripe about the problems hardware or software engineers create. It’s the only way they get feedback from us.
For those reporting problems with rail length: Once they are past the end of the server they become useless unless you like the cable management arms.
While you’re waiting for them to come out with a new design, get a SawzAll and lop of the back end of the rails. (Outside your datacenter, metal shavings don’t do much good for electronics)
Comment by Ben — September 20, 2007 @ 12:55 pm
Amen, brother.
Comment by Adrian Close — September 20, 2007 @ 9:01 pm
A very useful warning for those thinking of getting 1u servers to put in collocation. Thanks.
Comment by Paul Milligan — September 21, 2007 @ 3:52 am
Ben, the sawzall and porta-band are always in our toolbox. While we’ve never been hesitant to lop off annoying bits of equipment (cabinet rails for example, which always want to go right where the expansion bolt secures the cabinet to the floor!) in our case the server mounting rails usually belong to our customer, not us. I’m sure they would be a bit terrified if we strode over to their cabinet sawzall buzzing away in hand!
I appreciate the feedback and comments. Nice to know I’m not alone in this bit of frustration.
–chuck
Comment by cgoolsbee — September 24, 2007 @ 12:34 pm
Servers plus cable mgt over 800 mm deep became such a problem I designed rack extensions to increase the rack depth. Pop the doors off, attach a 42 or 45 U high extension cabinet (quickly attaches in 5 minutes) - each 100 to 130 mm deep - on front and back, then put the doors back on. I was lucky to have aisles 1200mm wide, and the original cabinets were only 800 to 900 deep. At the same time the 1 U servers etc started growing longer, fibre became more standard and there was no space to run managed fibre or patch. In the last 3 years I’ve gone from about 3-500 fibres in the Optical Fibre Distribution Frame to about 3000. The Telect ADF does 2304 LC’s in one bay
In the rack extension I have built Warren & Brown 60 x 40 mm fibre trunking and intergrated this to the Technology Centre (Telco and Data Servers) fibre mgt system going to Telect ADF OFDF
Its been really important to put blanking panels into enclosures/racks/cabinets to stop the heat recirculating - so when you repace a 3U with a 1 U you need 2 U of blanking panels
The devil is in the detail
Comment by Paul T — September 24, 2007 @ 6:55 pm
Almost a year ago we introduced several 1U Rack Servers that are 20″ deep. Additionally we offer NEBS Level-3 in 20″ depths. I guess it depends where people go looking for Servers or the key phrases entered into various search engines.
Here is a link to the current servers that are no more than 20″ deep.
http://www.freedomtc.com/specialized_chassis.php
Comment by Lanny — September 25, 2007 @ 6:56 pm
Guys, the extra 4 inches or so that the 1U servers are don’t pose a problem and you don’t have to redesign your datacenter. In fact, this doesn’t really affect anything unless you’re designing your datacenter with aisles too close in the first place. Also, realize in the datacenter space, sqftage is the least expensive part, you’ve got bigger issues to deal with besides how much space you’re taking up.
Comment by doubleshot — September 30, 2007 @ 3:41 am
This isn’t about 4″ doubleshot. 1U servers have grown OVER 12″ since their introduction. The question I’m asking is when will it end? Until the market says “enough!” the manufacturers will continue to make them longer. How long is too long? I say anything over 28″ is too long.
Square footage is the main metric ($ per sq̵
by which you calculate datacenter cost, not to mention density (W/sq̵ 
so I don’t see how it is the “least expensive part”.
–chuck
Comment by cgoolsbee — October 1, 2007 @ 9:59 am
[…] Server hardware management: Get tips from the trenches with our writers like Chuck Goolsbee and Kyle Rankin. Goolsbee busts Dell and Apple in this post for making servers that are too long to fit in the racks and vows to whip out the Sawzall. “The new [server] is two inches longer, the ports (network and power) have swapped sides, and the rack mounting hardware is completely different. What should have been a 5 minute operation turned into a multi-hour ordeal.” […]
Pingback by Biggest data center stories of 2007 — Server Specs — December 13, 2007 @ 6:43 pm
I have a 4U server that is 67CM in length. So, its not just 1U’s that are affected. I can only imagine them getting bigger. Density to me means processing power per U. I recently configured a 1U server that was 48GHz combined total power with a 70CM depth. This unit also had 4TB of disk space in a 1U server. I think you are wrong about the density factor.
Comment by Owen — January 24, 2008 @ 2:15 pm