Cathy Taylor is the IT operations supervisor at QS/1 Data Systems, which provides products and services to health care providers. Taylor offered some lessons learned based on her own experience with a natural disaster, which prompted her IT department to develop a disaster recovery (DR) plan.

---

What happened during the disaster; did you have plans in place?
In 2003, when Hurricane Isabel hit Richmond, Va., we didn’t have a disaster recovery plan in place. The BellSouth/AT&T central office there was completely flooded, so we lost all communication to our Richmond office. The hurricane made us realize the need to develop a plan. We only lost communication with the one office, and the office itself was not damaged. We also learned what problems were caused by losing communication to that office.

How should businesses ensure that their DR plans are up to date?
Making sure Express Metrix [Software Manager] is installed on every workstation and server is a must. We now make sure we have current backups of all key servers, along with off-site backups. We poll Express Metrix nightly, and all the information about the key servers is recorded in a Lotus Domino database. The information we get from Express Metrix includes the domain joined to, the IP address, manufacturer, model, serial number, number of processors, processor speed, memory, operating system, network adapter information, disk drive information and software installed. This information is replicated to two off-site locations. With this information, we could quickly replace any equipment we needed to.

A lot of companies have chosen to outsource the entire DR process. Have you considered this option?
No, this is not something we have considered or looked into. The argument against it would be cost and the lack of knowledge that an outside source would have of our business. The argument in favor of outsourcing would be that it takes the burden off employees to keep up with and make sure the plan is in place.

Ziff Davis recently performed a survey, sponsored by Symantec Corp., of almost 900 data center managers and found that data centers are understaffed and overworked, and are having a hard time finding qualified workers.

There were many other findings in the study, which Symantec announced this week. But here’s what caught my eye:

• 52% of respondents report their data centers are currently understaffed;
• 86% of respondents have difficulty finding qualified applicants; and
• 68% report staffing is challenging because data centers are too complex to manage.

So what makes for a qualified data center employee? I think it requires a range of IT and facilities knowledge that just doesn’t exist too often out there. It’s a combination of hardware, software, and facilities engineering know-how. That’s not to say that people can’t find their niche within an organization, but unless you’re in a large enterprise company with plenty of data center resources, you often have to be a jack of all trades, and that’s tough.
I think that is recognized in the survey, with 68% saying data centers are too complex to manage. They have become these intricate environments with their own ecologies, and it’s just hard for a lot of companies to keep up with that.

I just saw this cool story about a project at Wachovia Bank’s data center. The company is working with two vendors and the University of North Carolina at Charlotte to create a 3-D map of the firm’s sprawling data center operations. The company is also working with Tideway Systems and Intepoint on the project.

Here’s what the report had to say about the project:

Wachovia is rendering 3-D models down to the level of individual servers in buildings, according to Jacob Hall, who heads the data center group at Wachovia (WB). Wachovia hopes the project will provide better visualization of IT assets and their power consumption. Curiously, the project doesn’t include instrumentation of the equipment for real-time monitoring, a key focus of many data center management products.

In a similar vein, Hewlett-Packard introduced a 3-D mapping tool for data centers this summer, called Thermal Zone Mapping, which generates a 3-D image of a data center, its hot spots and airflow.

Being a child of the 1980s, I think anything in 3-D is pretty rad - even if the technology doesn’t require the ever stylish red-and-blue-lensed cardboard shades.

Data center design consultants, analyst firms and industry groups have lobbied the U.S. Green Building Council to develop a data center standard for its LEED (Leadership in Energy and Environmental Design) rating system for green buildings. But to no avail: USGBC has no plans to address data center facilities. The current LEED rating system does not address the particular needs of the data center environment — you don’t get LEED points for operating your data center efficiently, but you can get one for a bike rack — old news.

Nonetheless, more companies are looking for ways to score points on corporate social responsibility checklists, and municipalities and governments are turning to USGBC’s LEED and mandating it as a design standard. The metric is available, and people get it — imperfect or not.

The LEED system is based on several “green” practices, and if you do things right you get points. So why not copy the LEED format which has been widely accepted and apply it to data centers? Give data centers three points for using server virtualization for server consolidation, three points for implementing blanking panels and sealing off bypass airflow.

Some groups like the Green Grid are looking into this type of points system for the future. Is this something that will work? What are the drawbacks?

While it looks like IT-centric organizations are going to be forced to reinvent the wheel with the green data center certification, it may not be too late to get USGBC on board. Where IT organizations haven’t had much luck lobbying USGBC, you might. End users that are looking into LEED-certified data center facilities need to put pressure on the organization as well.

Check out our LEED data center articles:

Thinking green, data center aims for LEED certification: Hosting firm 365 Main pledges to go green in the data center despite 10% premium on the project.

IBM to build ‘green’ data center: Big Blue pledges to adopt LEED certification guidelines in its internal IT operations.

Green data centers tackle LEED certification: An overview of the first two LEED data center projects, Fannie Mae in Urbana, Md., and Highmark insurance company in Pennsylvania.

Green data center advice: Is LEED feasible? Syska Hennessy consultants offer green data center advice and outline the ramifications of pursuing a LEED data center.

UPDATE: I was talking to Joe Prisco from IBM this week and he has developed a points-based system for determining data center energy efficiency or green-ness. His checklist (and how to use it) will be published in the December issue of the ASHRAE journal magazine. Keep your eyes out for that in your mailbox, and we’ll try to get a link to an electronic copy up as soon as it is available. Prisco hopes the USGBC or The Green Grid will adopt his recommendations and ASHRAE may publish a book in the future on his work.

This week I am reporting from the Uptime Institute’s Data Center Design Charrette 2007, and I can tell you this is not a typical data center conference. Last night’s kickoff event was a panel discussion that took place in a church. We sat in weathered wooden pews in the New Mexico Museum of Art in Santa Fe and listened to the best minds in the IT industry outline the challenge to reducing data center energy consumption.

The attendees are data center managers, vendors and consulting engineers. The plan is to discuss the controversial issues in greening the data center and generate end-user feedback on new technologies and metrics.

There are three discussions going on at this event:

1. One group is delving into the issue of defining the benefits of alternating current (AC) versus direct current (DC) power distribution in the data center.

2. Another group is tackling the recommended temperature and humidity tolerance for servers. Can servers withstand broader ranges of cooling and humidity than currently recommended by vendors and the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE)? Can we harden the hardware to withstand less energy-intensive cooling? And what are the operational cost benefits?

3. Lastly, a group is responding to the Uptime Institute’s most recent green data center measurement strategy, which lays out four metrics for developing a holistic look at energy consumption.

Check in throughout the week for more reporting on the findings and user responses to these data center design questions.

Data center facility costs have grown from the traditional 1% to 3% of IT’s total budget to 5% to 15% today, the Uptime Institute reports.

At the same time, traditional IT total cost of ownership (TCO) calculations and cost justifications have either ignored or significantly underestimated the “true TCO,” because they fail to correctly capture and allocate both the capital and operating expenses of a data center in their planning analysis.

The problem with underestimating energy costs is that when CIOs order new hardware, they later discover that they lack the infrastructure capacity to handle it, the Uptime Institute reports.

For example, a major global financial institution purchased $15 million worth of blade servers, only to discover that the new or upgraded facilities necessary to handle the power and cooling requirements required an additional $50 million in unplanned and unbudgeted investment, said Ken Brill, founder and executive director of Uptime Institute.

IBM’s Deep Computing and Data Center Capacity on Demand business units commissioned the Uptime Institute to develop a True TCO Calculator and a white paper.

The institute will present its white paper, “A simple model for determining true total cost of ownership for data centers,” at its annual Design Charrette 2007 held this week in Santa Fe, N.M. The white paper and the associated True TCO Calculator offer a comprehensive framework for modeling the real lifecycle costs of building and operating a data center, the institute says.

The TCO paper and calculator looks at site infrastructure capital costs, computer equipment costs, energy bills, and other operating expenses to characterize the estimated costs to build and operate a new data center.

Also, the Green Grid released downloadable data center efficiency metrics that look pretty useful.

A handful of vendors at the Data Center Decisions conference in Chicago gave quick, speed-dating style overviews of their latest and greatest technologies yesterday, including a new Active Power Management tool from Cassatt, 10 Gbit Ethernet (GBE) intelligent network interface cards (NICs) from NextXen, iSCSI SANs from Sanrad Inc. and backup tools from Vizioncore.

Sanrad Inc. CEO Dave DuPont discussed “better ways to build storage networking.” While Fibre Channel SAN is widely used and relied upon, an iSCSI implementation can be done using existing storage, Ethernet networks, and Sanrad’s iSCSI switch.

George Pradel, Vizioncore’s senior systems engineer, showed what the company’s flagship product vRanger Pro can do. vRanger Pro, which works with VMware virtualization products, does full-image backups while virtual machines are running. It includes a VSS driver that uses Microsoft’s Volume Shadow Copy Service for consistent images and P2V disaster recovery capabilities.

Agilysys discussed the future of the data center and challenges such as density, power and cooling costs. The company specializes in data center infrastructure.

CXtec ran through their fiber optic cable infrastructure expertise and how to reduce cabling by one third to increase cooling efficiencies with Microcore, a new fiber trunk design. Another storage solution vendor, SANpulse Technologies Inc. discussed its products that automate the SAN.

NextXen pitched users on the benefits of 10 GbE intelligent networking networking technology that keep up with today’s increased CPU utilization, requiring greater bandwidth.

Cassatt proposed an active approach to power management and described a product that intelligently powers servers on and off, either by time of day, demand (i.e., turn off when idle), in power emergencies, or as operational policies dictate.

Pretty cool stuff, even if some of it has been around for a while and was simply spotlighted by vendors at the show.

John Toigo, the keynote speaker this morning at Data Center Decisions, is not very high on VMware ESX. During his speech on disaster recovery planning, he said that he is testing VMware in his lab and found that it’s full of bugs.

Toigo, the CEO and managing principal of Toigo Partners International, added that he can’t understand why people continue “wallow” to VMware like “penguins on their bellies.”

“Put in a z/OS mainframe,” he said. “The costs are actually less than having a bunch of tinker-toy servers running shoddy software.”

In a trend reversal that first emerged in late 2000, salaries for noncertified IT workers now average more than pay for workers with IT certifications, according to Foote Partners’ quarterly IT Skills and Certifications Pay Index, which monitors salaries of 74,000 IT professionals in the U.S. and Canada.

“The corner has officially been turned for IT professionals who choose to market the diversity of their talents, not just their technical skills,” said the 23-page report by Foote Partners, a Vero Beach, Fla.-based IT workforce research firm.

“IT jobs have changed substantially in eight years. In another few decades, the IT organization as we know it today will be hard to find, and so will entire segments of IT jobs. The hurt that has been put on the marketplace reputation of skill certifications is only a drop in the pond of fundamental changes that will reform or destroy dozens of long-held IT industry conventions, beliefs and rituals,” said the firm’s co-founder, CEO and Chief Research Officer David Foote in a statement.

On a more extreme note, I recently wrote about an analyst who said IT department jobs may one day be obsolete because users are becoming more self-reliant.

While I don’t agree with this prediction, it is clear that the scope of IT careers is changing. It seems the biggest issue now is whether students who earn degrees in computer science are learning skills that translate into real-world data center jobs.

For instance, 29-year-old Aaron Sawchuk, CTO of Marlborough, Mass.-based ColoSpace explained how he went to Middlebury College in Vermont with the intention of earning a degree in computer science but switched his major to economics because his studies didn’t pertain to relevant technologies. The IT skills he needed to found ColoSpace he learned on his own.

According to a report from the American Electronics Association (AeA), 118,500 jobs were added to the U.S. high-tech industry between January and June of 2007, bringing the total to 5.94 million jobs. Even so, there is a clear shortage of skilled IT workers to fill those openings.

I am interested in hearing from IT staffers about computer science education. Are today’s college courses relevant to the technologies actually used in today’s data center, or did you feel like a deer in the headlights at your first data center job? If you didn’t go to school to learn your trade, how did you learn? What advice would you give a young person considering a career in IT? Leave a comment here, or send me an email.

Indiana University (IU) broke ground Friday on a new “tornado proof” data center in the twister-prone state.

Compared with the other 49 states in the U.S. by the frequency of tornadoes per square mile, Indiana ranks seventh, so it makes sense for it to spend the extra bucks on a sturdy facility.

The new 82,700-square-foot data center will contain IU’s supercomputer, as well as other IT infrastructure and mission-critical systems. In addition to disasters like tornadoes and storms, the data center is also being designed to withstand electrical damage, power outages and malicious damage, the university reported.

IU’s information technology systems have long been housed in fragile, obsolete old-school buildings.

“Our current data center has long been vulnerable to acts of nature and is insufficient for the needs of IU,” stated IU Vice President of IT Brad Wheeler.

The new data center is designed to withstand an F5 tornado. It provides almost four times the current space and 10 times the electrical capacity of the severely constrained current facility, and has advanced fire suppression and security systems.

When complete, the data center will be connected to a similar facility at IUPUI in Indianapolis by the I-Light optical fiber network, which runs over two separate routes. This enables data to be instantly backed up and shared between both hardened sites, giving reliable backup capabilities.

The University Architect’s Office, SmithGroup, estimates the building will be finished in spring 2009.