The staff at SearchDataCenter.com recently split our blog Server Specs into three new blogs in order to narrow our focus on specific aspects of the data center industry:

Data Center Facilities Pro: A SearchDataCenter.com blog about data center facility management, engineering and design. This blog covers news, trends and tips on topics like data center cooling, data center backup power and data center energy efficiency.

Mainframe Propeller Head: A SearchDataCenter.com blog about the IBM mainframe and its alternatives. This blog will cover news, trends and tips on topics like the System z hardware, the z/OS, z/VM and z/VSE operating systems, mainframe jobs and software licensing costs.

Server Farming: This blog serves as forum to discuss the latest in server hardware, systems management, Unix-Linux-Wintel operating systems and large distributed computing systems.

Please bookmark the new blogs. Server Specs will remain online in archive mode. Thanks for reading.

It comes from IT Jungle: “IBM v PSI: The Operation Was a Success, But the Patient Died“

This, I think, is a pretty good summation of IBM’s purchase of plug-compatible mainframe company Platform Solutions Inc. (PSI), which was previously suing Big Blue.

South Dakota’s two largest counties had hours-long lines at their administration buildings this week due to glitches in a new statewide computer system after migrating off the mainframe.

A story in the South Dakota Argus Leader detailed how people waited in lines for hours yesterday waiting to renew their license plates. According to the story, the system changed from a “mainframe system to a Web-based system.” It’s not clear what the “Web-based system” is, but it’s likely to be an x86 infrastructure.

“When a transaction is done on the computer, the computer boots them out, or the system doesn’t do it the way it is supposed to,” Minnehaha County Treasurer Pam Nelson said, according to the Argus Leader. “It doesn’t calculate fees accurately, and they are having to do them manually.”

Oops.

The director of the state’s division of motor vehicles defended the change, saying the first day wasn’t “as bad as (she) thought it could have been,” although Nelson said the new computer system is slower. One last quote:

Over the weekend, the state’s Division of Motor Vehicles changed its old system to the new South Dakota Customized Automated Registration System.

If you didn’t catch it, the new system, Customized Automated Registration System, can be abbreviated as CARS. Well hey, the new computer system might not work, but at least they were able to implement that cute little acronym…

(Photo above courtesy of the Argus Leader.)

Last month I wrote a blog post questioning whether the mainframe specialty processors — and in particular the System z Integrated Information Processor (zIIP) and the z Application Assist Processor (zAAP) — can really save a mainframe shop money. It caused a little stir, from vendors and users alike, who contacted me and talked to each other in defense of the zIIP and the zAAP.

My goal of the post wasn’t to say that they can’t save money, because they can. My point was to explain that oftentimes, the savings comes in an indirect manner. In the case of the zIIPs and zAAPs, I said this:

So you might be able to save in software licensing costs, but the processors themselves cost about $100,000. In talking to users, it seemed to me that the benefit of the zIIP and zAAP was more indirect. By taking workloads to those processors, you can free up room on the central processors. That’s what might matter the most.

So instead of having to buy a new mainframe, you can just buy one or two of these zAAPs and/or zIIPs. The real savings comes not from the reduced software licensing costs as much as it does from being able to buy a six-figure specialty engine instead of a new seven- or eight-figure mainframe.

Some took umbrage with that, saying that savings can be realized immediately with software licensing costs, moreso than I was alluding to. Gregg Willhoit, the chief software architect for mainframe software at DataDirect, added that most customers of theirs don’t consider freeing up space on the central processors to be indirect.

“Most people consider deferred capacity upgrades and the ability to grow existing general purpose processors as close to real money as anything else,” he said. “A lot of people don’t mind considering deferred upgrades as immediate savings.”

DataDirect, which makes service-oriented architecture (SOA) software for the mainframe, has been working to get its applications eligible on the zIIP and zAAP. Willhoit said that for some products, such as Shadow z/Services, up to 85% of the work can be offloaded.

There are other software companies out there that are helping customers offload some work to the zIIP and zAAP, including CA, BMC, and Neon Enterprise Software. Hopefully the list will continue to grow.

Another issue at least one person took was how I said growth of the zAAP and zIIP hasn’t been the same as the Integrated Facility for Linux (IFL), which is another mainframe specialty processor. One person pointed out that IBM says year-over-year growth of specialty engines has been 85%. Impressive? Yes, if you look at the percentage. I would like to see the raw numbers to determine whether the large growth is due to a small base last year. I have yet to been able to get IBM to give me these raw numbers.

I remember a couple years ago when Sun was boasting that year-over-year growth of their x86/x64 servers was 81%. But that’s because they hadn’t previously been selling a lot of x86/x64 servers. If you sell two the first year and four the next, that’s a 100% increase, but it doesn’t necessarily mean there was a huge amount of growth.

In most things, using three systems is more difficult than using two systems. This is true for disaster restart programs as well, so it caught our attention that Hopkinton, Mass.-based EMC Corp. recently released a two-site configuration of its Geographically Dispersed Disaster Restart (GDDR) product. The company already had a three-site version on the market, apparently “cutting their teeth” on the more challenging configuration, and now making the product available to more  shops with smaller, two-site infrastructures.

Like it’s older sibling, the two-site version of GDDR provides a high level of availability in mainframe environments. The product “enables customers to automatically restart host mainframe systems, critical applications and EMC Symmetrix DMX storage systems to minimize the impact of unplanned or planned outages and enhance information availability and protection,” according to EMC’s press release.

This two-site version competes with IBM’s Geographically Dispersed Parallel Sysplex. While IBM sells GDPS as a service engagement, “EMC sells their offering as a product, with services strongly recommended, but not required,” noted Jim Baker, a research manager at IDC Corp. According to Baker, “This means that every change [in GDPS] is cause for a reopener with regard to the professional services price,” while EMC customers know what the costs are up-front with GDDR. And, depending on their sophistication levels (”Very, very sophisticated users,” Baker stressed), customers can implement GDDR without EMC’s assistance.

Another key difference is where the failover control comes from. IBM’s GDPS has K LPAR, meaning that its logical partition is inside the sysplex it controls. Instead, EMC’s GDDR “uses a heartbeat mechanism amongst the entities and can control failover from outside the sysplex,” Baker explained.

While companies that currently run GDPS would have to weigh the pros and cons of switching over to GDDR, Baker believes that the release of a product that competes with GDPS will give customers leverage over IBM.”Competition is a beautiful thing,” said Baker.

GDDR is intended for very large companies that have sysplex in multiple locations and use mainframe-only technology.

“This is a huge step forward for EMC in the mainframe business,” Baker said. “There is still a sharp EMC focus on the mainframe, where most of the world’s transactional data still resides.”

Whenever the California state Legislature needed more money in its coffers, the DMV would pay for it in maintenance costs on its mainframe.

Why? With many applications built on COBOL, changes to the system could lead to time and money spent on altering those legacy applications. One of those changes which came about regularly was whenever the state would hike licensing fees.

But as James Taylor writes, companies can learn to make meals from their mainframe leftovers. What in the world does that mean? It means identifying which parts of your mainframe architecture are static and don’t require a lot of changes — and which do.

In the case of the California DMV, licensing changes were frequent. Other processes, such as the application managing vehicle information, didn’t change much.

And so mainframe migration doesn’t have to mean moving every process off big iron because a few are changing all the time and causing maintenance costs to increase. It means migrating only those oft-changing processes to a distributed platform, where changes might be easier, even if it’s only because of who the company has working for them and what skills they possess. As Taylor puts it:

However, more careful analysis can lead to an interesting insight - much of a typical mainframe system is static, works fine and needs no maintenance. Often only a small portion of the system is responsible for much of the maintenance work.

This post was contributed by regular columnist, Robert Crawford. On the opposite end of our normal content, which provides helpful tips to smooth out your operations, this post is offered up for a bit of comedy. We encourage you to share your comments or your own practical jokes in the comments section at the end.

[Update: For those of you who missed the tongue in cheek nature of this post, it is in fact, a joke. Please don’t try this at work. — Matt Stansberry, Editor]

Work is getting to be too serious. Between the demand for 100% availability and doing more with fewer people, there is little room for those “water cooler” moments we used to enjoy. I say it’s time to revive joy in the workplace and build team spirit. This column suggests some practical jokes that will engender mirth and leave everyone in stitches.

Remap the 3270 emulator keyboard

When a colleague leaves his or her workstation unlocked our first temptation is to send a scathing e-mail to the CEO. But some situations call for something more subtle. Instead of a career-ending missive simply reprogram the victim’s 3270 keyboard mapping.

There are a lot of creative ways to do this. My favorite is to shift every key one to the right so that typing, “logon” will actually be “;phpm” on the screen. Keyboard macros also offer possibilities where a joker could map“left-shift-F1” to type, “tso delete sys1.linklib” and hit enter.

Rename ISPF profile dataset

The lowly ISPF profile dataset (ISPPROF) looks innocuous enough as a simple, 80 byte record library. However, over years of use it accumulates a user’s preferences, job card text, PF key assignments and favorite datasets. Most of us don’t realize how lost we would be without this saved information until it’s gone.

Make someone’s day by deleting his or her personal ISPPROF library. Be sure to stand near your victim’s cubicle so you can hear the hilarious exclamations of disbelief as he or she scrambles to recover. For extra points, be sure to delete any backups in Hierarchical Storage Manager (DFHSM).

DD Dummy a DBMS archive file

Each database management system (DBMS) from IMS to DB2 has an archival process that copies active log records onto tape or other media for safekeeping. Normally these archives aren’t needed except for database recoveries.

Someone with access to the archival JCL could alter the archive output dataset to DUMMY. The records are then copied off of the active logs, but are not saved. The records will soon be gone for good when, in a few hours, the DBMS comes around and overwrites the allegedly archived log.

Imagine how hard your database administrators (DBA’s) will laugh when they have to do a production database recovery only to find they have no records to do so. This also works in cases where a DBMS has to go into the archival logs to emergency restart.

Replace stand-alone dump (SAD)

SAD is the failure data capture tool of last resort designed to gather storage dumps when the system itself is so damaged it can’t recover. System-wide failures are rare nowadays but strict uptime requirements mean any full LPAR failures must be diagnosed on the first occurrence.

Messing with SAD isn’t easy as not everyone is prepared to do the type of primitive, low level programming required. However, a study of SAD’s structure and flow may allow a lesser programmer to cause mischief by replacing a CSECT or two.

There are a couple of variations on this joke. One tactic might be to replace the SAD prompts with questions such as, “What was that IPL volume again?” Another opportunity is to leave the prompts alone and execute enough code to keep the processor busy for the amount of time a SAD usually takes. But, instead of writing storage contents, fill the dump dataset with cheerful speculations about your coworkers’ personal lives. I’m sure IBM support will also appreciate your wit after a long day of looking at normal dumps.

Reinitialize database volumes

This practical joke comes with plausible deniability. After all, anyone could understand why a simple finger check or brain freeze might cause someone to reinitialize the wrong set of volumes.

What I like about this joke is the slow buildup. You can reinitialize the volumes at 08:00 in the morning. Nothing bad will happen immediately as the DBMS usually has the database datasets along with the necessary extent information. However, once the volumes are reinitialized they become available for allocation and the wide open spaces are too tempting of candidate volumes to System Managed Storage (SMS).

Accordingly, by 10:00 or so other datasets will start to pop up on the volumes. As they appear DBMS reports of missing records and I/O errors will trickle in. Soon the errors will become a flood just about the time the DBA’s realize what’s going on.

At that point you can explain your joke to the merriment of all. Better yet, you have your own topper when the DBA’s realize they can’t recover the databases because you dummied out the DBMS log archive a couple of days ago.

ABOUT THE AUTHOR: For 24 years, Robert Crawford has worked off and on as a CICS systems programmer. He is experienced in debugging and tuning applications and has written in COBOL, Assembler and C++ using VSAM, DLI and DB2.

IBM has been pushing mainframe specialty processors — the IFL, zAAP, and zIIP — as a way for mainframers to save money. But do they really?

Trevor Eddolls at the Mainframe Update blog writes about this very topic, calling it a “bit of a swings and roundabouts situation:”

An organization needs to pay for a specialty processor, and it needs to buy software that can make use of the specialty processor, and then it can start saving money – I hear the sound of Excel Pivot tables being used to present the result of what can be quite a complicated calculation. But is there any software that can make use of this hardware?

Eddolls goes on to mention DB2 as one prime workload able to be moved to the zIIP, and there are some other major software vendors out there — CA, BMC, DataDirect and Neon Enterprise Software — that are looking to offer customers the opportunity of running their software on zIIP or zAAP.

IFL, the Linux processor, has been a success for the most part, mainly because IBM has sold it — and customers have bought it — as a way to consolidate hundreds, if not thousands, of Linux instances onto a single box. That has potential savings implications in floor space, power, systems management, and staff.

Selling the zAAP for Java and zIIP for data applications hasn’t been as easy. Why? The consolidation attraction isn’t there for one, and the benefits are more subtle and indirect. The case for the zAAP and zIIP is that you can save software licensing costs by getting workloads off the central processors to the zIIP and zAAP, where workloads don’t count against the million service units (MSU) rating of the box that is used to calculate software licensing costs.

So you might be able to save in software licensing costs, but the processors themselves cost about $100,000. In talking to users, it seemed to me that the benefit of the zIIP and zAAP was more indirect. By taking workloads to those processors, you can free up room on the central processors. That’s what might matter the most.

So instead of having to buy a new mainframe, you can just buy one or two of these zAAPs and/or zIIPs. The real savings comes not from the reduced software licensing costs as much as it does from being able to buy a six-figure specialty engine instead of a new seven- or eight-figure mainframe.

At the beginning of this year, Redmond Magazine announced its Editors’ Choice Awards, handing VMware ESX the trophy for being “most reliable.” In second place? The IBM mainframe.

Why am I mentioning it now when the awards were handed out in January? Well, because I didn’t know of them. A couple colleagues were down at a VMware virtualization forum in New York City recently, and VMware was touting its awards from the magazine, and specifically noting how ESX beat out the mainframe in reliability.

Please keep in mind that this is a magazine focused on the Microsoft IT community, not the IT community as a whole. So for the mainframe, which doesn’t run Windows (yet), to even make it on this list is something. I’m pretty sure the mainframe was the only non-Microsoft related product that placed in any category. Anyway, here are Redmond Magazine’s descriptions for each.

On VMware ESX: “The least stable part of ESX is usually the administrator. The code is virtually bomb-proof.“

On the mainframe: “They’ve been running for more than 50 years, and probably will for another 50.”

Not everyone thinks ESX is “bomb-proof.” On the other end of the extreme spectrum, John Toigo said during a speech last year that ESX was “shoddy” and full of bugs. So the truth is probably somewhere in between. More reliable than the mainframe? That’s questionable, although maybe understandable coming from a Microsoft-focused magazine.

James Governor, an analyst at RedMonk, has a great post on CICS over at the Mainframe Typepad blog. His basic thesis: CICS is becoming a cash-cow because IBM is invested in SOA-ing it to death instead of in expanding the features of CICS itself:

What happens if you want to change the underlying enterprise data model, for example? You can’t do that without changing the code. You can service-enable all you want, but SOA is as much about component and service isolation, enabling flexibility and portfolio maintainability, than service reuse.

Governor, who lists IBM as one of RedMonk’s clients, adds that down at Impact 2008, an IBM conference on SOA held in Las Vegas earlier this month, it sounded to him like IBM was too interested in, as he wrote it, “extending existing investments” instead of trying to find “net new customers for the box.”

“Leverage existing workloads? I am most interested in net new workloads on Z - and I don’t just mean Linux-based,” he wrote.

It’s an interesting concept, and one that I’m always asking IBM about. New mainframe customers, especially in the United States, can be hard to find. It seems that IBM is investing in existing customers in the U.S. and then grabbing new customers outside the U.S. Whenever I ask about new mainframe customers, IBM always falls back on the BRIC acronym — Brazil, Russia, India, China. Hoplon Infotainment is one of IBM’s most-often touted new customers because a) they’re a new customer; and b) they do online gaming on a mainframe, which is a novelty all its own.

But as much as IBM talks about BRIC when it comes to new mainframers, it seems like they’re throwing bricks in the U.S. There may be some American companies new to the mainframe, but overall for new customers, IBM seems focused overseas.