How would you like to have your data center situated in a basement one floor below six hot tubs, each containing 1,000 gallons of water? And to make matters worse, you don’t have any fail-over site in the event of a disaster?
That is the situation facing Jeffrey Pelot, CTO at Denver Health and Medical Center.
“An actual failure at our main site served to highlight the problems inherent in having no disaster recovery infrastructure,” said Pelot.
The hospital is now planning to add a nearby disaster recovery (DR) facility so that it can duplicate its various configurations at another completely separate campus location.
It will utilize snapshot, remote IP copy and replication technologies to accomplish this. But all that takes money and the hospital doesn’t have enough in the 2005 budget to cover it.
“We’ve had to push our DR spending plans into 2006,” said Pelot.
Finding a Way
So how do you afford expensive DR projects in the face of tight budgets and a dozen other vital projects vying for priority? One way to cut costs is by prioritization of the data that needs to be fully protected.
Chip Nickolett, a DR specialist from Comprehensive Solutions, a DR consultancy, suggests evaluating systems, data, infrastructure and business operations in terms of categories of DR protection based on specific business requirements.
For example, Tier 1 might be “recover within 24 hours”, Tier 2 might be “recover within 72 hours”, and Tier 3 might be “recover within 10 business days.”
“It’s all a question of how much data can be lost from the point of the disaster going backwards,” said Nickolett.
Take the case of The Members Group, an Iowa-based company that provides card processing and mortgage services for credit unions. The company is implementing an IP SAN by StoneFly Networks for its DR set up.
It has a primary site in Des Moines, Iowa and replicates its data to another site in Minneapolis. As it didn’t have the money to build its own redundant data center, it kept costs down by renting space for its hardware at a network services provider.
According to Jeff Russell, Members Group CIO, this proved to be the make/break point of being able to implement its DR technology.
“Having our systems hosted remotely saves us about one third of the total costs of implementing a DR solution,” said Russell. “Renting made the project possible.”
But even then, the company can’t yet afford to hook up all of its 60 servers to the IP SAN. Therefore, it has prioritized its applications so that only 25 servers need be hooked into the StoneFly DR system.
“From a business standpoint, we decided that those servers that aren’t required to run critical business applications would not be on the IP SAN,” said Russell. “We also went through the business requirements of our applications to determine which ones have to be protected in real time, which need to be up within four hours, and others we can afford to have down for as long as eight hours.”
Slow Adoption
While the events of September 11and the 2003 New York City blackout have prompted some firms to invest more heavily in DR, the overall numbers are surprisingly unimpressive.
It would appear that the harsh realities of the current economic climate have largely curtailed the anticipated rise in spending.
“We see DR spending going down slightly, but spending in high-availability is going up,” said Roberta Witty, an analyst with Gartner.
According to Gartner, DR spending did accelerate in 2002, especially in the mainframe arena. In the past two years, though, mainframe DR budgeting has been slashed, canceling out the up-tick in high availability spending for the Windows, Linux and UNIX platforms.
But, Michael Croy, director of business continuity solutions at Chicago-based IT consultancy and infrastructure firm Forsythe Technology, said that trend seems to be turning around and purse strings are loosening on certain platforms.
“I have seen a three-to-five percent increase in DR technology spending,” said Croy.
Witty believes that part of the weakness in spending is that DR being regarded as purely IT function. Actually, the subject belongs to senior management and falls under the broader category of business continuity. Yet IT tends to either hold onto the function or is saddled with it.
“It is vital to have senior management sponsorship for DR planning,” said Witty.
Without top management involvement, the budgetary realities of DR are often greatly under-estimated.
The IT folks at Deutsche Borse AG, the German exchange for stocks and derivatives, for example, only focused on the thousands of technical details of their OpenVMS-based cluster over two sites situated five kilometers apart.
What came after was a rude awakening — the financial side of the DR equation required just as much planning and detail as the technical side.
“As well as the technical details, you have to plan the financial things in the same depth,” said Michael Gruth, head of systems and network management at Deutsche. “There is no getting around the fact that whatever you do, it is going to be expensive.”
Insurance Policy
Part of the problem may be that DR spending is still viewed as an expense.
When it has to queue up beside other technology projects, it is likely to lose out to higher ROI or mission-critical deployments. But with requirements like Sarbanes-Oxley (SOX) and HIPAA (Heath Insurance Portability and Accountability Act) now being enforced, it is becoming a requirement for companies to be able to protect data and guarantee its accuracy and integrity.
And that can give DR PO’s more leverage in the enterprise, particularly in public companies.
Interestingly, the legislative attention on privacy and records has raised the profile of the subject so much that it is quite common for a company’s DR practices to be placed under the microscope by a potential customer during a due diligence review.
The Members Group IT department, for example, found itself under increasing scrutiny from management and customers with regard to DR.
“When we analyzed our business requirements closely, we found our old DR practices to be inadequate,” said Russell. “We realized the risk to the business and had senior management buy-in from the early stages.”
When placed in this context the focus on doing things right often outweighs a budget target.
“We suggest to our clients that they view the cost of DR planning as insurance,” said Nickolett. “In most cases the cost of being prepared for a disaster is less than the cost of being unprepared for a disaster.”