Disaster Recovery Planning – How Far is Far Enough?

No doubt about it, losing bits bites. Even the loss of a tiny amount of information can prove to be a company’s undoing. Lose a lot of them and you can be in a world of trouble from lawsuits to financial ruin.

A big part of disaster recovery, then, is copying the bits and storing them somewhere else other than the main data center. But where should one squirrel that second data center where it will be both safe and readily accessible? In other words, how far away is far enough? It is, by all accounts, a tricky question.

The question of how far is far enough arose after 9/11 when a significant area of Manhattan was shut down as a result of the horrendous terrorist attack on the World Trade Center buildings. Some estimates show as many as 18,000 businesses were affected. “All of the recovery vendor sites were filled to capacity” said Ginnie Stouffer, vice president of Consulting at IDC Partners.

Shortly thereafter, the question on appropriate distance was raised by The Security and Exchange Commission, The Federal Reserve and The Office of the Comptroller. They produced a 2003 position paper, Interagency Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System, that “suggested that 200 miles would be appropriate between the primary and secondary sites,” said Stouffer. But even that suggestion comes under question.

“A backup data center hundreds of miles away can’t effectively mirror data in real time, so the risk of lost data between the last backup and the event is increased,” explained Bill Ashland, founder of Disaster Game―a game that helps organizations prepare for disasters by creating highly detailed event scenarios.

“Disaster recovery site placement isn’t about the IT, it is about the disaster,” said Matt Podowitz, executive director of Business Advisory Services at Grant Thornton. “If the CEO lies awake at night worrying about the Mayan 2012 apocalypse, then the moon or Mars might be good, if impractical, choices for DR site locations. Fortunately, most business executives have a narrower and more realistic view of the disasters they want to hedge against.”

Mapping the Threat

The first step to determining where your disaster recovery center should be is to map the probable threats to your company. Your geographic area will determine the nature of the most likely threat according to Bob Venero, president, CEO and acting CIO of Future Tech Enterprise, a firm that advises entities such as the NY Stock Exchange, JetBlue, Thermo Fisher Scientific, and Northrop Grumman, among others.

“If you’re located where there are natural disaster propensities, then close by may not be the best answer,” said Venero. “Another negative is having the DR facility located on the same power grid as the primary data center, which will then cause the same exact challenges if there is a blackout or brownout.

“If these environments are not a challenge for you, having your DR site close by is a good option because your staff can still get to work and service the facility. Some companies with multiple data centers construct a centrally located DR facility for this purpose.”

Before you stick a pin in the map and call it your new DR home, double-check the logistics of moving to and using the site.

“While almost all companies backup their data, there are other elements to consider in the realm of high availability and disaster recovery,” said Mike Reynolds, senior product marketing manager at Symantec. “The application and the infrastructure need to be protected as well, if the organization intends to continue business operations and/or to minimize downtime when an event occurs.”

Indeed, developing a checklist of needs is in order, then you will know exactly what to backup and how to evaluate the proposed DR site.

“The ‘distance debate’ really boils to risk and cost,” said Brian Barnier, a principal with ValueBridge Advisors who also serves on a variety of best practices committees relating to risk management and business continuity for organizations such as ISACA, the Shared Assessment Program and the Financial Services Roundtable.

Barnier answers the “How far is far enough?” question by pondering the following factors:

  1. The level of data protection and fiber optic distance limits.

  2. The topography of the area for vulnerability to natural disasters and malicious physical attack.

  3. The logistics and infrastructure to support the movement of people (including work at home options) and resupply goods such as diesel fuel.

  4. The nature of facilities options and cost of upgrade.

  5. The regulatory requirements/guidance for specific industries for “out of region” recovery.

  6. The other options to improve recoverability from a range of threats to business operations.

“This last one is vital,” said Barnier. “Some organizations get all wrapped up in distance issues without recognizing other gaping holes to maintaining operationally stable, available, protected and recoverable business systems. Efficient and effective management of risk to business operations is a ‘big picture’ game.”

Rules of Damage

While the perfect answer for each company will differ according to individual need, there are a few general rules to guide you through the site selection process. HP’s George Ferguson, product marketing manager of Security, Compliance, & Continuity Services offers the following tips:

Consider regional disasters first. “If you are on the U.S. Gulf Coast or in the California earthquake zones, then you should have your recovery center at least 100 or more miles away,” he advised. “Another risk factor might be a nuclear power plant.” One HP client based in Northern Europe with a global ERP environment was, for example, concerned about a second Chernobyl-type disaster with potential nuclear fallout.

“They used a local cluster with a tertiary recovery site in North America,” he said. “On a smaller scale, you obviously do not want your recovery center located in the same flood plain as your production data center.”

If wide-spread disasters are not at issue, think smaller distances. “As an absolute minimum, three miles should be sufficient to avoid problems,” he said. Generally, however, having a recovery center at least a 100 miles away should still be considered a best practice for data center recovery. “This is usually impractical, however, for office or work area recovery where transport of significant numbers of people is necessary.”

Don’t overlook people-issues in DR site planning. For work-area recovery, DR centers typically need to be within 20 to 30 miles of the standard work area. “In addition to the distance, ideally you should look for a site which has multiple means of transporting the people to the site (e.g., roads, light rail, train, boat, etc.),” Ferguson said. “This helps ensure that if one transportation mode is down or blocked that your people are still able to arrive at the recovery center.”

Cloud Cover

“By performing a risk assessment on the business that includes geographical location, physical location, labor force availability, point of where the revenue stream is generated, the business will be able to make informed decisions on ‘how far is far enough’, said IDC’s Stouffer.

“The ideal answer to this question is for the business to work towards implementing true Cloud computing. Then there is no dependency upon the physical office space,” she explained. “That means your telephone system, network infrastructure and workspace should all be through a hosted environment.”

Stouffer insists using the Cloud to cover DR issues is a viable option even though for “businesses that have large office pools, true Cloud computing may not be possible because of the constraints of the business operations.” But the Cloud still can help you avoid many of the problems associated with having your own DR site. The Cloud is even beginning to reverse the entire DR equation.

“Some companies are maintaining their data centers at their business location as their ‘secondary’ site with their ‘primary’ site at a hardened facility, often owned by a third party vendor,” she said.

A prolific and versatile writer,Pam Baker’spublished credits include numerous articles in leading publications including, but not limited to: Institutional Investor magazine, CIO.com, NetworkWorld, ComputerWorld, IT World, Linux World, Internet News, E-Commerce Times, LinuxInsider, CIO Today Magazine, NPTech News (nonprofits), MedTech Journal, I Six Sigma magazine, Computer Sweden, NY Times, and Knight-Ridder/McClatchy newspapers. She has also authored several analytical studies on technology and eight books. Baker also wrote and produced an award-winning documentary on paper-making.