In the wake of Hurricane Katrina, many businesses scrambled to update their disaster-recovery plans. Unfortunately, a second hurricane, Rita, was bearing down on them just as they had begun assessing the damage.
While Rita didn’t cause the kind of damage that Katrina did, it provided a cautionary tale: it is no longer enough to prepare for one disaster, since a second could be right behind it.
The last catastrophe to gain this much attention, 9/11, taught us much about disaster recovery. The problem, however, is that few businesses acted on what they learned because another terrorist attack of that scale was considered unlikely.
“After Katrina, I’m hoping that organizations will come to grips with the fact that while bombs may not be a regular enough occurrence to protect against, natural disasters are,” said Steve Duplessie, senior analyst and founder of the Enterprise Strategy Group.
Disasters come in many forms, so even if you aren’t a likely terrorist target, you will be vulnerable to something else—hurricanes, floods, fires, or even power outages.
Lesson No.1
Even if you diligently backup your data, you still may not be prepared for the next disaster. “Disaster recovery is no longer just an IT problem, but rather an overall corporate problem,” said Kirby Wadsworth, senior vice president of marketing and business for Revivio, a provider of continuous data protection systems.
If people are unable to access data, what good are backups? After 9/11 there was plenty of backed up data sitting idle in New Jersey. The data was there, but the people were stuck in Manhattan, or, in the worst case, they died when the towers came down.
“Another problem,” Wadsworth said, “was that when people got to their backup tapes, they had no place to run them. Their data centers had been destroyed, so you had people pulling their tapes from New Jersey and traveling to Chicago to access them.”
Sure, they managed to get their data restored and their critical applications back online, but it took several days to do so. Since the goal for most organizations is not simply data backup, but business continuity, this was a huge problem.
Lesson No.2
The second lesson to emerge from Katrina is also linked to 9/11: data has no value without people. Business continuity is about keeping people and applications connected. This lesson was clear after 9/11, but, again, few took it to heart.
So, when communications in the Gulf region were destroyed, even if organizations had a remote data center up and running, there was no way for key employees to get in touch with that data center. Those who were unfamiliar with disaster-recovery plans had no way of contacting those who were.
“You need a plan for putting people in touch with critical applications,” said David Palermo, vice president of marketing for SunGard Data Systems, a provider of business continuity services. “If the phone lines are down throughout an entire region, what do you do? You need to plan for that. You need a rallying point, such as a online bulletin board.”
Simply having a plan that involves how to back up data and how to get it back online is not sufficient. When planning for disasters, companies need to look at just what it takes to make key applications functional.
“Planning is about more than data and data centers,” Palermo said. “You need to consider supplier relationships, customer relationships, and internal business units. You need to ask who needs access to what.”
Beyond that, companies should prioritize, determining which applications are critical and which are not. For instance, if you have an inventory application that you only run once a month, it’s fine to rely on tape backups.
Unfortunately, most enterprises simply treat data as data, and it’s all backed up in the same way. Critical systems should have continuous backup, and you should know how to get them up and running remotely.
Lesson No.3
The fact that communications were disrupted throughout an entire region points to another, less obvious lesson in today’s “connected” world: distance matters.
“Many of the disaster-recovery sites for New Orleans were in Biloxi,” Duplessie said. A few hundred miles is not the distance you need to be protected from a Katrina-scale disaster, but transporting data long distances poses its own problems.
If you rely on tapes, those tapes are typically only taken off-site sporadically, so you run the risk of losing plenty of data between backup cycles. Online continuous backups solve that problem, but then distance is an issue.
“Until recently, it was not possible to get data far enough away from its source without affecting performance,” said Wadsworth of Revivio. “Typically, you would do a synchronous replication to a nearby location, say one 50 miles away. Then you would have to replicate that data again, hopping it to another site further away.”
For transaction-intensive businesses, such as financial institutions, they still risked losing critical data since the distant replications were not live. It’s quite difficult to get both enough recovery points that you have near real-time recovery, while also getting enough distance that something like Katrina won’t damage both your source and your backups.
Fortunately, new technologies are emerging that help.
Continuous data backup solutions, such as those from Revivio and XOSoft, only backup unique blocks of data, which saves bandwidth while providing near-instantaneous access to finely granular recovery points.
Wide-area file services, such as those from Riverbed and Tacit Networks, help eliminate the time lags associated with sending large data files over great distances, and business continuity services like those from SunGard provide not only backup services, but also the manpower to help you utilize your backups.
Lesson No.4
In the past, business continuity translated into doubling your operating expenses. Simply having backups allowed you to recover your data, but not without experiencing a lot of downtime. If you wanted to ensure 100% availability, when you put a server at site one, you had to have a backup server at a remote location.
For an event that might not happen, that’s simply not cost effective. However, with new business continuity, storage, and WAN delivery solutions available, the cost of not planning for a disaster can be prohibitive.
“These new solutions make it not only technically, but also economically viable to put disaster recovery facilities vast distances apart from primary sites,” Wadsworth said. “With continuous data protection, for instance, now you can have a backup data center hundreds or thousands of miles away.”
And it will cost you a whole lot less than being unprepared for the next Katrina.