Last September, Henry Schein, Inc., the Melville, New York-based distributor of medical equipment, was well into designing an updated disaster recovery plan for its network.
When the twin towers fell on the 11th, phone service to the northeast was disrupted, including the company’s centralized network on Long Island. As a result, over 700 employees in call centers around the country had no way of managing orders from the $2.6 billion company’s 400,000 customers around the globe. Its lifeblood was cut off.
“After September 11, our attention (to disaster recovery) went to a peak level,” says Jim Harding, chief information officer of Henry Schein. “It required us to have to execute a lot faster.”
Because 95 percent of the company’s orders are taken over the phone before they are routed to seven distribution centers around the country, rolling the network’s phone system to another location was a major consideration.
Then came the decision-making on which applications required “hot” backup, or duplication of systems somewhere else, and which applications could go to slower-recovery tape storage.
Of course, the faster the recovery time the company wanted, the more the costs went up. Restoring critical applications within two to four hours, for example, would boost its IT budget tenfold.
“There were thousands of details to think about, as you can imagine.”
Some procedures were relegated to manual processes, such as order-taking. Harding says the company also decided to stick with its somewhat outdated but reliable ISDN lines for phone backup too.
The company ended up buying 10 terabytes worth of backup for its data with some but certainly not all of it mirrored to another site in real time. The project effectively doubled its security budget from 2000 levels.
All in all, however, the company estimates its IT budget is about 10 percent higher as a result of its business continuity planning.
Disaster Recovery Plans: Still on the Drawing Board
Would Henry Schein’s disaster recovery experience rank as the exception or the rule across the nation’s businesses? Networking experts say the answer depends on the sector. In the financial services industry, for example, a firm without a disaster recovery system might be considered in breach of its fiduciary responsibilities.
But in other sectors, the extent of business disaster recovery planning isn’t so clear, even a year after the September 11 attacks.
AT&T recently polled over 1000 executives of companies with 100 employees or more in major cities around the nation, asking them about their business recovery plans. Although the results showed increased concerns about business continuity and recovery from disasters, it also showed that many plans remain on the drawing board.
The survey found one in four medium to large businesses nationwide had no business continuity plans in place. For the companies with plans, 19 percent said they had not been tested in the last five years. Forty percent had at least organized a team or process to focus on business continuity planning; almost 20 percent had a plan or planned to have one.
Tech experts say the issue is cost. Storage back up systems can be expensive.
Take a survey of just under 300 companies by Vista Research. It said 48 percent of primary decision makers in medium sized businesses increased their IT budgets since September of last year, more than half of them by 10 percent. And 57 percent of those polled said they planned to increase their IT spending over the next year.
But perhaps the more telling snapshot was that almost 40 percent said they increased their security budgets only by cannibalizing other parts of their IT systems and budget.
“I think it’s a large problem for mid-sized businesses especially,” says Michael Curry, a managing director with Richard Fleischman & Associates, an IT consultancy in New York. “A lot of IT managers feel like Oliver Twist asking for more money” for backup systems. “It puts people into a conundrum. They need to do it, but they don’t have the money.”
As a result, he sees companies doing what they can for less, such as choosing to replicate some department heads’ systems or senior managers’ systems across company divisions.
“I think in the future, (disaster recovery) won’t be seen so much as an investment but as the cost of doing business. But I don’t think that has really sunk in yet,” says Curry.
For starters, the nature of companies’ budget cycles means it can take about two years before new plans are actually put into action. So for now, many businesses are thinking of perhaps replicating 15 percent of their systems, and maybe 10 percent of back office systems. “But even that is probably not budgeted for,” Curry adds.