What does an ERP release, a bug fix, and maintenance on an electrical generator all have in common? They are all changes to an environment that must be managed. Let’s take these examples one step further and review the potential impacts on the organization.
What if the bug fix was for the Web portal which fed information to the ERP system, and the maintenance of the electrical generator also included a switch over to a backup system that shut down the data center for a few minutes? What if all of these changes were scheduled on the same day, at the same time, and the teams associated with these changes did not know about the other events?
If this collision occurred and caused the inevitable production outage, it could be a real disaster for the CIO.
Many organizations adopt ITIL (information technology infrastructure library)for communicating and identifying environmental changes. It does not matter if your business is small, medium or large; as mishandled changes, regardless of size, duration, or complexity, can drastically hurt any size company. What follows are several examples where miscommunicated production changes seriously impacted an organization, and in one case, was a factor in the dissolution of a company.
School’s Out
The first example is a university, where five significant Web outages were experienced within a four week period. The technology group did not have a formal change management process in place and was unaware of simultaneous implementations and the disastrous collisions: online courses became unaccessible, grades were not posted, and students became, as you would expect, irate.
Citing older technology as the issue, the CIO was able to convince management to make a sizable investment in a new hardware platform. The technology group then embarked on a project to manage future outages, including the creation of a hot-site data center, a communication plan to notify users of an outage and implemented ITIL’s change management. Even though outages are now a rare occurrence, the CIO and his team lost political capital and reputation of the IT group declined substantially.
Small Business to No Business
Next, we look at a small retail company that suffered for years with up to 20 Web outages per day. Each of these outages impacted its ecommerce site and revenue. Many of the outages were attributed to older hardware, bad coding practices and frequent implementation collisions between the application and infrastructure teams. Using a three-pronged approach, the company purchased new hardware, fixed the application and introduced ITIL change management.
Unfortunately, it was all a little too late: customer satisfaction levels were already low and the company’s revenues continued to decline with the ongoing loss of clients. These outages were clearly a factor in the ultimate demise of the company.
Big Business Works
A nationwide insurance company, having difficulty managing over 1M changes per year successfully implemented ITIL change management, complete with release and configuration management. The company used a top-down approach, with ongoing support from the CEO, CFO and CIO and implemented the processes across the entire organization, which included seven divisions and over 30,000 users. The organization is now able to effectively manage changes through the ITIL process and has increased the success rate for production implementations.
eTrailing
A major retailer, trying to upgrade their Web platform, experienced a disruption that lasted over a week. Used as the primary sales channel, this outage resulted in a $750,000 loss to the company’s bottom line. Using a SWOT analysis to determine the root cause, it was determined that the outage was caused by a lack of communication between IT support teams that led to an inappropriate number of staff on hand to support the upgrade. Implementing ITIL led to a higher level of communication between the application teams and reduced the number of production outages.
Healthcare Mishap
Lastly, IT employees at a nationwide healthcare provider inconsistently used the change management process, and saw a direct correlation between poor communication and production outages. These outages stopped transmissions between the remote sites and home office systems and led to patient information becoming corrupted and lost. The CIO stepped in, demanded the change management process be used 100% of the time, and the system disruptions subsided rapidly.