2. Planning
Depending on the volume of patches, it may help to sort patches on the basis of priority and identify whether the patches are to proceed, be placed on hold, or cancelled. Think of it as a form of triage because IT resources are always limited and decisions must be made early on. This does assume, however, that the people making the decisions will be adequately informed about the risks involved.
Patches should be reviewed by system, priority, and category, and also grouped. As opposed to installing each patch as it comes in, organizations need to strongly consider having a policy of grouping patches and deploying them periodically in batches following a solid testing process. For example, one step would be to only apply patches on a bi-weekly schedule. This grouping and delayed application approach need not apply to all situations.
In the case of high-priority patches, wherein the risks associated demand immediate patching, then there must exist means to handle emergency exceptions in an accelerated fashion while maintaining effective controls. In other words, yes, hot patches will come in and demand immediate installation. However, rather than bypass all review and testing steps, there still needs to be a means to review the hot patches and make informed decision about their expedient installation.
Part of the planning process should also define how the appropriate stakeholders would be notified about an upcoming series of patches. The communication plan should outline how the stakeholders will be updated of issues, progress, and completion, as well as any post-implementation reviews. The degree of communication depends on what the patch is, the level of risk, and the stakeholders in question.
3. Initial Testing
Ideally, all patches will be reviewed on segregated test systems that mirror the production environment as closely as possible. The intent, of course, is to test and discover problems prior to going into production. This allows time for issues to be investigated. Again, and it is an ideal, production systems would never be patched directly. However, as mentioned earlier, there are situations, such as Code Red, Nimda and MSBlaster, wherein the security risks are so high, that production systems may need to be patched directly. To reiterate, the risks must be identified and reviewed in order for an informed decision to be made.
Note that testing should not be ad hoc. In other words, testing of each system should follow a formal test plan that outlines the main applications, functionality, test process and expected results if the applications are performing as planned. Yes, this does take a while. However, if stable systems are desired, it is time well spent. If a flawed patch is erroneously approved, installed, and causes production systems to fail, the costs can skyrocket. A decision to bypass testing, or have poor testing, is a gamble that can have disastrous results.
4. Approval
The approval step must be formal. The intent is to take the list of patches, the implementation plan, test results, and present them to a governing body to gain approval to install. The governing body should have the technical knowledge to make an informed decision about the risks and adequacy of the planning.
Even emergency patches must have a defined fast-track process that still requires approval to proceed. Never underestimate the value of review to catch potential issues.
5. Installation
Part of the planning step should be a deployment plan. It may prove beneficial to roll a patch out in phases starting with the least critical systems to see if there are unknown issues that unexpectedly appear in the production environment. In terms of actually installing the patches, there are manual methods and increasingly often, automated update tools that can be used to expedite the installation process. The key here is that installation should follow an approved plan. The actual installation of the patches in production is a relatively small part of the overall patching process.
6. Post Deployment Testing
The military has a saying that few plans survive contact with the enemy. In the context of patches, we must be sure that the deployed patches do not break the production systems. At this point, failures could result due to the patches, due to issues with the deployment system, due to keyboarding error, etc. Regardless of why, it is important that there be previous coordination with stakeholders to quickly assess systems to ensure that they are still operating as planned.
7. Monitoring
Once the patches have been deployed, there should be long-term automated monitoring in place to detect anomalies. Again, because so many variables are in play, even involved test plans may fail to identify a combination of events and values that causes a system to fail. Part of the patching process should be a review of any impacts to the monitoring systems. It may be that patches necessitate changes to production monitoring in order for it to continue to be effective.
In the end, a patch process takes time and effort. As a result, some personnel may elect to attempt bypassing the process for one reason or another. To be successful, the process cannot be partially followed. Everyone must follow the formal patch process.
Configuration Integrity
As a side note, there are automated configuration integrity systems, such as Ecora and Tripwire, which should be used to detect changes. Detected changes must tie out to approved change orders and any others identified as unauthorized changes. All unauthorized changes must be investigated as to why they happened and corrective action taken to prevent them from happening again. Bear in mind a simple auditing tenet — there is no such thing as an immaterial control violation. If a control is bypassed, then a weakness exists and the next breech could be far worse if left uncorrected.
Summary
Software is complicated and there will continue to be issues that necessitate patching. As a result, organizations must develop processes that assess the risks associated with patching and make determinations about what to do and what not to do. Organizations can no longer afford to have a “patch and pray” mentality. Instead, they must view patching as a formal process that is going to be around for a long time.