Lately, many vendors have been using the automatic content download model to slipstream small features/fixes that would not have otherwise warranted a full upgrade release, and some enterprises are opting to enable these features on some or all of their machines.
In a rush to deploy patches and updates though, companies may inadvertently introduce other problems or not fix the original problem completely. New features could be unwanted, cause performance issues, or in the worst case could themselves introduce serious security vulnerabilities.
From an end-user perspective, there are four possible outcomes of a patch or update release:
None of these possible outcomes are as distinct as it may seem; they cross a continuum of all possible patching effects. For example, a patch may mitigate the severity of a vulnerability instead of actually fixing it, or a patch may not introduce a new vulnerability but inadvertently increase the severity of one that already latently exists.
Furthermore, none of these affects may be known at install or test time, they might take weeks or months of live usage to manifest. This runs counter to the assumption that testing will unearth potential landmines buried in a software patch.
Third Party Patches
Another issue to consider is the implication of installing a non-vendor patch for a problem. Installing third party patches can void support agreements even when the patch works perfectly without any adverse side effects.
These third-party patches often help users stay safe from vulnerabilities and attacks but sometimes are not considered because future support is so important. A great example is many of the open-source systems, such as Red Hat Linux, which provide support to enterprise customers.
Often a security patch is released very early after a vulnerability is discovered but won’t be available to the Red Hat customers until later because Red Hat requires each of its bug fixes to undergo testing before release. Installing a third-party patch may void any support agreement purchased from Red Hat, despite the fix potentially working and being available before the official Red Hat patch is released.
Deploying patches can also be especially difficult in a corporate environment as updates are often pushed out automatically to a wide variety of hardware and software configurations. Policies have been created to try to mitigate the risk of pushing out problematic patches, however all problems cannot always be foreseen and tested for. Additionally time spent testing a patch may put users at a high risk of contracting a virus or being exploited.
Let’s look at each of the possible outcomes of applying a patch in detail. The best case where the patch fixes everything it is supposed to with no added unwanted functionality, no adverse functionality/performance issues and no new security issues is wonderful when it happens, is generally expected.
In the next sections we delve into the other possible outcomes; namely what happens when the patch “fails”.
The patch does not fix the vulnerability but has no additional negative impact. This case also includes fixing the vulnerability incompletely.
Vulnerability fixes for security problems often go through rigorous code reviews and testing before they are released to the public for installation, but it’s common that the “fix” actually puts a band-aid on the problem instead of fundamentally removing it.
The patch fixes the vulnerability but introduces a performance, functionality or new security issue. One interesting possibility is that a patch may adversely affect the system yet still fix the vulnerability at hand.
For example, in an early release of a Windows XP security update Microsoft inadvertently introduced a serious performance problem on some systems (MS03-013). The performance issue occurred most commonly on systems with certain real-time third-party software applications, such as anti-virus applications.
The only way to mitigate this fault (until the patch was re-released a month later) was to uninstall the patch or disable real-time anti-virus checking which caused the system to be vulnerable to other types of threats.
The patch does not fix the vulnerability and introduces a performance or functionality issue. This is clearly the worst case, due to a mishap in the patching process the patch that is released fails to fix the vulnerability and introduces new issues.
Because this patch causes serious adverse effects with no benefit the decision to uninstall is easy, the time and productivity lost however makes this patch a costly one. The fact that the patch has caused downtime with no fix at all adds insult to injury and can cause the software company that released the patch to lose face and reputation; future bug fixes will be less trusted.
The lesson of this story is when applying patches you have to consider the range of outcomes and balance the need for protection against the possible instability that a software change introduces into an environment.
Streaming patches to a system directly from the vendor begs the question: Does the testing of the vendor meet the testing needs of my deployment environment?
Many enterprises are opting to do their own internal test pass before patch deployment, but given that a good set of the potential problems caused by a patch may not be uncovered during testing, companies are struggling with their patching strategy. Hence, the right mix of testing and direct vendor streaming is likely to be heavily debated in the months to come.
Joe Basirico’s primary responsibility at Security Innovation is to deliver the company’s security training curriculum to software teams in need of application security expertise. He is also author and content provider of the company’s “SI Security Report”, a quarterly intelligence report that provides an in depth analysis of the techniques and tools that an attacker could use to compromise enterprises.