Wednesday, January 23, 2008

Enterprise Patch management strategies.
I have seen three different mantras about patching. One mantra was patch early, patch often, another was patch the development servers and roll out to production after two week, the last mantra was never patch.
In the first shop we were constantly patching. It was like putting out fires, people running around frantically whenever a patch was released. Then when something went wrong, a microsoft sde patch broke the client side application, when we tried to back it out there was no option to uninstall. All of the machines with the new patch had to be reloaded. From this came the second mantra, patch development first.
If I break anything in development it is secluded and does not effect any of my clients. This worked successfully for a few years. Then one day the timezones changed. All of the severs that were updated had to be re updated with a new timezone patch and rebooted. After several years of working under this mantra, I ran into a better way.
While reading a book on ITIL I came across the Idea that patching introduced risk, and so did not patching. So the question became what is the best way to minimize risk. Systems level firewalls on all systems. all traffic coming in and going out blocked by default on all systems unless explicitly allowed. Patching only happens in the build process. systems have to be standardized, A new golden image is generated for the systems and rolled out. first to dev, then to test, then to D/R, then to production. This process happens once every three months on 96 servers. Once it is done it starts over again. At first people were skeptical about this approach. But, now it is an accepted norm.